JP2012161269A - Insect image processing apparatus, image processing method, image processing program, and computer-readable storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a system capable of counting and discriminating insects conveniently and promptly.SOLUTION: An insect image processing apparatus includes: an image acquisition unit 1 for acquiring an object image which may include insects; an image extraction unit 11 for extracting images of insects from the object image acquired by the image acquisition unit 1; an information analyzer 12 for analyzing information relating to insects by image processing from the extraction region extracted by the image extraction unit; a discriminator 13 for discriminating whether it is an insect based on the information relating to the insects analyzed by the information analyzer 12; a counter 15 for counting the number of the extraction regions discriminated as insects by the discriminator 13; and an output means for outputting the result counted by the counter 15 as the number of insects.

Description

  The present invention relates to an image processing apparatus, an image processing method, an image processing program, and a computer-readable storage medium for performing processing such as counting of insects contained in a captured image and discrimination of types.

  2. Description of the Related Art Conventionally, agricultural chemicals (chemical pesticides) are sprayed to remove pests and pests when growing crops. In addition to chemical pesticides, a method is also used in which natural enemy insects (beneficial insects) are introduced as biological pesticides against pests to eliminate pests. In any case, in order to confirm the spraying time and amount of these pesticides and biological pesticides, confirmation of the type of pests and counting the amount thereof have been performed. Conventionally, the user has to visually determine the number of pests and count the amount, and this method has a problem that it requires considerable skill and is extremely troublesome. For example, if a sheet-shaped adhesive trap coated with an adhesive is suspended in a greenhouse and the pests attached to each adhesive trap are visually counted, one adhesive trap is placed per are If the area is 8 hectares, 80 sticky traps must be used to investigate the occurrence of micro pests. The person in charge takes about 3 hours every week for this work. In addition, depending on the type of pest, the size may be several millimeters, and it is not easy to identify the type as a bug, a garbage, or an insect. In order to save such work, there has been proposed an image processing counting device that automatically counts worms, which are pests (for example, Patent Document 1).

JP-A-2005-21074 JP 2008-99598 A JP 2003-168181 A Japanese Patent No. 4469961 JP 2003-169484 A

Kenji Terada, Detection of micro pests using camera images, Image Lab, 2009.7.

  The apparatus disclosed in Patent Document 1 uses an apparatus as shown in FIG. 63 to blow air sucked from the vicinity of agricultural products onto the adhesive surface of the adhesive sheet, and photograph the adhesive sheet portion on which the stag beetle was sprayed. Analyzing images taken using a computer and counting the number of stag beetles captured.

  However, according to this document, the image processing method simply substitutes the RGB values of the captured image into a simple approximate expression and converts them into luminance, color difference signal 1, color difference signal 2, area, and circularity, and FIG. The threshold value shown in FIG. 5 is compared to determine whether or not it is a stag beetle. For example, there is no disclosure regarding a specific method for setting the threshold value.

  In addition, we are only counting the scale insects, and for other insects, “By changing the threshold in FIG. 65, various insects such as other insects and beneficial insects will be changed. It is possible to count. ”(0032 of Patent Document 1), only the possibility is suggested, and no specific conditions are disclosed. In addition, multiple types of insects are not distinguished.

  According to the test conducted by the present inventors, it was found that the determination based only on such a simple approximate expression is inaccurate and does not reach a level suitable for practical use. In addition, this method requires a considerably large device as shown in FIG. 63 and, as shown in 0026, “The insect attachment area is about 20 cm × about 20 cm, and if taken at intervals of about 3 mm, about 4,300 or more are taken. "The number of screens per day is about 1,000 because of the image processing time." It seems that it has not reached the level where it can be made.

  The present invention has been made in view of such a background. The main object of the present invention is to enable easy and quick counting and discrimination of insects, which can be used in a simple system, and can be read by an insect image processing apparatus, image processing method, image processing program, and computer It is to provide a storage medium.

Means for Solving the Problems and Effects of the Invention

  To achieve the above object, according to the insect image processing apparatus according to the first aspect of the present invention, there is provided an image processing apparatus for specifying information relating to an insect contained in a captured image. An image acquisition means for acquiring a target image that may be included, an image extraction means for extracting an image corresponding to a worm from the target image acquired by the image acquisition means, and an image extracted by the image extraction means Information analysis means for analyzing information about insects by image processing from the extraction area, determination means for determining whether or not the insects are based on information about the insects analyzed by the information analysis means, and Counting means for counting the number of discriminated extraction regions and output means for outputting the result counted by the counting means as the number of insects can be provided. As a result, information such as the type and number of insects contained in the acquired target image can be automatically acquired, work that has been performed visually by a skilled user can be automated, and determination criteria can be made constant. Benefits are gained.

  In addition, the insect image processing apparatus according to the second aspect further includes database holding means for holding a feature quantity database, wherein the discrimination means stores the feature quantity database held in the database holding means. With reference to this, it is possible to determine using the feature amount.

  Furthermore, according to the insect image processing apparatus according to the third aspect, the discrimination means can recognize the insect type.

  Furthermore, according to the insect image processing apparatus according to the fourth aspect, when the resolution of the target image is high, the information analysis unit performs processing using a feature amount, and when the resolution of the target image is low Processing using template matching can be performed.

  Furthermore, according to the insect image processing apparatus according to the fifth aspect, the feature extracted by the information analysis means may include any one of the overall length of the insect, the trunk, the limbs, and the perimeter as a feature amount. it can.

  Furthermore, according to the insect image processing apparatus according to the sixth aspect, when the discrimination means recognizes the type of insect, a plurality of ones with partially different surface colors are used for the same type of insect. And different feature amounts are given according to the color of the classified insect, and discrimination can be performed using the different feature amounts.

  Furthermore, according to the insect image processing apparatus according to the seventh aspect, the determining means assigns a common feature amount while assigning different feature amounts according to the color of the classified insect. The determination can be made by using the plurality of feature amounts.

  Furthermore, according to the insect image processing apparatus of the eighth aspect, the insect to be recognized can be any one of thrips, whiteflies, and stag beetles.

  Furthermore, according to the insect image processing apparatus of the ninth aspect, the image acquisition means can include an image sensor for capturing an image.

  Furthermore, according to the insect image processing apparatus of the tenth aspect, the image acquisition means can be a network camera.

  Furthermore, according to the insect image processing apparatus of the eleventh aspect, the image acquisition means can be a scanner.

  Furthermore, according to the insect image processing apparatus according to the twelfth aspect, the target image can be an insect image adhered to the adhesive sheet.

  Furthermore, according to the insect image processing apparatus according to the thirteenth aspect, the information analysis unit removes the background portion of the adhesive sheet from the target image, performs shaping and separation of each object to be acquired, Based on the size and shape, the determination means can determine insects.

  Furthermore, according to the insect image processing apparatus of the fourteenth aspect, a yellow or blue one can be used as the adhesive sheet. This provides an advantage that the background can be easily removed by image processing using colors that are rarely used in the greenhouse.

  Furthermore, according to the insect image processing method according to the fifteenth aspect, an image processing method for specifying information about an insect included in a captured image, the target image possibly including an insect A step of extracting the image corresponding to the insect from the acquired target image, a step of analyzing information on the insect from the extracted extraction region by image processing, and information on the extracted insect Based on this, it is possible to include a step of determining whether or not it is an insect, a step of counting the number of extraction areas determined to be insects, and a step of outputting the counted result as the number of insects. As a result, information such as the type and number of insects contained in the acquired target image can be automatically acquired, work that has been performed visually by a skilled user can be automated, and determination criteria can be made constant. Benefits are gained.

  Furthermore, according to the insect image processing apparatus according to the sixteenth aspect, there is provided an image processing program for specifying information relating to an insect included in a captured image, and a target image that may include an insect. A function to extract the image corresponding to the insect from the acquired target image, a function to analyze information about the insect by image processing from the extracted extraction area, and information about the extracted insect Based on this, it is possible to cause the computer to realize the function of determining whether or not it is an insect, the function of counting the number of extraction areas determined to be insects, and the function of outputting the counted results as the number of insects. As a result, information such as the type and number of insects contained in the acquired target image can be automatically acquired, work that has been performed visually by a skilled user can be automated, and determination criteria can be made constant. Benefits are gained.

  A computer-readable recording medium according to the seventeenth aspect stores the program. Recording media include CD-ROM, CD-R, CD-RW, flexible disk, magnetic tape, MO, DVD-ROM, DVD-RAM, DVD-R, DVD + R, DVD-RW, DVD + RW, Blu-ray, HD A medium that can store a program such as a magnetic disk such as a DVD (AOD), an optical disk, a magneto-optical disk, a semiconductor memory, or the like is included. The program includes not only a program stored in the recording medium and distributed but also a program distributed by download through a network line such as the Internet. Further, the recording medium includes a device capable of recording a program, for example, a general purpose or dedicated device in which the program is implemented in a state where it can be executed in the form of software, firmware, or the like. Furthermore, each process and function included in the program may be executed by computer-executable program software, or each part of the process or hardware may be executed by hardware such as a predetermined gate array (FPGA, ASIC), or program software. And a partial hardware module that realizes a part of hardware elements may be mixed.

1 is a block diagram showing an insect image processing apparatus according to an embodiment of the present invention. It is a block diagram which shows the insect image processing apparatus which concerns on a modification. It is a block diagram which shows the insect image processing apparatus which concerns on another modification. It is a flowchart which shows the procedure which performs discrimination | determination of an insect. It is an image figure which shows the image of the tomato leaf fly which was imaged by 80x80 pixels. The example of the target image which imaged the adhesive sheet is shown. It is an image figure which shows the example which imaged the extraction result of four extracted feature-values. It is an image figure which shows a synthesized image. It is an image figure which shows the result of having calculated | required distribution with respect to the synthesized image of FIG. It is an image figure which shows the position which performed the extraction process with respect to the synthesized image of FIG. It is an image figure which shows the synthesized image imaged at the distance of 6 m. It is the graph which showed the measurement result by the numerical value. It is a flowchart which shows the procedure which identifies a whitefly. It is a schematic diagram which shows a mode that the small area | region of an acquisition image is set in a target image. It is a schematic diagram which shows the state which extracted the whitefly candidate area | region from FIG. It is a schematic diagram which shows the state which removed the noise by an area from FIG. It is a schematic diagram which shows a mode that template matching is performed with respect to an image. It is an image figure which shows the enlarged image of a template image. It is a graph which shows scoring by template matching. It is a graph which shows scoring by an aspect ratio. FIG. 21A is a schematic diagram showing a vertical edge filter and a horizontal edge filter shown in FIG. FIG. 22A is a brightness image of a whitefly before being buried, and FIG. 22B is a schematic diagram showing an edge direction. FIG. 23A is a luminance image of a whitefly after being buried, and FIG. 23B is a schematic diagram showing an edge direction. It is a schematic diagram which shows an area | region image. It is a schematic diagram which shows the direction of the gravity center of the area | region image of FIG. FIG. 26A is a graph showing scoring of whiteflies before being buried, and FIG. 26B is a scoring according to the average value of the direction differences of whiteflies after being buried. It is a graph which shows scoring by a standard deviation. It is a flowchart which shows the procedure which identifies a stag beetle. It is an image figure which shows an acquired image. It is an image figure which shows the image of the candidate part of the stag beetle obtained with respect to the acquired image of FIG. It is an image figure which shows the background part obtained with respect to the acquired image of FIG. It is an image figure which shows a mode that a small area | region is set with respect to the acquired image of FIG. It is an image figure which shows the image which extracted the candidate of the scale insect from the acquired image of FIG. It is an image figure which shows the result of having removed by the area from FIG. FIG. 35 (a) is a hue value, FIG. 35 (b) is a hue, and FIG. 35 (c) is an image diagram showing the changed hue value. It is an image figure which shows a mode that a circumscribed rectangle is taken with respect to an image part. It is an image figure which shows a mode that a circumscribed rectangle is set with respect to the image part of a stag beetle. It is an image figure which shows a mode that a circumscribed rectangle is set with respect to the image part of noise. It is an image figure which shows a mode that a circumscribed rectangle is set without considering the attitude | position of an image part. It is an image figure which shows a mode that a circumscribed rectangle is set in consideration of the attitude | position of an image part. It is an image figure which shows a mode that a long axis and a short axis are set to the image part of FIG. Fig.42 (a) is a circular noise, FIG.42 (b) is an image figure which shows a string-like noise. It is an image figure which shows the state which matches the coordinate after rotation with the coordinate before rotation. It is a schematic diagram which shows the concept of a cubic interpolation method. It is an image figure which shows the concept of a three-dimensional polynomial approximation. It is an image figure which shows the acquired image before rotation. It is an image figure which shows the state which rotated and expanded with respect to the acquired image of FIG. It is an image figure which shows the state which arranged the rotation and the enlarged image. It is an image figure which shows the image which extracted the rotation and the expansion area. FIG. 50A is a colorful image of a stag beetle, FIG. 50B is a partially darkened state, and FIG. 50C is an image showing an entirely darkened state. 51A is an image diagram showing how candidate areas are classified with respect to FIG. 50A, FIG. 51B is FIG. 50B, and FIG. 51C is FIG. 50C. It is an image figure which shows the feature-value of each classification | category of the scale beetle of FIG. It is a schematic diagram explaining template matching. It is an image figure which shows a template image. It is a graph which shows scoring by template matching. It is an image figure which shows the state which drawn the ellipse by the moment characteristic. It is an image figure which shows the state which performed length comparison with respect to FIG. It is a graph which shows scoring by distance difference. It is a graph which shows scoring by a standard deviation. It is a graph which shows scoring by enclosure degree. It is a graph which shows scoring by a physical condition. It is a graph which shows scoring by color saturation. It is a perspective view which shows the conventional image processing counting device. 64 is a table showing threshold values used in the image processing counting device in FIG. 63. 64 is a table showing threshold values used in the image processing counting device in FIG. 63.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the embodiment described below exemplifies an insect image processing apparatus, an image processing method, an image processing program, and a computer-readable storage medium for embodying the technical idea of the present invention, The present invention does not specify insect image processing apparatuses, image processing methods, image processing programs, and computer-readable storage media as follows. Further, the present specification by no means specifies the members shown in the claims to the members of the embodiments. In particular, the dimensions, materials, shapes, relative arrangements, and the like of the components described in the embodiments are not intended to limit the scope of the present invention only unless otherwise specified, and are merely illustrative examples. Only. Note that the size, positional relationship, and the like of the members shown in each drawing may be exaggerated for clarity of explanation. Furthermore, in the following description, the same name and symbol indicate the same or the same members, and detailed description thereof will be omitted as appropriate. Furthermore, each element constituting the present invention may be configured such that a plurality of elements are constituted by the same member and the plurality of elements are shared by one member, and conversely, the function of one member is constituted by a plurality of members. It can also be realized by sharing.

  Insect image processing apparatus, image processing method, image processing program and computer-readable storage medium used in the embodiments of the present invention, and a computer for operation, control, display, and other processing connected thereto For example, IEEE1394, RS-232x, RS-422, RS-423, RS-485, USB, or other serial connection, parallel connection, or 10BASE-T, 100BASE- Communication is performed by electrical, magnetic, or optical connection via a network such as TX or 1000BASE-T. The connection is not limited to a physical connection using a wire, but may be a wireless connection using a wireless LAN such as IEEE802.1x, radio waves such as Bluetooth (registered trademark), infrared light, optical communication, or the like. Furthermore, a memory card, a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like can be used as a recording medium for exchanging data or storing settings. In this specification, the insect image processing apparatus, the image processing method, the image processing program, and the computer-readable storage medium include not only the insect image processing apparatus main body but also peripheral devices such as a computer and an external storage device. Is used to include an image processing system that combines

In addition, in this specification, the insect image processing apparatus, the image processing method, the image processing program, and the computer-readable storage medium are the image processing system itself, and the input / output, display, calculation, and communication related to the image processing. The present invention is not limited to an apparatus or method that performs other processing in hardware. Apparatuses and methods for realizing processing by software are also included in the scope of the present invention. For example, software or programs, plug-ins, objects, libraries, applets, compilers, modules, macros that run on specific programs, etc. can be incorporated into general-purpose circuits or computers to enable image generation itself or related processing. The apparatus and system also correspond to the insect image processing apparatus, image processing method, image processing program, and computer-readable storage medium of the present invention. In this specification, the computer includes a general-purpose or dedicated electronic computer, a workstation, a terminal, a portable electronic device, and other electronic devices. Further, in this specification, the program is not limited to a program that is used alone, such as a mode that functions as a part of a specific computer program, software, or service, a mode that is called and functions when necessary, an OS, and the like. It can also be used as an aspect provided as a service in the environment, an aspect that operates resident in the environment, an aspect that operates in the background, and other support programs.
(Insect image processing device)

FIG. 1 shows a block diagram of an insect image processing apparatus 100. The insect image processing apparatus 100 shown in this figure includes an image memory 2, a calculation unit 10, a storage unit 20, a display unit 30, and an operation unit 40. This insect image processing apparatus 100 is connected to the image pickup means 1, and takes in a target image picked up by the image pickup means 1 into the image memory 2. The target image captured in the image memory 2 is processed by the calculation unit 10 to perform insect count and type recognition, and display the result on the display unit 30.
(Calculation unit 10)

  The calculation unit 10 also realizes the functions of the image extraction unit 11, the information analysis unit 12, the determination unit 13, and the counting unit 15. The image extraction unit 11 selects and extracts an image corresponding to the insect as an extraction amount area from the target image captured in the image memory 2. Further, the information analysis unit 12 analyzes information about insects by image processing from the extraction region extracted by the image extraction unit 11. Furthermore, the discrimination means 13 discriminates whether or not it is an insect based on information on the insect. In addition, the counting means 15 counts the number of extraction areas that have been determined to be insects by the determination means 13 and outputs them to the output means.

  For example, the information analysis unit 12 removes the background portion of the adhesive sheet from the target image, shapes and separates each acquired object, and the determination unit determines the insect based on its size and shape. At this time, the pressure-sensitive adhesive sheet is preferably yellow or blue. By using colors that are rarely used in the greenhouse in this way, the background can be easily removed by image processing. Further, the information analysis means 12 can perform image processing on the extraction region, for example, separate the insect limbs and torso, and determine whether the insect is an insect based on the ratio of the separated limbs and torso. Alternatively, any of the total length of the insect, the trunk, the limbs, and the perimeter can be used as the feature amount. In this case, the information analysis unit 12 may be a feature amount extraction unit that extracts a feature amount to be determined from the target image by image processing.

In addition to counting insects, the determination means 13 can also include type recognition means 14 for recognizing insect types. Insect type recognition is performed with reference to a feature amount database in which feature amounts are recorded for each recognition target insect. In the insect type recognition and counting, the necessary processing capability can be reduced by using off-line analysis for analyzing the target image data once captured instead of real-time processing. Further, the information analysis means 12 performs processing using the feature amount when the resolution of the target image is high, and performs processing using template matching when the resolution of the target image is low. May be changed.
(Storage unit 20)

The storage unit 20 stores necessary information, setting contents, a database, and the like. For example, a fixed storage device such as a hard disk or an SSD can be used. In particular, it can function as a database holding means that holds a feature amount database.
(Display unit 30)

The display unit 30 functions as one form of output means for outputting the processing result by the calculation unit 10 to the outside. Here, a monitor or display using a CRT, liquid crystal, organic EL, or the like can be suitably used as the display unit 30. In particular, when the insect image processing apparatus 100 uses a form in which an insect image processing program is installed in a computer, a computer monitor can be used as the display unit 30. Further, the output means is not limited to the display unit 30, and for example, a form in which the processing result is output to the outside as data can be used.
(Operation unit 40)

The operation unit 40 is a member that performs an operation on the insect image processing apparatus 100, and a keyboard, a console, a pointing device such as a mouse, or the like can be preferably used. Moreover, when using a touch panel as the display part 30, a display part and an operation part can be made common.
(Image acquisition means)

  The image acquisition unit is a unit for acquiring a target image that may contain an insect. In the example of FIG. 1, an image sensor such as a CCD or C-MOS for capturing an image is used as the image acquisition means. Specifically, a network camera, a digital camera, an imaging unit 1 such as a mobile phone, or a scanning unit such as a scanner can be used.

  Moreover, it can also be set as the structure which acquires the imaged target image. For example, like the insect image processing apparatus 200 shown in the modified example of FIG. 2, by providing a data communication function on the image acquisition means 1B side, image pickup means such as a network camera, digital camera, mobile phone, etc. An image captured by a scanning unit such as a scanner may be captured by an image acquisition unit connected to a network via the terminal TM. In particular, by using a network connection, image data from each farmer can be concentrated in a single location for judgment, and the results can be fed back to each farmer. There is an advantage that it can be introduced only by preparing an environment in which a target image can be transmitted and received without installing a device.

Alternatively, the target image is captured via a semiconductor memory such as a USB memory in which image data of the target image is recorded or a recording medium MD such as a CD-ROM as in the insect image processing apparatus 300 shown in the modification of FIG. It can also be configured.
(Image acquisition by scanner)

  When an adhesive sheet image is acquired using a scanner, the resolution of the scanner is set to a high resolution such as 1200 dpi. In addition, since there is a possibility that the insect area cannot be copied with the standard brightness, the image is acquired with a slightly lower brightness. Note that the target image can be divided according to the image size. For example, when the size of the pressure-sensitive adhesive sheet is approximately 10 cm × 20 cm, the image size is approximately 4800 × 9600 pixels, which is too large to be processed, so that the image is divided into 16 parts. In recognition of a stag beetle, which will be described later, an image is obtained by such a scanner.

  In general, the higher the resolution of the target image to be captured, the more accurately the recognition can be done, but the amount of data increases. On the other hand, if the resolution is low, data processing and communication are lightly loaded, but accuracy is lowered. Therefore, an appropriate resolution is selected according to the required accuracy, processing capability, cost, and the like. Further, as will be described later, the image processing method used for recognition can be appropriately changed according to the resolution.

This insect image processing apparatus can be used in a system for monitoring the occurrence of minute pests using camera images. As a result, information such as the type and number of pests included in the acquired target image can be automatically acquired, work that has been performed visually by a skilled user can be automated, and the determination criteria can be made constant. Benefits are gained.
(Insect identification method)

Hereinafter, a procedure for discriminating insects using this insect image processing apparatus will be described with reference to the flowchart of FIG. Here, as Example 1, an example in which tomato leafhoppers, which are tomato pests, are extracted will be described.
(Example 1: Tomato leaf fly)

  The tomato leafhopper is 1-2 mm in length, was confirmed in Japan in 1999, has expanded its habitat mainly in western Japan, and now lives in 38 prefectures nationwide. It is mainly a pest on tomatoes, lays eggs on the tomato leaves, and when the pupae hatch, the larvae eat and destroy the leaves. Also called a leaf-eating insect, it eats and leaves the leaves and weakens the tomato tree. In addition to tomatoes, it affects a wide range of crops. On the other hand, the biological pesticide of tomato leafhopper is Isaea himebee. Isaea hinoki washi lays eggs by performing parasitic action on eggs of tomato leafhoppers laid on tomato leaves. It is possible to control tomato leafworms, because the eggs of Isaea himebee grows with the eggs of tomato leafhoppers as nutrients.

  In order to effectively utilize such biological pesticides (beneficial insects), it is necessary to input them efficiently when many pests are occurring. Humans confirmed and judged with the eyes. In addition, large-scale farms that are vast and have a large number of greenhouses require a lot of labor to observe by hand. The aging of farmers is also a big problem, and it is very difficult to observe each one. Therefore, in order to automate or support this, the insect image processing apparatus is used.

Here, a pressure-sensitive adhesive sheet is set in advance in the greenhouse of each production farmer, and a network camera is set in order to image a pest having a body length of about 1 mm attached to the pressure-sensitive adhesive sheet. The adhesive tape is a yellow plate having adhesiveness. Tomato leafhoppers are sensitive to yellow and can be easily captured by applying a sticky substance to the sheet surface. Utilizing this, the target image obtained by photographing the adhesive sheet is analyzed, and the number of tomato leafworms adhering to the adhesive sheet is counted, thereby making it possible to ascertain the timing of biopesticidal input.
(Step S4001: Acquisition of target image)

  First, in step S4001 in FIG. 4, the target image is acquired. Here, a target image is captured by a network camera connected to the network, and the image acquisition unit receives the image data by data communication.

In this example, the target image is acquired using a network camera such as a commercially available WEB camera having pan, tilt, and zoom functions. At least one adhesive sheet is installed in the greenhouse of each producer and an image of each adhesive sheet is taken. Here, the adhesive sheet is automatically searched from inside the greenhouse by panning and tilting the network camera. When an adhesive sheet is found, the image is zoomed in and images are acquired while dividing the adhesive sheet. Here, in order to obtain an image with a high resolution as much as possible, the entire adhesive sheet is not captured as a single sheet, but is acquired while dividing the adhesive sheet image using the zoom function, and then a composite image is acquired. The acquired images are sequentially transmitted, and one target image is restored by synthesizing the finally acquired images. Note that the image synthesis may be performed by the calculation means of the insect image processing apparatus, or may be configured to obtain the synthesized image on the imaging means 1 side before being captured by the image acquisition means. Here, a method of performing image composition processing by the imaging unit will be described.
(Search for adhesive sheet)

  A specific procedure will be described. First, an adhesive sheet is searched from the acquired target image. FIG. 7 shows an example of a target image obtained by imaging the adhesive sheet. Here, since the adhesive sheet is colored yellow, it is determined whether or not there is a yellow region in the target image. In this process, an HSV color space is used to extract pixels having a hue and saturation (Hue: 44-71, except for Saturation: 0-52) within a predetermined range. This parameter corresponds to the yellow area of the adhesive sheet.

  If there is no yellow region, the target image is reacquired by panning and tilting the network camera. Here, the network camera is moved so as to draw a diamond, and the acquisition of the target image and the determination of the yellow region are repeated. If this process is repeated and a yellow region is found in the target image, only the yellow region is extracted. Then, the area of the yellow region is counted, and if it is equal to or more than a certain number of pixels, it is determined that the sheet is an adhesive sheet, and the search ends.

Here, it is also conceivable that the target image includes yellow-discolored leaves or withered leaves. Even when there are a plurality of yellow regions as described above, after all the regions are once extracted, the area is counted in each region, and the maximum region is noted. And when the number of pixels of the maximum area is more than a threshold value, it determines with it being an adhesive sheet and complete | finishes a search.
(Composition of adhesive sheet image)

When the adhesive sheet is detected, the visual field is finely adjusted so that the center of gravity of the area is the center of the image. Further zoom in using the zoom function. Further, the network camera is tilted upward so that the upper end portion of the adhesive sheet enters the upper limit of the image. From there, images are acquired while tilting the network camera downward at regular intervals. And if the lower end of an adhesive sheet is detected, image acquisition will be complete | finished.
(Step S4002: Insect candidate region extraction)

Next, in step S4002 of FIG. 4, insect candidate regions are extracted using the obtained composite image as an input image. Detailed procedures will be described later.
(Recognition and counting of insects)

In step S4003, insects are recognized and counted. Here, first, the yellow region and the background portion are removed using the HSV color space as preprocessing for recognition. Here, in the recognition process, two types of methods are used according to the resolution. First, in the case of high resolution, recognition is performed focusing on the image features of tomato leafhoppers. On the other hand, in the case of low resolution, recognition is performed by template matching. Then, the recognition results are integrated into a counting result.
(Recognition using features)

Here, the recognition method when the distance between the adhesive sheet and the network camera is close and a detailed input image is obtained will be described. First, recognition using feature quantities is performed. The feature amount is defined as follows, focusing on the feature that the tomato leaf fly is a stubby and has short limbs.
(A) Insect area (b) Insect body area (c) Insect foot and wing area (d) Peripheral length of the insect (e) Template similarity and each feature are extracted For the insect area, the calculation is as follows.
(A) Regarding the area of the insect, attention is paid to the number of pixels of the entire extracted insect area. Since the image is acquired by zooming so that the size of the adhesive sheet is constant, the size can be compared by comparing the number of pixels in the insect region.
(B) The area of the body portion is extracted by paying attention to the hue in the HSV color space. This is because attention is paid to the difference in color because the limbs are thin as regions and the trunk is thick.
(C) The area of the foot and the wing part is obtained from the area of the whole (a) by the difference in the area of the body part (b).
(D) The perimeter is calculated after the binarization process is performed on the clipped image and the contraction process is performed.
(E) The similarity is calculated from a correlation value using a sample image of tomato leafhopper as a template.

  Each feature value is normalized so that the closer to the feature value obtained from the sample tomato leafhopper, the higher the score. Then, the values of the five feature values are weighted according to the importance and totaled. And when the value is above a certain value, it is determined that the insect is a tomato leafhopper. This is performed for all extracted insect candidate regions, and the number determined to be a tomato leafhopper is used as the final counting result.

An appropriate image processing method for insect recognition can be changed according to the resolution. Here, when the resolution of the target image is high, processing using the feature amount is performed, and when the resolution of the target image is low, processing using template matching is performed. Hereinafter, each method will be described.
(Recognition by high resolution image)

First, insect recognition using a high-resolution image will be described. FIG. 5 shows, as an example of high resolution, an image of a tomato leaffly captured at 80 × 80 pixels. By using such a high-resolution image, the shape, shading, etc. become clearer.
(Recognition using features)

FIG. 6 shows an example of imaging the extraction results of the four extracted feature quantities. With such a resolution, recognition using the feature amount is possible. Here, 28 insects were treated. Of the 28 animals, 8 were tomato leafhoppers and 20 were other insects. In this example, 27 animals were correctly recognized. One of the tomato leafhoppers could not be detected, but insects other than tomato leafhoppers were not recognized as tomato leafhoppers.
(Recognition by template matching)

Next, insect recognition processing based on low-resolution images will be described. FIG. 10 shows an example of an image obtained by extracting insect regions from the synthesized image of FIG. The operation of extracting detailed features from this image is difficult. In such a case, recognition is performed by correlation using a template image.
(Template image)

For the template image, an image of four typical tomato leafhoppers was used. Here, the size of one template image is 21 × 21 pixels here. Further, rotation is applied to each template image as necessary. For example, each template image is rotated to 0 °, 90 °, 180 °, and 270 °, and the logical sum of the results of template matching is obtained 16 times in total. And it normalizes so that the maximum may be set to 1.0, and this value is made into the value of the tomato leaf-likeness. Further, here, it is not determined whether each animal is a tomato leaffly, but the total value of the uniqueness values is used as the final counting result.
(Recognition by low resolution image)

An example of the result of processing the composite image shown in FIG. 8 is shown in the distribution display of FIG. This figure shows where each bar graph has a high correlation value of the template and its position. It can be seen that the correlation value increases corresponding to the extracted position in FIG. Here, 20.6969 tomato leafhoppers were output. When rounded off to 21 animals by an expert user familiar with tomato leafhoppers, 21 of 23 animals were recognized as being tomato leafhoppers, and good results were obtained.
(Imaging distance)

  Next, an example is shown in which a change in the processing result is examined by installing every 1 m from the network camera. FIG. 11 shows an example of a synthesized image that is captured and synthesized at a distance of 6 m. In FIG. 11, it can be confirmed that the background portion is mixed and only a considerably small image can be acquired. A graph that numerically represents an example of the measurement result is shown in FIG. In the graph of FIG. 12, the detection accuracy is a line, and the bar graph represents the number of animals. It can be confirmed that the detection accuracy rapidly decreases at 5 m and 6 m. This is considered to be because the object cannot be extracted from the adhesive sheet at 5 m and 6 m.

As described above, according to the insect image processing apparatus that automatically discriminates pests and counts the number thereof, it is possible to detect tomato leafworms attached to the pressure-sensitive adhesive sheet by image processing and to count the number.
(Step S4004: Output of results)

The recognition and counting results obtained as described above are output (step S4004 in FIG. 4). Here, the result is displayed on the display unit 30. It is also possible to output data to an external device.
(Example 2: whitefly)

Next, a method for identifying whiteflies using the network camera will be described with reference to the flowchart of FIG. Whitefly is a micro pest having a body length of 1.0 to 2.0 mm. Since it may occur suddenly at an appropriate temperature, it is necessary to control if several animals are found. Whiteflies are uniformly white and have elongated wings. Whiteflies cause damage to many plants such as flowers, vegetables, flowering trees, and parasites mainly on the back of leaves and prefer juice. In places affected by whiteflies, the chlorophyll is lost, and it becomes white scabs and grows worse. In addition to this symptom, soot disease may occur on the excrement and the leaves and fruits may become black. In addition, it mediates viral diseases and promotes damage in the same way as Brassica. It is difficult to control because it completely transforms into eggs, larvae, pupae and adults, and effective pesticides differ depending on the growth process. Because whitefly is attracted to yellow, the adhesive sheet is yellow.
(Step S1301: Acquisition of a whitefly target image)

First, a target image is acquired in step S1301 of FIG. This procedure is the same as the procedure in step S4001 for acquiring the target image of the tomato leafhopper described above, and detailed description thereof is omitted.
(Step S1302: Extraction of whitefly candidate regions)

  In step S1302, candidate regions that are candidates for whitefly are extracted from the target image. In order to detect whiteflies adhering to the adhesive sheet from the target image acquired from the network camera, it is necessary to first extract only the object adhering to the adhesive sheet. Therefore, first, the background portion is removed. To remove the background, a technique using color information such as HSV is generally used. However, since the hue may not be constant for each obtained image, it is difficult to perform stable detection. Therefore, a background removal method using a variable threshold that is relatively resistant to noise is used.

  FIG. 14 shows how a small area of the acquired image is set in the target image. First, as shown in FIG. 14, a small area is set around one pixel of the acquired target image, and the average luminance and the average saturation in the small area are obtained. Here, 21 × 21 pixels are set as a small region. In the case of a small area where only the background exists, the brightness and color are almost the same, so the luminance value and saturation value at the center are almost the same as the average value. However, in a region where an object other than the background exists, it is considered that there is a gap between the luminance at the center and the average luminance value because brightness other than the background exists.

Since the body of whitefly is white, the luminance value is high and the saturation value is low. However, when whiteflies are buried in an adhesive, they are not characteristic white and have a slightly darkened color. Therefore, among the pixels having a saturation value lower than the average saturation value, an area having a brightness value higher than the average brightness value is left as an area having a possibility of whitening before being buried, and is more An area having a lower luminance value is left as an area where there is a possibility of whitening after being buried, and a pixel having a luminance value with a very small difference is considered to be the background and is removed. The whitefly candidate area before being buried is displayed in red, and the whitefly candidate area after being buried is displayed in black. FIG. 15 shows an image obtained by extracting only the stag beetle candidate obtained for the acquired image shown in FIG.
(Step S1303: Selection of candidate areas for whitefly)

Next, for the region obtained in step S1302, in step S1303, a region that is roughly whitefly is selected. In the result of the image shown in FIG. 15, the background portion is removed, but there are unnecessary portions such as insects other than whiteflies. Since whiteflies are relatively small compared to other insects, it is possible to extract only whitefly candidate regions by deleting areas above a certain threshold. Further, when a dark whitefly candidate area is extracted, a shadow area may be erroneously extracted. Therefore, if a dark whitefly candidate area is adjacent to the white whitefly candidate area, it is determined that it is a shadow area and is removed. FIG. 16 shows an image obtained by removing the area and shadow from the extracted image of FIG.
(Step S1304: Extraction of feature amount of whitefly)

In step S1304, a feature amount for determining only whitefly is extracted from the obtained whitefly candidate extraction image.
(Features of whitefly)

As feature amounts for determining only whiteflies from the extracted candidate regions, three feature amounts mainly focusing on the shape are extracted.
(Template matching for whiteflies)

  Looking at the whiteflies adhering to the adhesive sheet, it can be seen that almost all the whiteflies adhere in a similar shape. Therefore, determination using template matching is performed. Template matching is a process of comparing and collating an input image and a template image to determine whether they match. FIG. 17 shows a template matching method. When the position of the template t (x, y) is detected from the input image f (x, y), t (x, y) is overlapped with the position of the point (u, v) in f (x, y). , T (x, y) and the similarity between the overlapping partial patterns of the image are measured. In the case of position detection, the image size of t (x, y) is usually smaller than f (x, y). In the range where t (x, y) is not defined, assuming that t (x, y) = 0, the similarity m (u, v) at the point (u, v) is expressed by Equation 1. The similarity m (u, v) represents the probability that the object exists at the point (u, v), and the smaller the value, the more likely the object is.

Here, in order to detect only whiteflies, matching is performed using four template images created in advance based on whiteflies. An enlarged image of the template image is shown in FIG. The actual size of the template image used here is 20 × 20 Pixel, and the minimum similarity of each whitefly candidate is obtained. Then, the obtained minimum similarity is set as the similarity of the whitefly candidate. Since the similarity is the difference between the template image and the input image as shown in Equation 1, the closer the shape is, the closer to 0. Therefore, the score match_point by template matching is scored as shown in FIG. The maximum value of match_point is 100, and approaches 0 when the similarity decreases.
(Aspect ratio)

  As a characteristic of whitefly, there is a point that there are large wings, and there is a characteristic that the image has a round shape as a whole. Therefore, determination using the aspect ratio is performed. The height of the region is Height, the width of the region is Width, and the aspect ratio Ratio is obtained by Equation 2. Since the whitefly is a feature close to a circle, the score Ratio_point is scored as shown in FIG. In the graph of FIG. 20, the horizontal axis represents the aspect ratio Ratio, and the vertical axis represents the score Ratio_point according to the aspect ratio. The maximum value of Ratio_point is 100 and takes a maximum value in the vicinity of 1 to 1.5 where the aspect ratio is close to a square.

(Direction degree)

  As a characteristic of the whitefly, the color near the center of the adhesive sheet becomes closer to the outer peripheral part as the area near the center is white. In addition, whiteflies buried in black are closer to the color of the sheet as the vicinity of the center is blacker and closer to the outer periphery. Therefore, the edge direction is obtained for all the pixels in the candidate area, and a determination is made by comparing the edge direction with the direction of the center of gravity. The direction of the edge indicates in which direction the edge is strong. The edge value Edge_H in the vertical direction of the edge is obtained by applying the vertical edge filter shown in FIG. 21A to the luminance value, and the edge value is obtained by applying the horizontal edge filter shown in FIG. The lateral strength Edge_W is obtained, and the edge direction e is obtained by Equation 3. The direction of the edge is from the pixel with high luminance to the pixel with low luminance. Therefore, the whitefly before being buried shown in the luminance image in FIG. 22 (a) tends to be directed toward the center of gravity as shown in FIG. 22 (b), and buried in the luminance image shown in FIG. 23 (a). The latter white fly tends to be directed toward the center of gravity as shown in FIG.

(Judgment by direction degree)

  After determining the direction of the edge of each pixel as described above, the determination based on the direction degree is performed. The degree of direction is compared between the direction of the edge and the direction of the center of gravity. In the case of the area image of FIG. 24, the center of gravity direction is a direction toward the center of gravity of the area as shown in FIG. The difference between the edge direction and the direction of the center of gravity is obtained, and the average value Ave_Dir and the standard deviation SD_Dir of the direction difference are obtained. Here, the scoring graph based on the average value of the direction difference of the whitefly before being buried in FIG. 26A and the whitefly after being buried in FIG. 26B, and the scoring graph based on the standard deviation are shown in FIG. , Respectively. In the case of the whitefly before being buried, the direction of the edge and the direction of the center of gravity are opposite to each other. Therefore, the score Dir_point1 is scored using Ave_Dir as shown in FIG. The maximum value of Dir_point1 is 100, and the average value Ave_Dir takes a maximum value near 180 degrees. In the whitefly after being buried, the direction of the edge and the direction of the center of gravity are the same direction, so the score Dir_point2 is scored using Ave_Dir as shown in FIG. The maximum value of Dir_point2 is 100, and the average value Ave_Dir takes a maximum value near 0 degrees. The standard deviation SD_Dir pays attention to the variation in the direction difference, and the difference in the direction is considered to have the same size to some extent. Therefore, the score Dir_point3 is scored as shown in FIG. The maximum value of Dir_point3 is 100, and the maximum value is obtained when the standard deviation SD_Dir is around 0 degrees. Then, the direction degree Dir_point is obtained by Equation 4. The maximum value of Dir_point is 100, and the angle is 180 degrees before being buried, around 0 degree after being buried, and the largest value when the standard deviation is around 0.

(Extraction of whitefly degree)

  Next, the whitefly is determined using the above-described three feature amounts. Of the three feature quantities, the score match_point by template matching is highly reliable because the shape is compared with the template. Also, the score Ratio_point by aspect ratio is high in reliability, but it cannot be said that it is effective alone for an insect whose aspect ratio is similar to whitefly. The direction degree Dir_point is highly reliable because the whitefly is a simple shape. Therefore, each feature amount is weighted to be one feature amount called a degree of whitening. The whitefly degree konaji_point is obtained by Equation 5. This whitefly degree is used to determine the whitefly.

(Visualization of whitefly level, whitefly count)

Further, the extracted whitefly candidate images are colored and visualized according to the level of whitefly. Here, visualization is performed in eight stages according to the whitefly degree. For example, using a color visualization bar, from left to right, it is divided into 8 colors: red, pink, orange, yellow, green, light blue, blue, and indigo. Coloring is done so that the degree of whitefly becomes lower. Accordingly, the image shown in FIG. 16 and the like can be colored and displayed, and the whitefly candidate extraction image can be easily visually distinguished from others.
(Count of whitefly)

  Next, the whitefly is counted from the visualized image. Here, it is not necessary to accurately determine which whitefly candidate is a whitefly, and it is only necessary to know roughly how many whitefly are in the image. The counting formula is expressed in Equation 6. Red whitefly candidates with a high degree of whitefly were counted as one animal, and when the number was less than that, 0.9 to 0 animals were counted depending on the color. The indigo color has a count of 0.

(Step S1305: Output of result)

The result of recognition and counting of whiteflies obtained as described above is output. Thus, it was confirmed that it is possible to automate counting of whiteflies, which are minute pests.
(Example 3: Extraction of stag beetle candidate)

Furthermore, recognition of a stag beetle will be described as Example 3. Since the stag beetle has a very small body length, a scanner that scans and reads the surface of the adhesive sheet is used as the imaging means 1 for acquiring the target image, instead of a network camera. Hereinafter, a method of removing the background portion from the target image obtained from the scanner, extracting only the object region, and classifying the extracted candidate region will be described based on the flowchart of FIG.
(Shoebug scale)

The stag beetle is a small pest having a body length of 0.7-0.9 mm for male adults, 1.1-1.3 mm for female adults, and about 0.3 mm for larvae. As a characteristic of the stag beetle, adult males have orange-red wings, females have a pale yellow to orange-yellow elliptical shape, and larvae have a pale orange-like color with a nearly circular ellipse. The stag beetle settles on the tea branch and trunk, inserts a mouth needle and sucks the sap, causing damage to the tea tree. The stems that have been damaged by the insects of the stag beetle are weakened, the growth becomes poor, the leaf growth worsens, and the new shoots die. In recent years, it also causes damage to trees such as cherry, ume, and peach. The occurrence of stag beetle occurs three times in May, July and September, and is likely to occur in large quantities when dried at high temperatures. In addition, stag beetles are covered with shells except for a short period of the larval hatching period, and are considered to be difficult-to-control pests that are difficult to be applied to drugs and difficult to control with pesticides. The natural enemy insects are Chibitobachibachi and Salmensjabachi. Here we deal with the larvae stag beetle.
(Step S2801: Acquisition of target image of stag beetle)

First, in step S2801 in FIG. 28, a target image of a stag beetle is acquired. Here, the larvae of the scale insects are as small as 0.3 mm in length, so the adhesive sheet image is scanned using a scanner instead of the network camera used for tomato leafhoppers and whiteflies. Here, the resolution of the scanner is set to 1200 dip.
(Step S2802: Extraction of candidate area of stag beetle)

  Next, in step S2802 of FIG. 28, candidate areas that are candidates for stag beetle are extracted from the target image. Here, it is necessary to extract only the object attached to the adhesive sheet in order to detect the stag beetle attached to the adhesive sheet from the target image acquired from the scanner. Therefore, first, the background portion is removed. In order to remove the background, a method using color information such as HSV is generally used as described above. However, since the hue may not be constant for each obtained image, it is difficult to perform stable detection. In addition, there is a method such as the background removal method using the variable threshold described above, but there are cases where many regions other than insects are extracted. Therefore, here, a method using a method using HSV color information and a background removal method using a variable threshold is used. In this embodiment, the user encloses the position of the insect with a blue marker or the like by visual observation on the adhesive sheet of the stag beetle before the acquisition at the agricultural test site or the scanner.

First, a candidate portion of a stag beetle is acquired for the acquired target image. As a method for selecting a candidate portion, it is acquired by HSV color information. The range to be acquired is that the body color of the stag beetle will darken over time, so that the insects that have passed over time are extracted, so the pixel to be selected is a part slightly close to red yellow and a black part as candidate parts select. FIG. 30 shows an image of the candidate portion of the stag beetle obtained for the acquired image shown in FIG. Also, a part to be used as a background part for the acquired image is acquired. This is because, since the insects are surrounded by blue circles in the above-described image, the blue portion which is not so much in nature is excluded and used as the background portion. As a portion used as a background portion, an area excluding a portion surrounded by blue is selected as a background. The background portion obtained for the acquired image is shown in FIG.
Then, a small region is set around the acquired background portion around one pixel of the obtained candidate portion as shown in FIG. 32, and the average luminance in the small region is obtained. Here, 51 × 51 pixels are set as a small area. In the case of a small area in which only the background exists, the brightness is substantially the same, so that the brightness at the center is substantially the same as the average brightness value. However, in a region where an object other than the background exists, it is considered that there is a gap between the luminance at the center and the average luminance value because brightness other than the background exists. Therefore, the pixel having a luminance smaller than the average luminance value is left to be considered as an object other than the background, and the pixel having a luminance value with a very small difference is considered to be the background, and is removed. FIG. 33 shows an image obtained by extracting only the stag beetle candidate obtained for the acquired image shown in FIG.
(Step S2803: Selection of candidate area for stag beetle)

Next, in step S2803, a region that seems to be a stag beetle is roughly selected from the region obtained in step S2802 of FIG. In the result of the image shown in FIG. 32, the background portion is removed, but unnecessary portions such as insects and bubble portions other than the stag beetle exist. For this reason, areas and color information are compared to remove those that are clearly different from stag beetles.
(Selection by area)

Labeling is performed on the image from which the background has been removed, and the area value of each object is obtained. Since the size of the stag beetle is small compared to other insects, it is considered that objects with large area values are other insects and bubbles and are not stag beetles, so they are removed and noise is detected for objects that are too small Remove it because it is possible. FIG. 34 shows the result of removal by area.
(Selection with HSV average value)

Next, in order to pay attention to the difference in color of each object, the average value of each HSV of each object is obtained. At this time, when the average value of the hue is obtained, the hue value is represented by a value from 0 to 360 as shown in FIG. 35 (a), but the actual hue is circular as shown in FIG. 35 (b). Therefore, when the red value is set as the reference value (0), a value different from the actual hue average value is obtained and may be erroneously removed. Therefore, in this embodiment, as shown in FIG. 35C, the average value of the correct hue can be obtained by setting the hue of the removed blue portion as the reference value (0). Since the color of the stag beetle is close to red or orange, an object whose hue average value is not near red is removed. In addition, the stag beetle before darkening is colored, and since the colored portion remains after darkening, the removal is performed even when the average saturation value is low.
(Selection by density)

  Subsequently, a certain amount of objects are selected from the shape of each object. In many cases, the noise at the boundary between bubbles is elongated and curved. Therefore, determination using the density of each object is performed. As shown in FIG. 36, the density is obtained by taking the circumscribed rectangle of the object, the height of the object is high, the width of the object is width, and the area is area.

The density Density approaches 1 as it becomes dense, and approaches 0 as it becomes sparse. As shown in FIG. 37, the density of the stag beetle is dense, and in the case of the boundary part of the bubble, the density is often sparse as shown in FIG. 38. Therefore, the density Density is 0.45 or less. In the case of, noise is determined and removed.
(Selection by aspect ratio)

  A characteristic of the shape of the stag beetle is that it is rounded and elliptical as a whole. Therefore, determination using the aspect ratio is performed. However, in the method of obtaining the aspect ratio from the circumscribed rectangle as shown in FIG. 39, the correct aspect ratio cannot be obtained when the object is shown obliquely, and the stag beetle may be erroneously removed. Therefore, here, a circumscribed rectangle is acquired in consideration of the direction of the object shown in FIG. 40, and the aspect ratio is acquired. As a method for obtaining the orientation of an object, the orientation is obtained using a moment feature. The moment in the image satisfies the condition that the straight line with the smallest secondary moment passes through the center of gravity. The vertical coordinate i, the horizontal coordinate j, and the moments in the vertical direction and the horizontal direction are defined as the p-th order and q-th order moments, respectively, and the angle of the axis passing through the center of gravity is obtained by Equations 8 and 9.

  The long axis Long_Axis and the short axis Short_Axis are obtained from the obtained angles. As shown in FIG. 41, in the method of obtaining the length of the axis, if the long side and the short side of the circumscribed square with the angle θ as a reference are the long axis Long_Axis and the short axis Short_Axis, respectively, the aspect ratio Ratio is obtained by Equation 10. Since the shape of the stag beetle is rounded and oval as a whole, an object close to a circle as shown in FIG. 42 (a) or an elongated object as shown in FIG. 42 (b) Is considered not to be a stag beetle, it removes an object whose Ratio is not within the range of 1.2 to 2.5.

(Step S2804: Image enlargement / rotation / re-extraction)

Next, a method of rotating and enlarging the range in order to make the stag beetle candidate area a clear image in the same direction in step S2804 in FIG. 28 will be described. Here, since the enlargement is performed on the acquired image, it needs to be extracted again.
(Enlarge / Rotate image)

  Since the extracted stag beetle candidate region is in various postures, it is difficult to accurately extract the feature value as it is. In addition, since the extracted stag beetle candidate region has only information of several tens of pixels, it is difficult to accurately extract the feature amount. Therefore, in order to make the direction of the stag beetle candidate area constant and to obtain a clear image, the image using the rotation and sub-pixel information is performed on the stag beetle candidate area. First, using the angle θ of the region axis by the moment obtained above, the coordinates (x, y) after rotation are returned to the coordinates (x ′, y ′) before rotation as shown in FIG. Need to search. The distance Dist between the barycentric coordinates (xg, yg) and the rotated coordinates (x, y) is obtained using Equation 11, and the inclination ψ of the rotated coordinates (x, y) is obtained using Equation 12. Then, the coordinates (x ′, y ′) before the rotation before the rotation are obtained using the equations 13 and 14.

  Once the corresponding points are obtained, pixel interpolation is performed to create an enlarged image. As a method of interpolation, there is a nearest neighbor method. However, since this method gives the density value of the pixel closest to the pixel to be interpolated, it is not suitable for clear image interpolation. Therefore, interpolation is performed using a cubic interpolation method as an interpolation method used here. In the cubic interpolation method, as shown in FIG. 44, interpolation is performed using a cubic equation using density values of 16 grid points around coordinates (x ′, y ′) for obtaining density values. As shown in FIG. 45, the cubic equation uses an approximate expression of the function sin πx / (πx) that appears in the sampling theorem, and theoretically the concentration of the coordinates (x ′, y ′) can be almost completely restored. It is.

  Here, interpolation by a cubic equation is performed using the density values of the coordinates (x ′, y ′) before the rotation by using the cubic interpolation using the equations 15 and 16. With cubic interpolation, clearer subpixel information can be obtained, and accurate feature extraction can be performed. FIG. 47 shows an image obtained by rotating / enlarging the acquired image before the enlargement / rotation shown in FIG.

(Region re-extraction)

Since the created rotated / enlarged image is obtained with respect to the acquired image, it is necessary to extract a region in the rotated / enlarged image. The re-extraction is performed in the same manner as the above-described method, and the region is re-extracted. At this time, the rotated / enlarged image is created by arranging the regions as shown in FIG. 48, but if another region exists nearby, it may be erroneously extracted. Therefore, unnecessary areas can be deleted by excluding labels other than the labels in the vicinity of the center of gravity of each rotated / enlarged portion after labeling. FIG. 49 shows an image obtained by extracting the region of the rotated / enlarged image in FIG.
(Step S2805: Classification of stag beetle)

Next, in step S2805 in FIG. 28, the stag beetle is classified. As for the stag beetle, the body darkens when time passes after adhering to the adhesive sheet, and the characteristics change before and after darkening as shown in FIGS. 50 (a) to (c). Therefore, here, the stag beetle candidate area is classified into a plurality of types. For a while after adhering to the pressure-sensitive adhesive sheet, there is a characteristic of the original colorful stag beetle shown in FIG. Therefore, a candidate area having a high average luminance value or high average saturation value is considered to be a colorful stag beetle, and is therefore classified as a colorful stag beetle candidate area. Further, when a certain amount of time has elapsed, as shown in FIG. 50 (b), there is a characteristic that a part of the body of the stag beetle darkens. Therefore, it is considered that the candidate area where the average luminance value and the average saturation value are not so high and the average low luminance value, which is an average value of only the low luminance value, is low, may be a partly dark stag beetle. This is classified as a stag beetle candidate area that is partially darkened. In addition, as time passes further, as shown in FIG. 50 (c), there is a characteristic that the entire body of the stag beetle darkens. Therefore, a candidate area having a low average luminance value or a low average saturation value is considered to be an all-black stag beetle, so it is classified as an all-black stag beetle candidate area. As shown in FIG. 51, the classified variegated scale insects are displayed with the colorful stag beetle candidate area in green, the partially stag beetle candidate area in red, and the entire stag beetle candidate area in blue as blue. To do.
(Step S2806: Extraction of feature amount of stag beetle)

Further, in step S2806 in FIG. 28, the feature amount of the stag beetle is extracted. Hereinafter, a procedure for extracting a feature amount for determining only a stag beetle from a stag beetle candidate extraction image obtained in the above-described steps will be described. First, template matching and ellipticity will be described as feature quantities. Next, one feature amount for each classification will be described. Furthermore, the quasi-degree of integrating each feature amount will be described. Finally, a method for counting stag beetles will be described.
(Characteristics of stag beetle)

A feature amount for determining only the stag beetle from the extracted candidate area will be described. As described above, the stag beetle has been classified in step S2805 of FIG. 28, and therefore, as shown in FIGS. 52 (a) to (c), two common feature quantities (here, match degree and ellipticity) and the respective classifications. One feature amount is different (in the colorful example shown in FIG. 52 (a), the degree of enclosure is shown, in FIG. 52 (b) is partly black, the body is shown in FIG. 52 (c), and in the whole example shown in FIG. 52 (c) is black. A total of five types of feature values (color saturation) will be described.
(Template matching)

  Looking at the stag beetle adhering to the adhesive sheet, it can be seen that almost all of the variegated scale insects are attached in a similar shape. Therefore, determination using template matching is performed. Template matching is a process of comparing and collating an input image and a template image to determine whether or not they match. FIG. 53 shows a template matching method. When the position of the template t (x, y) is detected from the input image f (x, y), t (x, y) is overlapped with the position of the point (u, v) in f (x, y). , T (x, y) and the similarity between the overlapping partial patterns of the image are measured. In the case of position detection, the image size of t (x, y) is usually smaller than f (x, y). In the range where t (x, y) is not defined, assuming that t (x, y) = 0, the similarity m (u, v) at the point (u, v) is expressed by Equation 17. The similarity m (u, v) represents the probability that the object exists at the point (u, v), and the smaller the value, the more likely the object is.

Here, in order to detect only stag beetles, matching is performed using template images created in advance based on a plurality of stag beetles. FIG. 54 shows an enlarged image of the template image. The actual size of the template image is 36 × 54 Pixel, and nine types of template images having slightly different shapes are used. The minimum similarity of each stag beetle candidate is obtained from this template image. Then, each obtained minimum similarity is set as the similarity M of the stag beetle candidate. Since the similarity M is the difference in density value between the template image and the input image as shown in Equation 17, the closer the shape is, the closer to 0. Therefore, the score match_point by template matching is scored as shown in FIG. In FIG. 55, the horizontal axis indicates similarity M, and the vertical axis indicates score match_point by template matching. The maximum value of match_point is 100, and the maximum value is obtained with a similarity of 0, which is the maximum match.
(Ellipticity)

  As an external feature of the stag beetle, it has a shape close to an ellipse. Therefore, the ellipses of the outer periphery are compared using ellipses. The ellipticity is a feature value that measures how close a region is to an ellipse. First, in order to obtain an ellipse to be compared with the outer periphery of the region, the orientation and axis of the candidate region are obtained from the moment feature. Although the candidate areas are aligned in the same direction as described above, the inclination, major axis, and minor axis of the area are obtained again by the moment feature because there is a fine displacement of the pixels. From the inclination and the length of the axis, an ellipse is acquired and compared with the outer circumference as shown in FIG. For comparison with the outer periphery, as shown in FIG. 57, a distance difference Diff_dist between the outer periphery and the ellipse in the same direction with respect to the center of gravity is obtained. Based on the distance difference Diff_dist, it is determined whether there is a deviation between the ellipse and the region. Diff_dist is obtained up to 360 degrees at intervals of 5 degrees, and the average distance difference Ave_dist is obtained from the total distance difference Total_dist of each angle and Equation 18. Further, a standard deviation SD_dist of the distance difference is obtained. Based on the standard deviation SD_dist of the distance difference, it is determined whether there is a steep deviation between the ellipse and the outer periphery.

  Scoring is performed based on the obtained average distance difference Ave_dist and the standard deviation SD_dist of the distance difference. Since the stag beetle has a shape close to an ellipse, the score Ave_point based on the average distance Ave_dist is scored as shown in FIG. Further, since the shape of the stag beetle is close to the shape of a simple ellipse, the score SD_point based on the standard deviation SD_dist of the distance difference is scored as shown in FIG. Then, the ellipticity oval_point is obtained from Equation 19 using Ave_point and SD_point. The highest point of the ellipticity oval_point is 100 points, and the higher the score is, the more the region having a simple elliptical shape is.

(Degree of enclosure)

  The colorful stag beetle has a deeper color and higher saturation than the outside. Therefore, determination using the degree of enclosure is performed. The degree of enclosure is a feature amount that measures whether low saturation values are distributed so as to surround pixels with high saturation values. If the centroid point obtained from the high saturation value in the candidate area is (X, Y) and the centroid point obtained from the other pixels is (x, y), the distance d between the centroid points is expressed by Equation 20. Since the stag beetle has a pixel with a high saturation value at the center of the body, it is considered that the centroid point of the high saturation value coincides with the centroid point of other pixels. Using the distance d, the surrounding degree surround_point is scored as shown in FIG. The maximum value of surround_point is 100, and approaches 0 as the distance d increases.

(Health)

The stag beetle candidate area that is partially dark may be partially black. Therefore, the determination using the physical strength is performed. Here, the physical condition is a feature amount that measures whether or not pixels having low luminance values are gathered in one place and look like a body. As shown in FIG. 61, scoring is performed on the body_point from the density B_Density only in the low luminance value in the candidate area. The maximum value of body_point is set to 100, and it approaches 0 as the density B_Density decreases.
(Color saturation)

The stag beetle candidate area that is entirely dark is originally colored but darkened over time. Therefore, determination using color saturation using color information is performed. Here, the color saturation is a feature amount to be obtained by using a colored portion. The average high saturation value High_Sat of the high saturation values in the region is obtained, and the color saturation color_point is scored as shown in FIG. The maximum value of color_point is set to 100, and the region with no color approaches 0.
(Step S2807: Extraction of quasi-degree)

  Further, in step S2807 in FIG. 28, the quasi-degree is extracted. Here, the stag beetle is determined using the five feature amounts described above. Of the three feature quantities, the score match_point by template matching is highly reliable because the shape is compared with the template. In addition, the score oval_point based on the ellipticity is highly reliable because the stag beetle has a simple shape. Surround_point and body_body_point based on the degree of enclosure are low in reliability because they have few pixels for small noise. The color saturation color_point has low reliability because it may be colored in noise. Therefore, each feature value is weighted to be one feature value called the quasi-degree. The quasi-degree kuwashiro_point is expressed by Equation 21. The stag beetle is determined using this stagnation degree. At this time, since the possibility that the ellipticity is extremely low is not a stag beetle is very high, it is set to zero.

(Step S2808: Visualization of degree of stagnation and counting of scale insects)

  Further, in step S2808 of FIG. 28, the degree of stagnation is visualized and the number of stag beetles is counted. Here, according to the level of the stagnation degree, the extracted stag beetle candidate image is colored and visualized. Visualization is performed using a visualization bar that is color-coded in eight levels according to the degree of quasi. In the visualization bar, the quasi-degree is higher as it goes to the left of the bar, and the quasi-degree is lower as it goes to the right.

  Next, the stag beetle is counted from the visualized image. Here, it is not necessary to accurately determine which stag beetle candidate is a stag beetle, and it is only necessary to know roughly how many stag beetles are in the image. Here, the counting formula is represented by Formula 22. A red stag beetle candidate with a high stagnation degree is counted as one animal, and if it is less than that, it is counted as 0.9 to 0 animals depending on the color.

(Step S2809: output)

  Finally, the obtained result is output.

The results of recognition of whiteflies and stag beetles using the above evaluation method will be described below.
(Experiment environment)

  For whiteflies, an adhesive sheet (trade name: Horibar) was installed at a position 2 meters from the camera, and images were acquired from a network camera. The experiment was conducted under a white fluorescent lamp, and the brightness and white balance of the network camera were fixed. This is to prevent the camera from changing the brightness and white balance automatically when the zoom is repeated and the entire image becomes dark.

For the stag beetle, images were acquired using a scanner. The resolution was fixed at 1200 dpi, and the brightness of the image was fixed at -20. The reason for fixing the resolution is to prevent the size of the scale insects extracted when performing template matching from becoming unstable. The lightness is fixed at −20 in order to prevent the possibility that the initial lightness of the scanner will not appear in the acquired image because the stag beetle is too small.
(Experimental result)

  In the experiment with whitefly, one adhesive sheet was divided into 14 sheets and photographed. In the experiment on the scale insect, a single adhesive sheet was divided into 16 images. Table 1 shows the whitefly count results in each adhesive sheet, and Table 2 shows the stag beetle count results in each adhesive sheet.

  In each pressure-sensitive adhesive sheet, 58 whitefly were present when the whitefly was counted visually. The total number extracted as whitefly candidates by the above method was 145, and the result of the whitefly count was determined to be 73, and good results were obtained.

  On the other hand, when the stag beetle was counted visually on the adhesive sheet, there were 915 stag beetles. The total number extracted as candidates for stag beetles by this method was 861, and as a result of counting stag beetles, it was determined that there were 889 stag beetles. As an overall evaluation, the extraction rate was about 95% and the stag beetle count was almost the same as the overall number of stag beetle, indicating excellent results.

  As described above, whiteflies and stag beetles can be counted from the acquired target image. In other words, by creating a background-removed image that removes the background portion of the adhesive sheet from the acquired image, and further extracting the whitefly degree and stagnation degree from the feature amount, the number of whitefly and stag beetle present in the image Counting can be performed automatically. In this way, by using the insect image processing device, it is possible to detect pests and count their occurrence, so this information can be used to grasp the optimal timing of natural enemy insects, pesticides, etc. It becomes possible. This eliminates the need for the grower to visually check the pests attached to the adhesive sheet one by one as in the prior art, and can greatly reduce the burden. As a result, it can be expected to effectively use biological pesticides instead of chemical pesticides.

  Insect image processing apparatus, image processing method, image processing program, and computer-readable storage medium of the present invention can be used for counting insects captured in a greenhouse, determining the type of insects, or determining insects in food factories and pharmaceutical factories. It can be suitably used for purposes such as type determination.

DESCRIPTION OF SYMBOLS 100, 200, 300 ... Bug image processing apparatus 1 ... Imaging means 1B ... Image acquisition means 2 ... Image memory 10 ... Calculation part 11 ... Image extraction means 12 ... Information analysis means 13 ... Discrimination means 14 ... Type recognition means 15 ... Counting Means 20 ... storage unit 30 ... display unit 40 ... operation unit TM ... terminal MD ... recording medium

Claims (17)

An image processing apparatus for specifying information about insects included in a captured image,
Image acquisition means for acquiring a target image that may contain insects;
Image extraction means for extracting an image corresponding to an insect from the target image acquired by the image acquisition means;
Information analysis means for analyzing information about insects by image processing from the extraction area extracted by the image extraction means;
Based on the information on the insect analyzed by the information analysis means, a determination means for determining whether it is an insect,
A counting means for counting the number of extraction regions determined to be insects by the determination means;
Output means for outputting the result counted by the counting means as the number of insects;
An insect image processing apparatus comprising: The insect image processing apparatus according to claim 1, further comprising:
It has database holding means that holds the feature database,
An insect image processing apparatus characterized in that the discrimination means performs discrimination using a feature quantity with reference to a feature quantity database held in the database holding means. The insect image processing apparatus according to claim 1 or 2,
An insect image processing apparatus, wherein the discrimination means recognizes the type of insect. An insect image processing apparatus according to any one of claims 1 to 3,
The information analyzing means is
If the resolution of the target image is high, perform processing using the feature amount,
An insect image processing apparatus that performs processing using template matching when the resolution of a target image is low. An insect image processing apparatus according to any one of claims 1 to 4,
The insect image processing apparatus characterized in that the feature extracted by the information analysis means includes any one of the total length, body, limbs, and perimeter of the insect as a feature amount. An insect image processing apparatus according to any one of claims 1 to 5,
When the discrimination means recognizes the type of insect, the same type of insect is classified into a plurality of partially different surface colors,
An insect image processing apparatus characterized by assigning different feature amounts according to the color of the classified insects and performing discrimination using the different feature amounts. The insect image processing apparatus according to claim 6,
An insect characterized in that the discrimination means assigns a different feature amount according to the color of the classified insect, while assigning a common feature amount, and makes a discrimination using the plurality of feature amounts. Image processing apparatus. An insect image processing apparatus according to any one of claims 1 to 7,
An insect image processing apparatus characterized in that the insect to be recognized is any one of thrips, whiteflies, and a scale insect. An insect image processing apparatus according to any one of claims 1 to 8,
The insect image processing apparatus, wherein the image acquisition means includes an image sensor for capturing an image. An insect image processing apparatus according to any one of claims 1 to 9,
An insect image processing apparatus, wherein the image acquisition means is a network camera. An insect image processing apparatus according to any one of claims 1 to 9,
An insect image processing apparatus, wherein the image acquisition means is a scanner. An insect image processing apparatus according to any one of claims 1 to 11,
An insect image processing apparatus, wherein the target image is an insect image adhered to an adhesive sheet. An insect image processing apparatus according to any one of claims 1 to 12,
The information analyzing means is
Remove the background part of the adhesive sheet from the target image,
Perform shaping and separation of each acquired object,
An insect image processing apparatus characterized in that the determination means determines an insect based on its size and shape. An insect image processing apparatus according to any one of claims 1 to 13,
The insect image processing apparatus, wherein the pressure-sensitive adhesive sheet is yellow or blue. An image processing method for specifying information about insects included in a captured image,
Obtaining a target image that may contain insects;
Extracting an image corresponding to an insect from the acquired target image;
A process of analyzing information about insects by image processing from the extracted extraction area;
A step of determining whether or not an insect is based on information about the extracted insect;
Counting the number of extracted areas identified as insects;
Outputting the counted result as the number of insects;
An insect image processing method comprising: An image processing program for specifying information about insects included in a captured image,
A function to acquire target images that may contain insects;
A function to extract images corresponding to insects from the acquired target images;
A function to analyze information about insects by image processing from the extracted extraction area,
A function to determine whether it is an insect based on information about the extracted insect,
A function for counting the number of extracted areas identified as insects;
A function to output the counted results as the number of insects;
An insect image processing program characterized by causing a computer to realize the above.   A computer-readable recording medium storing the program according to claim 16.