JP2015132994A - Program, image recognition device, and image recognition method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a program for improving recognizing accuracy in the case of using filtering.SOLUTION: The program is characterized to allow a computer to execute: an image data acquisition step S10 for acquiring second image data; a search object extraction step S20 for extracting a living thing in a habitat environment including an imaging condition under which the second image data have been picked up as a search object; a recognition step S30 for acquiring the featured value of the living thing from the second image data, and for determining first image data of a living thing having a featured value whose similarity to the featured value of the living thing as the search object extracted by the search object extraction step is equal to or more than a threshold; and an update step S80 for, when the first image data of the living thing having the featured value whose similarity is equal to or more than the threshold are determined in a state where there exists one or more habitat environments which are not used for extracting the search object by the search object extraction step, updating the habitat environment of a data storage part which is not used for extracting the search object under the imaging condition.

Description

  The present invention relates to a program for recognizing a living organism from a captured image.

  Since digital cameras and devices with cameras have become widespread, there are increasing opportunities for users to image a desired subject. There are various subjects, but there are many cases in which a user wants to know what is captured. For example, if the names of captured plants, flowers, insects, animals, etc. can be known, more detailed information can be searched.

  In order to meet such a demand, there is an application that returns a flower name from a flower image. In such an application, when a flower image is transmitted to the server, the server transmits a candidate flower name or image to the application.

  However, the accuracy of flower recognition by these applications is not always good. In order for an application to recognize the type of flower from a flower image, it is conceivable to perform some image processing. However, it is known that there are over 19,000 types of flowers, and many of them have very similar shapes and colors, making it difficult for the server to identify the same flowers as the images taken by the server. There are many cases.

  Therefore, in the conventional application, the user can directly input the flower color, shape, size, number of petals, etc., and filter the flower information database based on such information to reduce the search target and improve the recognition accuracy. It's striking. However, in this method, work performed by the user occurs, and if the user makes an input mistake, the recognition accuracy is lowered.

  In order to improve the recognition accuracy, the application automatically acquires position information with GPS etc. when capturing a flower image, further acquires the date and time information, and searches the flower information database for the search area and the image capturing time. A technique for filtering has been devised (for example, see Patent Document 1).

  However, there is a problem in that the search accuracy may be reduced by filtering the search target in the imaging region / imaging time. That is, in recent years, changes in habitats such as plants and flowering seasons have occurred due to climate changes such as global warming. In addition, alien species that should not originally live in the area often move and settle between each other.

  Therefore, there has been a possibility that a flower that originally did not inhabit the imaging area or imaging time may inhabit the imaging area or imaging time. For this reason, when a user captures such a flower, the application narrows down the flower information database according to the image capturing area and the image capturing time, so that a situation in which a correct recognition result cannot be obtained occurs.

  On the other hand, it is preferable to use filtering because not performing filtering at all causes a decrease in recognition accuracy and an increase in recognition time.

  An object of this invention is to provide the program which improves the recognition precision at the time of using filtering in view of the said subject.

  The present invention provides a computer with an image data acquisition step of acquiring second image data in which a living organism is imaged, a step of accessing a data storage unit in which the first image data and the habitat environment of the organism are registered, and , A search target extraction step for extracting from the data storage unit the organisms in the habitat environment including the imaging conditions under which the second image data was captured, and acquiring the feature quantities of the organisms from the second image data And a recognition step for determining first image data of a living organism having a similarity with a feature amount of the searching target organism extracted by the search target extracting step, and the search target in the search target extraction step. In the state where there is one or more habitats that are not used for extraction, the recognition step determines the first image data of an organism whose similarity is equal to or greater than a threshold value. If it provides a program for causing to execute, and updating step of updating the habitat of the data storage unit that is not used for extraction of the search target in the imaging condition.

  A program for improving the recognition accuracy when filtering is used can be provided.

It is an example of the figure explaining the outline of the recognition system of this embodiment. It is an example of the schematic block diagram of the recognition system of this embodiment. It is an example of the hardware block diagram of the communication terminal which concerns on this embodiment. It is an example of the hardware block diagram of the database server which concerns on this embodiment. It is an example of the functional block diagram of a recognition system. It is an example of the figure explaining the structure of DB for flower search. It is an example of the figure explaining flower information DB. It is an example of the functional block diagram of an image recognition process part. It is an example of the figure explaining extraction of the feature of a flower. It is an example of the figure explaining the search object of DB for flower search when the filters 1-3 are applied. It is an example of the figure explaining update of DB for flower search. It is an example of the flowchart figure which shows the operation | movement procedure of a recognition system. It is a figure which shows the example of a display on the display of a communication terminal. It is a figure which shows an example of the detailed information displayed on the display. It is a figure which shows the example of a display of the flower habitat environment using an update history. It is an example of the schematic block diagram of the recognition system of a server client system.

Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.
FIG. 1 is an example of a diagram illustrating an outline of a recognition system according to the present embodiment. As shown in FIG. 1A, the recognition system includes a flower search DB (database) 16, one or more filters, and an image recognition processing unit 14. First, a case where the recognition result is good will be described with reference to FIG.
(1) The image recognition processing unit 14 first performs an image recognition process that searches all the filters 1 to 3 for a flower similar to the captured image from the flower search DB 16. Filters 1 to 3 are filters that filter the search target of the flower search DB 16 according to the imaging conditions (having the imaging location and the imaging timing). By filtering according to the imaging conditions, it is possible to exclude flowers that do not live at the imaging location and imaging time from the search target.
(2) When the recognition rate is sufficiently high (when it is equal to or higher than the threshold value), the image recognition processing unit 14 determines that the search result is correct and provides it to the user.

  In this way, even when searching through all filters 1 to 3, if a flower with a high recognition rate matches the search, the same flower as the imaged flower does not leak from the search target by filters 1 to 3. It can be determined that the search was successful.

Next, a case where the recognition result is not good will be described with reference to FIG.
(1) The image recognition processing unit 14 first searches the flower search DB 16 through all the filters 1 to 3.
(2) It is assumed that the image recognition processing unit 14 has not obtained a recognition result with a recognition rate equal to or greater than a threshold value.
(3) In this case, the image recognition processing unit 14 releases the filters 1 to 3 in order. The order of cancellation is the order of filters with less influence. Since there are few ways to increase the number of recognition objects, the increase in recognition time can be suppressed.
(4) For example, when the filters 1 and 2 are removed, it is assumed that a search result having a recognition rate equal to or higher than a threshold value is obtained. Therefore, it is considered that a correct recognition result was obtained. In this case, it is considered that the habitat filtered by the filters 1 and 2 does not correctly represent the current flower habitat. However, in the present embodiment, it can be seen that by removing the filter, a correct search result can be obtained even when an image of a flower that does not originally live is captured due to weather changes or alien species.
(5) When a correct search result is obtained by removing the filters 1 and 2, the recognition system adds the habitat environment removed by the filters 1 and 2 to the flower search DB 16. For example, it is assumed that the habitat environment and imaging conditions of the flowers for the DB 16 for flower search are as follows: “Habitat: Honshu”, “Habitat: Spring”
“Hokkaido is the place where the flowers were imaged”, “The time when the flowers were imaged is summer”
In this case, the recognition system determines the habitat environment of this flower in the DB 16 for flower search.
Update the habitat environment to “Habitat: Honshu, Hokkaido” & “Habitat: Spring, Summer”.

  Therefore, the flower search DB 16 can learn a new habitat environment for flowers whose current habitat environment is not registered in the flower search DB 16 because it is a climate change or an alien species. This makes it possible to create a more realistic habit map instead of the habitat map on the pictorial book.

  Hereinafter, a description will be given by taking a flower as an example of a living thing. In addition to a flower, any living thing that is imaged such as a plant, an insect, a mushroom, or a moss can be a recognition target of the present embodiment.

[Configuration example]
FIG. 2 shows an example of a schematic configuration diagram of the recognition system 500 of the present embodiment. The recognition system 500 includes a communication terminal 100 and a database server 200 that are communicably connected via a network 300. The network 300 is, for example, a LAN, a WAN (Wide Area Network) to which a plurality of LANs are connected, the Internet, or a combination thereof. Further, a part or the whole of the network 300 may be constructed by a mobile phone network, a wireless LAN network, or the like.

  The communication terminal 100 is, for example, a mobile phone, a smartphone, a PDA (Personal Digital Assitant), a tablet terminal, a notebook PC (Personal Computer), a tablet PC, a music player, a car navigation system, or the like. The communication terminal 100 only needs to have a communication function via the network 300. Although it is preferable that the camera is mounted, it is also possible for the communication terminal 100 to acquire an image that has already been captured and transmit it to the database server 200.

  The database server 200 is an information processing device called a server, a PC, a computer, or the like. Further, NAS (Network Attached Storage) or file server whose main function is to store data may be used for the database server 200.

  The communication terminal 100 acquires a flower information DB 15 and a flower search DB 16 (which is an example of a data storage unit in claims), which will be described later, from the database server 200 and recognizes a flower similar to the imaged flower. I do. Then, the recognition result is displayed on a display or the like. As will be described later, the communication terminal 100 may transmit an image, and the database server 200 may transmit an image similar to the captured image recognized from the flower search DB 16 to the communication terminal 100. The communication terminal 100 is an example of the biological recognition device in the claims.

  FIG. 3 is an example of a hardware configuration diagram of the communication terminal 100 according to the present embodiment. The communication terminal 100 includes a CPU 101, ROM 102, RAM 103, SSD (Solid State Drive) 105, media drive 107, operation button 108, power switch 109, telephone communication unit 110, network I / F 111, camera 112, microphone 113, speaker 114, A display I / F 115, an external device I / F 116, and a GPS receiver 117 are included.

  The CPU 101 executes a communication terminal program 118 stored in the ROM 102 and controls the operation of the entire communication terminal. The ROM 102 stores the communication terminal program 118. The communication terminal program 118 is distributed in a form downloaded from a server (not shown). The RAM 103 is used as a work area for the CPU 101. The SSD 105 is connected to the flash memory 104, and controls reading and writing of various data with respect to the flash memory 104 according to the control of the CPU 101. An HDD (Hard Disk Drive) may be used in addition to the SSD. The flash memory 104 stores various data such as image data and audio data.

  The media drive 107 controls reading or writing of data with respect to the recording medium 106. The recording medium 106 is a non-volatile memory such as a flash memory. The operation button 108 is a button for accepting an operation on the communication terminal 100 by the user, and includes a touch panel described later. The power switch 109 is a switch for switching ON / OFF of the power supply of the communication terminal 100. The telephone communication unit 110 communicates with an information processing apparatus such as a server or another communication terminal via a base station such as a mobile phone network. Audio data and various data are transmitted and received by communication.

  The network I / F 111 is a communication card that communicates with an information processing apparatus such as a server or another communication terminal via an access point such as a wireless LAN network. Audio data and various data are transmitted and received by communication. The camera 112 captures an image of the subject under the control of the CPU 101 and creates image data. The microphone 113 collects sound and converts it into an electrical signal. The speaker 114 outputs sound. The display I / F 115 outputs image data to the external display 120 according to the control of the CPU 101. The display 120 is configured using a liquid crystal element, an organic EL element, or the like that displays a subject image, an operation icon, and the like. The display 120 integrally has a touch panel, displays an image of a subject, an operation icon, and the like, and detects a user operation. The external device I / F 116 is, for example, a USB host, Bluetooth (registered trademark), NFC (Near Field Communication), TransferJet (registered trademark), or the like that transmits and receives various data to and from an external device. In addition, a bus line 119 such as an address bus or a data bus is provided for electrically connecting the above components. A GPS (Global Positioning System) receiving unit 117 receives radio waves from GPS satellites and estimates the position of the communication terminal 100.

  Note that the recording medium 106 is detachable from the communication terminal 100. The communication terminal program 118 is a file in an installable or executable format, and may be recorded on a computer-readable recording medium such as the recording medium 106 and distributed.

  The camera 112 includes a solid-state imaging device (for example, a CCD (Charge Coupled Device) device, a CMOS (Complementary Metal Oxide Semiconductor) device, etc.) that converts light into electric charges and digitizes an image (video) of a subject. Yes. In addition to being built in the communication terminal 100, the camera 112 may be externally attached via an external device I / F 116, for example. The same applies to the microphone 113 and the speaker 114. When an external camera, an external microphone, or an external speaker is connected, according to the control of the CPU 101, the external camera, external microphone, or An external speaker is driven.

  FIG. 4 is an example of a hardware configuration diagram of the database server 200 according to the present embodiment. The database server 200 includes a CPU 201, ROM 202, RAM 203, HDD 205, display I / F 208, external device I / F 215, network I / F 209, keyboard 211, mouse 212, media drive 207, and optical drive 214.

  The CPU 201 controls the operation of the entire database server. The ROM 202 stores BIOS (Basic Input Output System) and the like. The RAM 203 is used as a work area for the CPU 201. The HDD 205 controls reading and writing of various data with respect to the HD 204 according to the control of the CPU 201. The HD 204 stores the database program 230 and various data. The display I / F 208 outputs image data (various information such as a cursor, menu, window, character, and image) to the display 220 under the control of the CPU 201.

  The external device I / F 215 transmits / receives information to / from an external device. Examples of the external device I / F 215 include a USB interface, BlueTooth (registered trademark), NFC, TransferJet (registered trademark), and the like.

  A network I / F 209 is a communication card that communicates with an information processing apparatus such as a server or another communication terminal 100 via an access point such as a wireless LAN network. The keyboard 211 includes a plurality of keys for receiving input of characters, numerical values, various instructions, and the like. The mouse 212 accepts operations such as selection and execution of various instructions, selection of a processing target, movement of a cursor, and the like. The media drive 207 controls reading and writing of data with respect to a recording medium 206 such as a flash memory. The optical drive 214 controls reading and writing of data with respect to an optical disk 213 such as a CD (Compact Disc), a DVD (Digital Versatile Disk), and a Blu-ray which are removable recording media. The bus line 210 is an address bus, a data bus, or the like for electrically connecting the above components.

  FIG. 5 shows an example of a functional block diagram of the recognition system 500. The communication terminal 100 includes an imaging unit 11, a position information acquisition unit 12, a time acquisition unit 13, an image recognition processing unit 14, a recognition result display unit 18, and a communication system 17a as main functions. Each of these functions is realized by the CPU 101 of the communication terminal 100 shown in FIG. 3 executing the communication terminal program 118 stored in the ROM 102 and cooperating with the hardware shown in FIG.

  The imaging unit 11 outputs to the image recognition processing unit 14 image data captured by the user using the camera 112 with a flower as a subject. The position information acquisition unit 12 acquires position information of the current location (imaging location) when the imaging unit 11 captures image data. The position information is obtained by latitude / longitude and altitude. The position information is estimated by the GPS receiving unit 117, but may be estimated by the radio wave intensity from a wireless LAN access point or a mobile phone network base station. The timing acquisition unit 13 acquires the current time (hereinafter referred to as imaging time) from the calendar function or the like when the imaging unit 11 captures image data. The time includes, for example, at least a month and day, and may further include a time.

  The image recognition processing unit 14 compares the feature amount of the captured image captured by the image capturing unit 11 with the feature amount of the sample image registered in the flower search DB 16, and the same type of flower as the flower included in the captured image is found. Estimate which flower. Details will be described later.

  The recognition result display unit 18 displays the recognition result obtained by the image recognition processing unit 14. When displaying, the captured image captured by the imaging unit 11, the most probable candidate image determined to be most similar to the image recognition processing unit 14, and other probable candidate images are displayed. Other leading candidate images are all images with a recognition rate, which will be described later, equal to or higher than a reference value.

  The communication system unit 17a communicates with the database server 200. Thereby, the flower information DB 15 and the flower search DB 16 can be acquired from the database server.

  The flower information DB 15 and the flower search DB 16 are the same as or a part of the flower information DB 15 and the flower search DB 16 of the database server 200. By holding the flower information DB 15 and the flower search DB 16 on the communication terminal side, the recognition time can be shortened. The flower information DB 15 stores detailed information on flowers, and there is almost no change. The flower search DB 16 registers habitat environments and sample images for performing flower recognition processing. The data in the flower search DB 16 is changed by updating. When the image recognition processing unit 14 obtains the flower search DB 16 in response to a user operation, periodically obtains the flower search DB 16, or inquires of the database server 200 and finds that it has been updated, the image recognition processing unit 14 DB16 is acquired.

  The database server 200 includes a communication system unit 17b, a flower information DB 15, and a flower search DB 16. The communication system unit 17b communicates with the communication terminal 100. The flower information DB 15 and the flower search DB 16 are constructed by a server administrator, for example.

[Configuration of DB for flower search]
FIG. 6 is an example of a diagram illustrating the configuration of the flower search DB 16. The flower search DB 16 has fields of “flower name”, “habitat”, “maximum altitude”, “minimum altitude”, “flowering time”, and “withering time” as the habitat environment. Each field of “path to sample image” is included.
・ "Flower name" is a general name for flowers. The “flower name” is linked to the flower information DB 15, and the image recognition processing unit 14 can acquire the detailed information of the flower information DB 15 using the “flower name” as a key.
・ "Habitat" indicates the area where flowers inhabit by the map number. The map is managed by being divided into two-dimensional square areas, and one square is assigned a number (map number). Here, the list number used in the 1 / 50,000 map of the Geospatial Information Authority of Japan is used as the map number. However, if unique, the map number may be a number unique to the flower search DB 16. Therefore, a certain range of areas can be specified by the map number. For example, map number 9404 (list number 94-4) indicates an area including “Hakusan”, and map number 9403 indicates an area including “Shirakawa Village”. Moreover, although the map number of 1 / 50,000 scale is shown in the example of FIG. 6, you may construct | assemble DB16 for flower searches by other scales, such as a map number of 1 / 5,000.
・ "Highest altitude" indicates the upper altitude of the altitude range in which flowers can live in four stages. The “minimum altitude” indicates the lowest altitude in four levels within the altitude range in which flowers can live.
4 = 2500 [m] or more 3 = 1500 [m] or more 2500 [m] or less 2 = 500 [m] or more 1500 [m] or less 1 = less than 500 [m] ・ "Flowering time" is the time when flowers bloom 36 groups are shown. “Dead time” indicates the time when the flowers wither in 36 groups.
1 = early January 2 = mid-January 3 = late-January 4 = early February 5 = mid-February 6 = late February
34 = early December, 35 = mid December, 36 = late December The number of “highest altitude” and “lowest altitude” is just an example, and “highest altitude” and “lowest altitude” are expressed in 2 to 3 stages. Or may be expressed in five or more stages. In addition, the number of groups of “flowering time” and “withering time” is merely an example, and “flowering time” and “withering time” may be represented by less than 36 or more than 37 groups.
In “path to sample image”, a path to typical image data (photograph) of each flower is registered. The image of a flower registered in the flower search DB is called a sample image. Sample images of each color are registered for flowers called by the same name even if the colors are different. In addition, since the shape of the flower differs depending on the imaging direction, several sample images with different imaging directions may be registered. Thereby, recognition accuracy can be improved.

  FIG. 7 is an example of a diagram illustrating the flower information DB 15. The flower information DB 15 has fields of “flower name”, “scientific name”, “eyes”, “family”, “genus”, and “remarks”. “Flower name” is the same as the flower search DB 16. “Scientific name” is a Latin flower name, and “eyes”, “family” and “genus” are plant classifications. In "Remarks", flower words and trivia are registered.

[About image recognition]
Next, detailed functions of the image recognition processing unit 14 will be described with reference to FIG. FIG. 8 is an example of a functional block diagram of the image recognition processing unit 14. The image recognition processing unit 14 includes an image search unit 22, a filter management unit 21, and a search DB update unit 23. The image search unit 22 extracts a flower region from the captured image captured by the imaging unit 11 and searches for a similar flower from the sample image in the flower search DB 16.

  FIG. 9A is an example of a diagram for explaining extraction of a flower region. In the captured image, the color of the flower is often very different from the background and stem portion. Therefore, the flower region can be extracted by paying attention to the pixel value. For example, a histogram of pixel values of the captured image is created and clustered to about 16 colors. Since flowers are often imaged from the center to the top of the captured image, it is presumed that the most common color as the pixel value in this region is the color of the flower. The image search unit 22 determines a pixel area clustered to a pixel value estimated to be a flower color as a flower area. Thereby, the flower area | region which has a petal as a main element can be determined.

  The feature amount of the flower is used for searching for the flower. For example, the color and shape of the flower are recognized as features. In addition, information that can be extracted from the appearance of a flower can be used as a feature amount. When the flower region is determined, the image search unit 22 detects a minimum value, a maximum value, a median value, and the like of the color of the flower region in a predetermined color space. In addition, the shape of the flower region is classified into several standard patterns. For example, as shown in FIG. 9B, circular, semicircular, triangular, trapezoidal, hexagonal, etc. are prepared as standard patterns. By comparing these and the shape of the outer edge of the flower region by pattern matching and determining the most similar standard pattern, the shape of the imaged flower can be specified by the shape name. In addition, the image search unit 22 may detect the aspect ratio of the flower region, the number of vertices (for example, edge detection is performed to determine whether a straight line forms a vertex), or the like.

  In this way, the image search unit 22 obtains the flower color (color minimum value, maximum value, median value) and flower shape (standard pattern, aspect ratio, number of vertices), and uses the sample image in the flower search DB 16. Compare with the color and shape of the extracted flower. The feature amount of the sample image flower can be extracted in advance.

  The image search unit 22 calculates a difference between items having the same feature amount, and normalizes the difference by, for example, a value of 1 to 10 (the closer the feature amount is, the higher the numerical value is). And each item is weighted and totaled, and it converts into the value of 0-100. This is called a recognition rate. When the recognition rate is equal to or higher than the reference value, the image search unit 22 determines a captured image candidate.

  The reference value shall be determined experimentally. By confirming whether or not the sample image of the same flower as the captured image is actually searched for the recognition rate, the recognition rate that can be expected that the correct sample image is suitable for the search can be calculated. The non-patent document “FIT 2004 (3rd Information Science and Technology Forum), I-042, September 2004“ Flower Information Retrieval System Using Mobile Phones ”, Wakayama University Koji Hara, etc., has a description of recognition rate. Therefore, this recognition rate may be referred to.

  Instead of extracting feature quantities as described above (or in addition to feature quantity extraction), image recognition may be performed by pattern matching. For example, the edge of the flower region is detected, binarized, and scaled to the same size, and the similarity is calculated by a calculation formula such as SSD (Sum of Squared Difference) or SAD (Sum of Absolute Difference) by pattern matching. The similarity is zero when they are completely matched, and the number of pixels is matched when they are not completely matched. Therefore, if the similarity 0 to the maximum value is assigned to 0 to 100, the recognition rate can be obtained with a numerical value of 0 to 100.

[About filters]
Returning to FIG. 8, the filter management unit 21 will be described. The filter management unit 21 combines filters 1 to 3 to narrow down or relax the recognition target of the image search unit 22. The more types of filters, the shorter the search time because there are fewer search targets. For this reason, the filter management unit 21 searches the flower search DB 16 by filtering with all the filters 1 to 3 in the first (first) search.

When the sample image that satisfies the reference value does not match the search target that has been filtered by the filters 1 to 3, the filter management unit 21 removes the filters one by one. The filter 1 filters the search target for latitude and longitude (imaging location), the filter 2 filters the search target for the altitude of the imaging location, and the filter 3 filters the search target for the imaging time. In this embodiment, the filters 1 to 3 are combined as follows.
First search: filter 1 + filter 2 + filter 3
-Second search: Filter 2 + Filter 3
-Third search: Filter 3
-Fourth search: no filter That is, since the recognition rate with the captured image does not satisfy the reference value, the filter is removed one by one every time the search is performed again. Since the sample images searched once in the flower search DB 16 are not searched again, the number of sample images searched is the maximum number of sample images registered in the flower search DB 16.

This will be described with reference to FIG. FIG. 10 is an example of a diagram for explaining a search target of the flower search DB 16 when the filters 1 to 3 are applied. First, imaging conditions are as follows.
(i) Imaging location: Hakusan
(ii) Altitude: about 1600 [m]
(iii) Imaging time: September 15 A case where filtering is performed with all the filters 1 to 3 will be described.
FIG. 10A shows an example of the flower search DB 16 filtered by “habitat”. Since the imaging location is Hakusan (map number 9404), only the record of “habitat” 9404 is the search target.

  FIG. 10B shows an example of the flower search DB 16 filtered by “habitat”, “highest altitude”, and “lowest altitude”. Since the altitude is 1600 [m] (stage 3), only the records in which stage 3 is included in the “highest altitude” and “lowest altitude” in the sample image of the habitat 9404 are searched.

  FIG. 10C shows an example of the flower search DB 16 filtered by “habitat”, “highest altitude”, “lowest altitude”, “flowering time”, and “withering time”. Since the imaging time is September 15 (group 26), stage 3 is included in the "highest altitude" and "lowest altitude" of the sample image of the habitat 9404, and the "flowering time" and "withering time" Only records including the group 26 are to be searched.

  In this way, since the sample images to be searched can be narrowed by filtering, the search time can be shortened.

  On the other hand, when filter 1 is removed, filtering by “habitat” is not performed, so all sample images are filtered by “highest altitude”, “lowest altitude”, “flowering time”, and “withering time”. A sample image is a search target. In addition, when filter 1 and filter 2 are removed, filtering based on “habitat”, “highest altitude”, and “lowest altitude” is not performed, so all sample images are filtered according to “flowering time” and “withering time”. The sample image that has been set becomes the search target. Further, when all the filters 1 to 3 are removed, since all the sample images are not filtered, all the sample images are searched.

  By removing the filter step by step in this way, the search can be performed while increasing the search target little by little, so that it is possible to prevent the user from waiting for a long time. Moreover, the order which removes the filters 1-3 is an order with low influence. In other words, the filter is removed from the filters that are difficult to increase even if the filter is removed. Thereby, even if a filter is removed, it can suppress that search time increases.

[Update of flower search DB]
If the sample image whose recognition rate satisfies the reference value by removing the filter is suitable for the search, there is a high possibility that the flower habitat environment captured in the flower search DB 16 is not correctly registered. Therefore, the search DB update unit 23 updates the habitat environment of the flower search DB 16 by adding a habitat environment corresponding to the filter that was removed when the sample image was matched.

  FIG. 11 is an example of a diagram for explaining the update of the flower search DB 16. When the filters 1 to 3 are applied, the search target is limited to two sample images as shown in FIG. 10C. However, when a sample image that matches the captured image is not recognized, the filter management unit 21 selects the filter 1. remove. That is, although “habitat” is not filtered, it is filtered by “highest altitude”, “lowest altitude”, “flowering time”, and “withering time”. Therefore, as shown in FIG. 11A, among all the sample images, “3” is included in “highest altitude” and “lowest altitude”, and “26” is displayed in “flowering time” and “withering time”. The included sample image is a search target.

  If the flower of the captured image matches the search from the search target, the search DB update unit 23 updates the habitat environment of the flower search DB 16 corresponding to the removed filter 1. FIG. 11B shows an example of the updated flower search DB 16. For example, if the flower “Kinugasasso” matches the search, the search DB update unit 23 adds 9404 of the imaging location to the “habitat” of “Kinugasasso”. As a result, the “habitat” of “Kinugasassou” becomes 9403 and 9404, and when the user takes a picture of the goldfish at the imaging location 9404 next time, “Kinugasassou” is not leaked from the search target even if the filters 1 to 3 are applied. , You can search for “Kinugagassou” in one search.

  In FIG. 11, “habitat” is updated, but if the removed filter is “highest altitude” and “lowest altitude” and a flower imaged at an altitude higher than “highest altitude” matches the search, it matches. The “highest altitude” of the flower is updated. If the removed filter has a “highest altitude” and “lowest altitude” and a flower imaged at an altitude lower than “lowest altitude” matches the search, the “lowest altitude” of the matched flower is updated. Similarly, when the removed filter is “flowering time” and “withering time” and a flower imaged before “flowering time” matches the search, the “flowering time” of the matching flower is updated. When the removed filter is “flowering time” and “withering time” and a flower imaged after “withering time” matches the search, the “withering time” of the adapted flower is updated.

  In addition, in the state where a plurality of filters are removed, when the captured flowers match the search, the habitat environments corresponding to the removed filters among the adapted flower habitats are updated.

  Further, the search DB update unit 23 records an update history when the flower search DB 16 is updated. The update history is the date when the new habitat was registered and the type of the updated habitat. As a result, it is recorded which habitat was updated and when, so that it is possible to record from the actual imaging result that the habitat and altitude of the flower expands with time. In addition, the flowering time and the time of withering can be recorded from actual imaging results, such as the time when each flower is brought forward or the time of withering is delayed.

[Operation procedure]
FIG. 12 is an example of a flowchart illustrating an operation procedure of the recognition system. The procedure of FIG. 12 is started, for example, when the user starts the communication terminal program.

  First, when a user performs an imaging operation, the imaging unit acquires a captured image of a flower (S10). Thereby, an imaging location (position information) and an imaging time are acquired.

  The image recognition processing unit 14 reads the flower search DB 16 (S20).

  First, the filter management unit 21 filters the flower search DB 16 with three filters 1 to 3 of an imaging location (habitat), an altitude (maximum altitude / minimum altitude), and an imaging timing (flowering time / dead time). (S30).

  The image search unit 22 performs image recognition processing on the sample image filtered and narrowed by the filter management unit 21 (S40).

  The image search unit 22 determines whether there is a sample image with a recognition rate equal to or higher than a reference value (S50).

  If there is no sample image with a recognition rate equal to or higher than the reference value (No in S50), the filter management unit determines whether all filters have been removed (S60).

  If all the filters have been removed (Yes in S60), it means that there is no sample image that satisfies the reference value, so the processing in FIG. 12 ends. In this case, the recognition result display unit 18 displays an error message or the like on the display 120.

  When all the filters have not been removed (No in S60), the filter management unit 21 cancels the filtering in ascending order of influence (S70). That is, the filtering is stopped. Then, image recognition is performed again (S40).

  When there is a sample image whose recognition rate is equal to or higher than the reference value (Yes in S50), the search DB update unit 23 determines whether or not the filtering is canceled (S80).

  When the filtering is not canceled (No in S80), it is not necessary to update the flower search DB 16, and thus the process of FIG. 12 ends.

  When there is a sample image whose recognition rate is equal to or higher than the reference value by canceling the filtering (Yes in S80), the search DB update unit 23 is the habitat environment that has been filtered with the filter of the sample image that matches the search. Is updated (S90).

[Screen display example]
FIG. 13 is a diagram illustrating a display example of the recognition result on the display 120 of the communication terminal 100. As shown in FIG. 5, on the display 120, the captured image (“Your photo” in the figure) 52, the most likely candidate image (displayed together with the flower name “Metakarako” in the figure) 53, and others The leading candidate images 51 are displayed. The user can compare the captured image 52 with the most probable candidate image 53 or compare the captured image 52 with another promising candidate image 51 to determine whether or not the search is suitable.

  When the user selects the most probable candidate image 53 or another promising candidate image 51, the image recognition processing unit 14 reads the “flower name” record selected by the user from the flower information DB 15. Then, the detailed information of the flower is displayed on the screen.

  FIG. 14 is a diagram showing an example of detailed information displayed on the display 120. In FIG. 14, a captured image 52, a sample image 55, and detailed information 54 are displayed. The user can check the detailed information to check whether the sample image is a correct image of the captured flower.

  FIG. 15 is a diagram illustrating a display example of the flower habitat environment using the update history. FIG. 15A shows an update history of a “highest altitude” of a certain flower displayed on the display 120. The “highest altitude” before the database is updated is “2”, and the “minimum altitude” is “1”. Since this flower was imaged at an altitude of 1500 [m] or more and 2500 [m] or less on “December 10, 2013”, the “maximum altitude” is updated to “3”. Moreover, since this flower was imaged at an altitude of 2500 [m] or more on “November 20, 2015”, the “maximum altitude” is updated to “4”.

  Therefore, by recording the update history of “highest altitude” and “lowest altitude” in the flower search DB 16, the user can change the altitude at which each flower blooms with time (lower at higher altitude and lower). Blossoms at altitude) and when it has changed.

  FIG. 15B shows an update history of a flowering time displayed on the display 120. The “flowering time” before the database is updated is “10 (early April)”, and the “dead time” is “18 (late July)”. Since this flower was imaged on “March 15, 2013”, the “flowering time” is updated to “8 (mid-March)”.

  Accordingly, the update history of “flowering time” and “withering time” is recorded in the DB 16 for flower search, so that the user can change the time period during which each flower is flowering with time (delayed or delayed). And it becomes possible to grasp when it has changed.

  FIG. 15C shows an update history of a certain flower habitat displayed on the display 120. The “habitat” before the database is updated is “9404 (Hakusan). And on December 15, 2013”, because this flower was imaged in Shirakawa village, “habitat” is “9403 ( Shirakawa village) has been updated to include "". For example, the difference between the “habitat” before and after the update is clearly indicated by changing the color of the “habitat” before and after the update.

  Therefore, by recording the update history of “habitat” in the flower search DB 16, the user can change the habitat of each flower with time (for example, going north, going south), and when. You will be able to figure out.

[Other configuration examples]
FIG. 16 shows an example of a schematic configuration diagram of a recognition system of a server / client system. Since components having the same reference numerals in FIG. 5 perform the same function, only the main components of the present embodiment may be mainly described.

  In FIG. 16, the communication terminal 100 has an image recognition processing unit 14, and the communication terminal 100 has an image transmission unit 19 and a recognition result acquisition unit 20. The image transmission unit 19 transmits the captured image captured by the imaging unit 11, the positional information acquired by the positional information acquisition unit 12, and the imaging timing acquired by the timing acquisition unit 13 to the server via the communication system unit 17a. .

  In addition, the recognition result acquisition unit 20 acquires the most probable candidate image and other probable candidate images as recognition results from the database server 200 and outputs them to the recognition result display unit 18. Thereby, the recognition result display part 18 can display a recognition result similarly to the structure of FIG. The function of the image recognition processing unit 14 of the server is the same as that of the communication terminal 100.

  Thus, since the server has the image recognition processing unit 14, the flower information DB 15, and the flower search DB 16, the server can perform processing with a high load. Further, when the size of the flower information DB 15 and the flower search DB 16 is large, the SSD 105 and the RAM 103 of the communication terminal 100 are not compressed.

  The best mode for carrying out the present invention has been described above with reference to the embodiments. However, the present invention is not limited to these embodiments, and various modifications can be made without departing from the scope of the present invention. And substitutions can be added.

  For example, it has been described that the position information and the time are acquired by imaging by the imaging unit, but the image is recognized in the same manner even when the imaging location and the imaging timing are attached to the captured image in which the flower is captured in advance. Or updating the flower search DB.

  In the present embodiment, only the date is used for filtering as the imaging time, but the time may be used for filtering. In this case, the time zone for blooming as a habitat environment is also registered in the flower search DB. This makes it possible to search for flowers that bloom in a fixed time zone and update the time zone.

DESCRIPTION OF SYMBOLS 11 Imaging part 12 Position information acquisition part 13 Time acquisition part 14 Image recognition process part 15 Flower information DB
16 DB for flower search
21 filter management unit 22 image search unit 23 DB update unit for search 100 communication terminal 200 database server 500 recognition system

JP 2007-133816 A

Claims (9)

On the computer,
An image data acquisition step of acquiring second image data in which a living organism is imaged;
Accessing a data storage unit in which the first image data and the habitat of the organism are registered;
A search target extraction step of extracting, from the data storage unit, organisms in the habitat environment including an imaging condition in which the second image data is captured;
The feature amount of the organism is acquired from the second image data, and the first image data of the organism whose similarity with the feature amount of the search target organism extracted in the search target extraction step is a threshold value or more is determined. A recognition step to
In the state where there is one or more habitats that are not used for extraction of the search target in the search target extraction step, the first image data of a living thing whose similarity is equal to or higher than a threshold is determined by the recognition step. An update step of updating the habitat environment of the data storage unit not used for extraction of the search target with the imaging conditions;
A program characterized by having executed. In the state where the habitat environment is used to extract the search target in the search target extraction step, when the first image data of the organism whose similarity is equal to or higher than a threshold is not determined by the recognition step,
In the search object extraction step, stop using the habitat environment for the extraction of the search object in the order of little influence on the search time,
The program according to claim 1.   In the updating step, when the habitat environment that is not used for extraction of the search target in the data storage unit is updated with the imaging condition, the date on which the second image data was captured is registered. The program according to claim 1. A display step of displaying the habitat environment before being updated and the habitat environment after being updated on a display device together with the updated time;
The program according to claim 3, further causing the computer to execute. The habitat is one or more of a habitat, an altitude range to inhabit, and a time range to inhabit;
The imaging condition is one or more of an imaging location, an imaged altitude, and an imaged time,
In the state where one or more of the habitat, the altitude range where it lives, or the time range where it lives is not used for the extraction of the search object in the search object extraction step, When 1 image data is determined,
In the update step, the imaging location is added to a habitat of the data storage unit that is not used for extraction of the search target, and the altitude range of the data storage unit that is not used for extraction of the search target The imaged altitude range is added to or the imaged time is added to the time range of the data storage unit that is not used for extraction of the search target. 4. The program according to any one of 4 above. The organism is a flower, plant, insect, mushroom, or moss.
The program according to any one of claims 1 to 5, wherein: Image data acquisition means for acquiring second image data in which a living organism is imaged;
A data storage unit in which the first image data of the organism and the habitat environment of the organism are registered;
Search target extraction means for extracting organisms in the habitat environment including the imaging conditions under which the second image data was captured from the data storage unit as search targets;
The feature amount of the organism is acquired from the second image data, and the first image data of the organism whose similarity with the feature amount of the search target organism extracted by the search target extraction unit is equal to or greater than a threshold is determined. Recognition means;
In a state where there is one or more habitats that are not used for extraction of the search target by the search target extraction unit, when the recognition unit determines the first image data having a similarity equal to or higher than a threshold value, the search target Update means for updating the habitat environment of the data storage unit that is not used for extraction with the imaging conditions. An image data obtaining unit obtaining second image data in which a living thing is imaged;
A recognizing means accessing a data storage unit in which the first image data of the organism and the habitat of the organism are registered;
A search object extraction means for extracting, as a search object, organisms in the habitat environment including an imaging condition in which the second image data is captured;
The recognizing unit obtains the feature amount of the organism from the second image data, and the first image of the organism whose similarity with the feature amount of the search target organism extracted by the search target extraction unit is a threshold value or more. Determining the data;
In a state where there is one or more habitats that are not used by the search target extraction unit to extract the search target, when the recognition unit determines the first image data having a similarity equal to or higher than a threshold value, an update unit And updating the habitat environment of the data storage unit that is not used for the extraction of the search target with the imaging condition. A biological recognition method in which a communication terminal communicates with an information processing device and recognizes a living organism,
The communication terminal is
Transmitting second image data in which a living thing is imaged to the information processing apparatus;
Receiving a recognition result from the information processing apparatus,
Recognizing means of the information processing apparatus accessing a data storage unit in which first image data of a living organism and a habitat environment of a living organism are registered;
A search target extraction unit of the information processing apparatus extracts, as a search target, organisms in the habitat environment including an imaging condition in which the second image data is captured;
The recognition unit of the information processing apparatus acquires a feature amount of the organism from the second image data, and the similarity of the feature amount of the search target organism extracted by the search target extraction unit is greater than or equal to a threshold value. Determining the first image data;
In a state where there is one or more habitats that are not used by the search target extraction unit to extract the search target, the recognition unit determines the first image data having a similarity equal to or higher than a threshold value. An update unit of the apparatus updates the habitat environment of the data storage unit not used for extraction of the search target with the imaging condition;
The biological recognition method characterized by having.