JP2008172586A - Shape acquisition device, program and information retrieval system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To correctly extract the shape of an object to be shape-extracted even when an object other than the object to be shape-extracted is reflected in image data. <P>SOLUTION: First, as an initial value of a diaphragm value, an open value or a value close to it is set to perform imaging. The shape extraction of a leaf is attempted to the image data generated by the imaging, when the shape extraction is impossible (when a closed loop can not be formed), the set diaphragm value is increased by n steps to perform the imaging and the shape extraction again. When the closed loop is formed, shape data at the time is adopted as the shape data of the leaf. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an apparatus and program for acquiring the shape of an object by extracting contours from image data, and a system for performing information retrieval based on the shape data.

  For example, when a flower is imaged, the feature (shape and color) of the flower is extracted from the obtained image data by image recognition, the name of the flower is searched from the flower database based on the feature data, and the search result is displayed. An electronic camera that can be used has been proposed (see, for example, Patent Document 1). By providing such a search function to the camera, when a flower whose name is unknown is found, it is possible to easily know the name of the flower simply by photographing the flower. In addition to flowers, this type of search function can be widely applied to birds, insects, animals, structures, and the like.

JP 2002-247436 A

  In the image recognition, the shape can be specified by extracting the outline of the target object (flower or the like). However, various things other than the target object are often reflected in the image data, and it may be difficult to accurately extract the shape of the target object from the image data. For example, in image data in which a plurality of objects of the same type and the same color as the object whose shape is to be extracted are captured, there is a possibility that the plurality of objects may be erroneously recognized as one object depending on the situation.

The shape acquisition device according to the present invention captures the same object with a larger aperture value when shape extraction means for extracting the shape of the object by extracting the contour of the object from image data and shape extraction is impossible Control means for controlling the shape extraction means to perform shape extraction again on the obtained image data, and the output of the shape extraction means when the shape extraction is performed normally is the shape data of the object And
An image pickup unit that picks up an image of the subject and generates image data may be further provided, and the image data to be subjected to the shape extraction may be obtained by the image pickup unit. In that case, imaging and shape extraction may be performed alternately while increasing the aperture value until shape extraction is performed normally.
Another shape acquisition apparatus according to the present invention includes a shape extraction unit that extracts the shape of an object by extracting the contour of the object from image data, and an aperture value of image data obtained by capturing the same object with different aperture values. Control means for estimating a region occupied by an object from small image data and controlling the shape extraction means to extract the shape of the object from the estimated region from image data having a large aperture value. .
An imaging unit that captures an image of a subject and generates image data may be further provided, and image data that is a target of shape extraction may be obtained by the imaging unit.
Search means for searching for object information from a predetermined database may be further provided using the shape data obtained by shape extraction as a search key.
You may further provide the transmission means which transmits the shape data obtained by shape extraction as a search key outside.
The shape acquisition program according to the present invention includes a function for extracting the shape of an object by extracting the contour of the object from image data, and an image obtained by photographing the same object with a larger aperture value when the shape cannot be extracted. This is for causing the computer to realize a function of performing shape extraction again on the data and a function of using the shape extraction result when the shape extraction is performed normally as the shape data of the object.
Another shape acquisition program according to the present invention has a function of extracting the shape of an object by extracting the contour of the object from image data and a small aperture value among image data obtained by photographing the same object with different aperture values. This is for causing a computer to realize a function of estimating an area occupied by an object from image data and extracting the shape of the object from the estimated area from image data having a large aperture value.
An information search system according to the present invention receives the shape acquisition device according to claim 7 and shape data transmitted from the shape acquisition device, and uses the received shape data as a search key to retrieve object information from a predetermined database. And an external server for searching and transmitting the searched information to the shape acquisition device.

  According to the present invention, it is possible to accurately extract the shape of the target object even when the object other than the shape extraction target object is reflected in the image data.

-First embodiment-
A first embodiment of the present invention will be described with reference to FIGS.
FIG. 1 is a configuration diagram of an information search system in the present embodiment. This search system allows the feature data of the flowers and leaves taken by the digital camera 100 to be sent to the search server 200, and enables the search server 200 to search for information such as the name of the plant and send it back to the camera 100. It is. The search server 200 has a pictorial book database 201 storing information on flowers and leaves and a search engine 202, and is placed on a network. The camera 100 can be connected to the search server 200 via the communication network 300 and can exchange information with the server 200.

  FIG. 2 is a control block diagram of the camera 100. The subject luminous flux that has passed through the photographing lens 1 and the diaphragm 2 is received by an imaging device 3 such as a CCD and subjected to photoelectric conversion. The image processing circuit 4 performs various processes on the photoelectric conversion output of the image sensor 3 to generate image data, and the image data is recorded on the image recording medium 6 via the recording circuit 5. The image processing circuit 4 and the recording circuit 5 are controlled by the CPU 7.

  The aperture drive unit circuit 8 drives the aperture 2 based on an aperture value (set aperture value) set by the photographer or the CPU 7 in accordance with an instruction from the CPU 7. The lens driving circuit 9 performs focusing and zooming of the photographing lens 1 in accordance with instructions from the CPU 7. A predetermined image recognition program is stored in the memory 10, and the CPU 7 extracts features of flowers and leaves from the image data using this program. The display device 11 includes, for example, a liquid crystal monitor and a driving unit thereof, and can display an image display, a menu display, a warning message, and the like. The wireless communication circuit 12 communicates with the outside via the communication line network 300 in accordance with instructions from the CPU 7.

  In flower information retrieval, plant flower feature information (petal color, shape, number, etc.) is a basic retrieval key, but it is often impossible to narrow down candidates. This is because there are multiple cases of heterogeneous plants with very similar flower characteristics. In this regard, it has been found that the search results can be narrowed down considerably by adding not only flowers but also leaf-shaped features as search keys. Therefore, in the present embodiment, for the same plant, the feature information of the flower is extracted from the image data obtained by photographing the flower, the feature information (shape data) of the leaf is extracted from the image data obtained by photographing the leaf, and these are combined. Use as a search key.

  FIG. 3 shows an example of classification of leaf-shaped features. This is a classification based on "cuts", (a) "single leaf" is a single leaf with almost no cut, (b) "deep cut" is a deep cut but one leaf, The compound leaf of (c) shows the leaf which divided | segmented into the several leaflet by the cut. In the leaf shape extraction, it is determined which of the nine types in FIG.

  The extraction of the shape of a flower or leaf is performed by extracting the outline of one flower or leaf using the image recognition program described above. For the contour extraction, differentiation of pixel information is basically used, but a known method such as a prewitt filter or a sobel filter can be used. After the contour extraction, the shape of the flower or leaf can be cut out by performing a process of surrounding the contour with a closed loop. Flowers are generally colorful and green flowers are rare, so the green, brown, and gray lines in the image data are the leaves, soil, stones, and blocks, respectively. By excluding them as non-flower parts such as buds, the shape of one flower can be extracted relatively easily.

  On the other hand, most of the leaves are green regardless of the type, and a plurality of leaves of the same color are usually shown in one image. In order to accurately extract the shape of one leaf (one group in the case of double leaves) from such an image, the boundary between one leaf and its surroundings is clear, and the boundary is ambiguous in other parts. There is a need. In other words, it is ideal that the entire edge of one leaf is in focus and the other leaves are greatly blurred.

  The present inventors conducted an experiment to extract the shape of a plant leaf from image data. At the time of photographing, as shown in FIG. 4, a single lens reflex type digital camera 100 was equipped with a macro lens of 105 mmF2.8 and fixed at a distance of 30 cm from the potted monocotyledonous plant. Then, with the focus on one leaf of the plant, imaging was performed a plurality of times with different apertures, and the obtained plurality of image data was taken into the personal computer 400, and the respective contours were extracted with the prewitt filter. Next, a process of surrounding the extracted contour with a closed loop was performed to test whether or not the extraction can be accurately performed. 5 to 7 schematically show the contour extraction image at that time. Reference numeral 50 indicates a focused leaf, a solid line indicates a clear outline, and a dotted line indicates an unclear area. ing.

  FIG. 5 shows an aperture value of F3.2. Since the depth of field is considerably shallow, leaves other than the leaves 50 can be greatly blurred. However, only a part of the leaves 50 is in focus. The edge of the leaf 50 was partially blurred. For this reason, the closed loop of the shape of the leaf 50 could not be formed, and the shape of the leaf 50 could not be cut out.

  On the other hand, FIG. 6 shows a case where the aperture value is set to F20. Since the depth of field is considerably deep, the entire edge portion of the focused leaf 50 can be clarified. However, some of the leaves in the background of the leaf 50 are also within the depth of field, and the leaves that do not enter the depth of field have a small amount of blur, so the edges of the leaves other than the leaf 50 are also clear. I have. As a result, a plurality of leaves were recognized as one leaf, and the shape of the leaf 50 could not be cut out.

  FIG. 7 shows the aperture when the aperture is set to F8.0, the depth of field is medium, the entire edge portion of the leaf 50 can be clarified, and other leaves can be moderately blurred. . As a result, the shape of the leaf 50 could be accurately cut out.

  From the above, it can be understood that not only the focus state but also the selection of the aperture value is important when capturing an image used for leaf shape extraction. However, since it is difficult for the photographer himself to find the optimum aperture value, in this embodiment, the camera 100 finds the optimum aperture value as described below, and the image data photographed with the aperture value is used as the feature of the leaf. Select as extracted image data.

8 and 9 are flowcharts showing a specific example of information search processing including feature extraction.
When the information search mode is set, the CPU 7 executes this program. First, in step S1, a message prompting the user to shoot a flower is displayed on the display device 11, and in step S2, a photographer's shooting operation is awaited. When the photographer puts the target flower in the composition and operates the release button, the photograph is taken in step S3, and the feature of the flower is extracted from the generated image data.

  Next, in step S4, the CPU 7 displays a message for prompting the photographing of the leaves, and waits for a photographing operation in step S5. When the photographer puts the target leaf in the composition and operates the release button, photographing and leaf shape extraction are performed in step S6.

FIG. 9 is a flowchart showing details of step S6.
In step S61, the initial value of the set aperture value is set to the open value or a value close thereto. In step S62, the aperture 2 is driven based on the set aperture value, and imaging is performed. In step S63, leaf shape extraction is attempted on image data generated by imaging. In step S64, it is determined whether a closed loop is formed. As described above, when the aperture value is small and the depth of field is shallow, there is a high possibility that a closed loop cannot be formed. In this case, the process proceeds to step S65. In step S65, it is determined whether or not the set aperture value is the maximum aperture value. If the aperture value is the maximum aperture value, an error message indicating that the shape cannot be extracted is displayed in step S66, and the process returns.

  If it is determined in step S65 that it is not the maximum aperture value, the set aperture value is increased by n steps in step S67, and the process returns to step S62 to perform another imaging and shape extraction. Possible values of n are 1, 2, 1/2, and the like. The photographer may be able to set n in advance.

  On the other hand, if it is determined in step S64 that a closed loop has been formed, the leaf shape data has been acquired, and the process returns. In this case, since the closed loop is formed with the smallest possible aperture value, there is a high possibility that the shape of the focused leaf can be accurately cut out.

  In step S7 of FIG. 8, the flower feature information obtained in step S3 and the leaf feature information (shape data) obtained in step S6 are transmitted to the search server 200 as search keys. However, if the leaf shape data cannot be obtained (if step S65 is affirmed), only the flower feature information is transmitted as a search key. Note that transmission may be performed only after the feature information of both flowers and leaves is acquired.

  The search server 200 that has received the search key searches for information suitable for the search key from the pictorial book database 201 using the search engine 202, and transmits the search result to the camera 100. The CPU 7 of the camera 100 displays the received search result on the display device 11.

-Second Embodiment-
In this embodiment, leaf shape data is acquired using image data captured with a small aperture value and image data captured with a large aperture value. In the above-described experiment, in the image data photographed at F3.2, although the closed loop of the leaf 50 could not be formed as shown in FIG. 5, the approximate area occupied by the leaf 50 can be estimated. On the other hand, in the image data photographed at F20, a closed loop of the leaves 50 can be formed as shown in FIG. Therefore, by applying a filtering process that leaves only the approximate area (stored as coordinates) of the leaf 50 estimated in FIG. 5 to the shape data of FIG. 6, only the shape data of the leaf 50 can be cut out. .

As specific control, the flow of FIG. 10 may be used instead of the flow of FIG.
In FIG. 10, in step S101, the set aperture value is set to the open value or a value close thereto. In step S102, the aperture 2 is driven based on the set aperture value, and imaging is performed. In step S103, an attempt is made to extract the leaf shape of the image data generated by the imaging. In step S104, it is determined whether or not a closed loop has been formed. If the determination is affirmative, the shape data can be acquired, and the process directly returns. On the other hand, if step S104 is negative, the process proceeds to step S105.

  In step S105, an approximate region of the leaf is estimated from the shape data obtained in step S103 and stored. In step S106, the aperture value is set to a larger value (for example, F20). In step S107, the aperture 2 is narrowed down based on the set aperture value, and imaging is performed. In step S108, shape extraction is performed on the image data obtained by imaging. In step S109, the shape data obtained in step S108 is subjected to a filtering process that leaves only the area stored in step S105, and the shape of the leaf is cut out to be shape data.

  In the experiment, a single-leaf plant was used. However, if the methods of the first and second embodiments are used, as shown in FIGS. 11 and 12, shape data of deep cuts and compound leaves can be obtained accurately. Increases nature.

  Although the leaf shape extraction has been described above, the method of the present invention may be used for flower shape extraction. For example, when a scene where flowers of the same type and color are densely blooming is photographed, the shape of one flower can be accurately extracted from the image data. In addition to the plant, the technique of the present invention can be similarly applied to shape extraction of birds, insects, animals, structures, and the like.

  In the above description, the camera is used. However, the present invention can also be applied to an apparatus that does not have a photographing function and performs shape extraction on image data input from the outside. For example, image data obtained by shooting the same subject with different aperture values can be imported to a computer, and the shape can be extracted sequentially from image data with a smaller aperture value using software built into the computer, or a specific object can be extracted from image data with a smaller aperture value. The area occupied by the object is estimated, and the shape of the object is extracted from the estimated area from the image data having a large aperture value.

  Furthermore, an information search system comprising a camera and a search server has been shown. However, if a picture book database and a search engine are incorporated in the camera, the camera can be searched alone. The present invention can also be applied when extracting the shape of an object for purposes other than information retrieval.

1 is a network diagram of an information search system in an embodiment of the present invention. The control block diagram of the digital camera which comprises a search system. The figure which shows the shape classification of the leaf of a plant. The figure explaining the experimental method by the present inventors. The figure showing an experimental result, and a figure showing a shape extraction result of a leaf when an aperture value is F3.2. FIG. 5 is a diagram similar to FIG. 5 and shows the leaf shape extraction result when the aperture value is F20. FIG. 5 is a view similar to FIG. 5 and shows the leaf shape extraction result when the aperture value is F8. The flowchart which shows the example of a procedure of a search process. The flowchart which shows the detail of the shape extraction of the leaf in 1st Embodiment. The flowchart which shows the detail of the shape extraction of the leaf in 2nd Embodiment. The shape extraction result of the “deep cut” leaf is shown. The shape extraction result of the “double leaf” leaf is shown.

Explanation of symbols

2 Aperture 3 Image sensor 4 Image processing circuit 7 CPU
8 Aperture drive circuit 11 Display device

Claims (10)

Shape extracting means for extracting the shape of the object by extracting the contour of the object from the image data;
Control means for controlling the shape extraction means to perform shape extraction again on image data obtained by photographing the same object with a larger aperture value when the shape extraction is impossible;
A shape acquisition apparatus characterized in that an output of the shape extraction means when the shape extraction is normally performed is object shape data.   The shape acquisition apparatus according to claim 1, further comprising an imaging unit that captures an image of a subject and generates image data, and the imaging unit obtains image data to be extracted.   The shape acquisition apparatus according to claim 2, wherein imaging and shape extraction are alternately performed while increasing the aperture value until the shape extraction is performed normally. Shape extracting means for extracting the shape of the object by extracting the contour of the object from the image data;
The area occupied by the object is estimated from image data having a small aperture value among image data obtained by photographing the same object with different aperture values, and the shape of the object is extracted from the image area having a large aperture value. A shape acquisition apparatus comprising: control means for controlling the shape extraction means as much as possible.   The shape acquisition apparatus according to claim 4, further comprising an image pickup unit that picks up an image of a subject and generates image data, and the image pickup unit obtains the image data to be extracted.   The shape acquisition apparatus according to claim 1, further comprising a search unit that searches for information on the object from a predetermined database using the shape data obtained by the shape extraction as a search key. .   The shape acquisition apparatus according to claim 1, further comprising transmission means for transmitting the shape data obtained by the shape extraction to the outside as a search key. A function of extracting the shape of the object by extracting the contour of the object from the image data;
A function of performing shape extraction again on image data obtained by photographing the same object with a larger aperture value when the shape extraction is impossible;
A shape acquisition program for causing a computer to realize a function of using a shape extraction result when the shape extraction is normally performed as shape data of an object. A function of extracting the shape of the object by extracting the contour of the object from the image data;
The area occupied by the object is estimated from image data having a small aperture value among image data obtained by photographing the same object with different aperture values, and the shape of the object is extracted from the image area having a large aperture value. Shape acquisition program for realizing functions and functions on a computer. The shape acquisition device according to claim 7,
An external server that receives the shape data transmitted from the shape acquisition device, searches for information on an object from a predetermined database using the received shape data as a search key, and transmits the searched information to the shape acquisition device. An information retrieval system characterized by:

Images