JP2007179389A - Photographing device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To obtain an accurate feature when extracting the feature of a subject based on an image data. <P>SOLUTION: This photographing device extracts the feature of the subject based on the image data assigned by an area assignment means for assigning an optional area of the image data in response to an operation, and outputs the feature as a retrieval key in subject information retrieval. A shape and a size of a frame for determining the area, and a position of the image data, for example, are made to be changeable in a method of assigning the area. Subjects within the area more than enough (affecting unfavorably the extraction of the feature) are removed by this manner. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a photographing apparatus such as an electronic camera or a mobile phone with a camera function, and more particularly, to a subject search by extracting a feature of a subject from a captured image.

  For example, when a flower is imaged, the feature of the flower is extracted from the obtained image data by image recognition, the name of the flower is searched from the flower database based on the feature data, and the search result can be displayed. A camera has been proposed (see, for example, Patent Document 1). By providing such a search function to the camera, when a flower whose name is unknown is found, it is possible to easily know the name of the flower simply by photographing the flower.

JP 2002-247436 A

  Whether or not accurate feature data can be extracted when extracting the features of the subject from the image data depends on the content of the target image data. For example, in flower search, if part of the target flower is missing, or if a flower other than the target flower is included in the image, the exact feature of the target flower cannot be extracted, and the accurate search result Cannot be obtained. However, in actual shooting, it is often difficult to shoot except for the target subject.

An imaging apparatus according to the present invention is based on an imaging unit that images a subject to generate image data, an area designation unit that designates an arbitrary area of image data according to an operation, and image data of the designated area. Feature extraction means for extracting features of the subject and outputting the features as a search key in the subject information search.
The area specifying means may include any or all of a means for specifying the size of the area, a means for specifying the shape of the area, and a means for specifying the position of the area.
The area designating means includes means for selecting a frame representing the shape of the area, and means for designating the size and position of the selected frame on the image data. The feature of the subject may be extracted on the basis of it.
The area specifying means includes means for selecting one of a plurality of frames according to the shape of the subject, and extracts the feature of the subject based on the image data of the area surrounded by the selected frame on the image data. It may be.
Search means for searching subject information from the database based on the extracted features, and notification means for notifying the searched subject information may be further provided.

  According to the present invention, harmful subjects in target image data can be removed without paying attention to the composition, and accurate search results can be obtained by accurate feature extraction.

An embodiment of the present invention will be described with reference to FIGS.
FIG. 1 is a schematic view of an electronic camera 100 according to this embodiment as viewed from the back. A release button 1 is provided on the upper surface of the camera, and a liquid crystal monitor 2 capable of displaying an image and zoom up / down buttons 3U and 3D for zooming up / down the photographing lens 10 (FIG. 2) on the back surface. A four-way button 4 capable of four operations, up, down, left and right, a reproduction button 5 for reproducing an image, and a search button 6 for performing an information search described later. The four-way button 4 is a multi-function button, and is used for item selection at the time of menu setting, reproduction image selection, other selections, and display movement. In particular, the up / down operation of the four-way button 4 is also used as an input of a reply (Yes or No) to the confirmation message.

  FIG. 2 is a control block diagram of the electronic camera 100. The CPU 11 that controls all operations of the electronic camera 100 includes a release switch 1s that is linked to the operation of the release button 1 and zoom up / down switches 3Us and 3Ds that are linked to the operations of the zoom up / down buttons 3U and 3D, respectively. Connected are up / down / left / right switches 4Us, 4Ds, 4Ls, 4Rs linked to the up / down / left / right operation of the four-way button 4, a playback switch 5s linked to the operation of the playback button 5, and a search switch 6s linked to the operation of the search button 6. . The CPU 11 reads the state of these switches and detects whether or not each button is operated.

  The subject luminous flux that has passed through the photographing lens 10 is imaged by an image sensor 12 such as a CCD, and converted into an electrical image signal. The imaging circuit 13 processes this image signal to generate image data in a predetermined format, and the CPU 11 records the image data in the large-capacity memory 14 as an image file that can be handled by a computer. The large-capacity memory 14 may be built in the electronic camera 100 or a removable memory card.

  Further, the liquid crystal monitor 2 is connected to the CPU 11 via the display circuit 15. On the liquid crystal monitor 2, images taken every predetermined time are sequentially updated and displayed via the finder memory 16 (through image display), and the photographer can determine the composition while viewing the images. Further, the liquid crystal monitor 2 can display image data recorded in the large-capacity memory 14 in accordance with the operation of the playback button 5 and can display a menu.

  Furthermore, a picture book database 17 in which flower information is registered is connected to the CPU 11. In the pictorial book database 17, ecological information such as the type, name, and habitat of a flower and a sample image of the flower are stored in association with feature data such as the color, shape, and number of petals.

Hereinafter, flower information search using the pictorial book database 17 will be described in detail.
In the flower information search, a feature of a flower is extracted from image data of a photographed flower, and the name of the flower is searched from the pictorial book database 17 and displayed using the extracted feature information as a search key.

  When searching for flower information, the operator operates the playback button 5 to display and display image data including the target flower (a flower for which information such as a name is desired) on the liquid crystal monitor 2. When the playback button 5 is operated, the CPU 11 reads an image recorded in the large capacity memory 14 to the finder memory 16 and displays it on the liquid crystal monitor 2 via the display circuit 15. On the liquid crystal monitor 2, for example, a search frame F as indicated by a broken line in FIG. The search frame F determines a cut-out area when image data is cut out later, and the target flower needs to be stored in the search frame F. In the present embodiment, the shape, size, and position of the search frame F can be changed, which will be described in detail later.

  When the search button 6 is operated in a state where the image data is reproduced, the CPU 11 cuts out an area in the search frame F from the image data and sets the cut-out image data as analysis target image data. FIG. 3 shows analysis target image data cut out from the image data of FIG. The CPU 11 analyzes the analysis target image data using a predetermined image recognition program, and extracts flower features. Features to be extracted are the color, shape, number of petals, etc. Since these features are used as search keys in the flower information search, satisfactory search results cannot be obtained unless accuracy is lacked. In order to accurately extract the features, it is necessary to clearly distinguish the flower portion and the non-flower portion in the analysis target image data. For this reason, the image recognition program recognizes, for example, those of the green system, the tea system, and the gray system in the analysis target image data as non-flower parts such as leaves, soil, stones, and block corals.

  By the way, the image analysis as described above takes time as the number of pixels of the analysis target image data increases. In recent years, the number of pixels of an electronic camera has been increased. For example, if the captured image data is 6 million pixels, the analysis target image data may be 1 million pixels. However, a general image recognition program does not require 1 million pixels to extract the feature of a flower. Therefore, if the cut-out image data to be analyzed is used as it is, the analysis time only becomes long.

  Therefore, in this embodiment, the pixel number conversion circuit 18 (FIG. 2) is used to reduce the number of pixels of the analysis target image data prior to image analysis. For example, if an accurate analysis result can be obtained even with 300,000 pixels, conversion from 1 million pixels to 300,000 pixels is performed. In some cases, 100,000 pixels are acceptable, or 600,000 pixels must be present. The number of pixels to be reduced or the number of pixels to be reduced depends on the number of effective pixels of the image sensor 12, the size of the search frame F, the specification of the image recognition program, the processing speed of the CPU 11, and the like. It is desirable to go and decide. Ideally, it is desirable to reduce to the minimum number of pixels that can provide an accurate analysis result.

  In addition, when the size of the search frame F is variable, the initial number of pixels of the analysis target image data is not constant, and the reduction rate may be determined according to the initial number of pixels. For example, the number of pixels may not be reduced when the initial number of pixels of the analysis target image data is below a certain level. Alternatively, the reduction rate may be constant regardless of the size of the search frame F.

  The pixel number conversion itself can be performed in a short time using a known technique. For example, a method of thinning out pixels at a predetermined rate or a method of averaging the colors of a plurality of adjacent pixels as one pixel can be considered.

  In this way, the number of pixels in the image data to be analyzed is reduced prior to image analysis, so even with a high-pixel electronic camera, flower features can be extracted in a short time and the search can be completed quickly. It becomes.

  When the feature data of the flower (color, shape, number of petals, etc.) of the flower is extracted, the CPU 11 searches the pictorial book database 17 using these as search keys, and extracts ecological information and sample images suitable for the conditions. Depending on the format of the pictorial book database 17, in addition to the feature data, other data can be used as a search key. For example, when adding “flower blooming season” as a search key, a GPS receiver is installed in the camera to acquire position information, and the flower blooming season (spring, summer, autumn, winter) from the position information and date data at the time of shooting. ).

  When the search is completed, the CPU 11 displays the search result on the liquid crystal monitor 2. FIG. 4 shows a display example, which is an example in which one type of flower can be specified based on the features extracted from the analysis target image data of FIG. The liquid crystal monitor 2 can display flower names from the first candidate to the fifth candidate. Note that ecological information other than the name may be displayed together.

  In the example of FIG. 3, one flower is included in the analysis target image data, and the other is leaves and soil. Therefore, in the image analysis, a flower part and a non-flower part can be accurately distinguished, and an accurate flower feature can be obtained. Can be extracted. If the feature data is accurate, the candidate can be narrowed down to one by searching as shown in FIG. 4, and the reliability of the search result is extremely high.

On the other hand, FIG. 5 shows an example of photographing a flower having a shape completely different from that in FIG. 1, and FIG. 6 shows analysis target image data cut out from the image data in FIG. The shape, size, and position of the search frame F are the same as those in FIG. The image recognition program is based on the premise that only one flower is included in the image data to be analyzed, and in the case of a spike-shaped flower as shown in FIG. 5, a collection of a plurality of flowers blooming on one flower stem is used. Recognize two flowers. However, in this example, two flower stems, that is, two flowers of the same kind, are included in the search frame F, so that two flowers are recognized as one flower, and the extracted feature data is actually There is a risk that it will be far from the characteristics of. In addition, in this example, a part of the target flower is missing, and there is a possibility that an accurate feature cannot be extracted from this point.
FIG. 6 shows an example in which two flowers of the same type are included. However, when two or more flowers of different types are included, the extracted feature data is more inaccurate.

  When inaccurate feature data is used as a search key in this manner, the reliability of the search result is naturally low. For example, a plurality of candidates are displayed as shown in FIG. 7, or “not applicable” as shown in FIG. It may be displayed. In either case, the name of the flower cannot be identified, so it must be searched again.

  In order to make the feature data accurate, the desired flower is completely placed in the search frame F, and at the same time unnecessary ones (those that have a negative effect on feature extraction: different types of flowers, etc.) are found in the search frame F. It is necessary not to enter. This may be achieved to some extent if the composition is devised at the time of shooting, for example, but this often does not result in the composition intended by the photographer. The quality of the composition does not matter if it is taken for performing flower information retrieval. However, if it is desired to perform a flower retrieval with image data that remains as a work, the composition cannot be changed for the flower information retrieval. Also, when flowers are densely blooming, you can't exclude unwanted flowers no matter how you change the composition.

  Therefore, in this embodiment, the shape, size, and position of the search frame F can be changed according to the shape of the flower. FIG. 9 shows an example of a selection frame F selection screen, and this screen is displayed on the liquid crystal monitor 2 prior to the search (display timing will be described in a later-described flow). In this example, six types of sample frames F1 to F6 having different shapes are displayed as selection candidates. Selection is performed using the four-way button 4 and the selected sample frame is highlighted. The selection is determined by operating a determination button (not shown), for example.

  For example, when the sample frame F1 is selected, a circular search frame F is displayed superimposed on the image as shown in FIGS. As shown in FIG. 1, when the overall shape of the flower is circular or close to a regular polygon, this circular search frame F is suitable. On the other hand, in the spike-shaped flower as shown in FIG. 5, unnecessary things are easily included in the circular frame F, and the vertically long frame F3 or F6 is appropriate. In addition, when taking a picture of a large chrysanthemum from the side, a horizontally long frame F2 or F5 is appropriate. Note that the types of selectable sample frames are not limited to those shown in FIG.

  FIG. 10 shows an example in which the sample frame F6 is selected for the spiked flower. Prior to the search, the CPU 11 displays a message for prompting the search frame F to match the flower as illustrated. To match the search frame F to the flower, the position and size of the search frame F are changed. In the present embodiment, the position of the search frame F can be changed up, down, left, and right in the screen in accordance with the up / down / left / right operation of the four-way button 4. The search frame F can be enlarged / reduced by operating the zoom up / down buttons 3U, 3D. For enlargement / reduction, only the area may be changed while maintaining the aspect ratio of the frame F, or the aspect ratio may be freely changed. In this way, the position and size of the search frame F can be changed with the existing four-way button 4 and zoom up / down buttons 3U, 3D, so there is no need to provide a dedicated operation member, and the number of parts can be reduced. Reduction can be achieved.

  FIG. 11 shows a state in which the search frame F is aligned with one flower by the above operation, and FIG. 12 shows analysis target image data cut out by the search frame F. As can be seen from FIG. 12, the analysis target image data includes only one flower and does not include unnecessary ones. Therefore, accurate flower feature data can be extracted and an accurate search result can be obtained. . In addition, if an unnecessary thing enters the corner of the rectangular frame F6, the elliptical frame F3 may be selected.

  As described above, in this embodiment, the shape, size, and position of the search frame F can be changed in accordance with the shape of the flower, so that any flower can be stored in the search frame F and unnecessary. It can be removed from the search frame F, and the possibility of obtaining an accurate search result can be increased. Further, it is not necessary to change the composition depending on whether or not the flower information is searched.

13 and 14 show an example of a processing procedure for realizing the above-described flower information search in software. Note that illustrations of processes not related to the present invention are omitted.
When the camera is turned on, this program is activated by the CPU 11 and waits for any one of the release button 1, the playback button 5, and the search button 6 in a loop of steps S1 to S3. During this time, the above-described through image display is performed on the liquid crystal monitor 2 by a process (not shown), and the search frame F is also displayed. The shape, size, and position of the search frame F displayed are in accordance with the previous selection setting. The photographer decides the composition so that the target flower is in the search frame F when shooting based on the flower information search. Note that the shape, size, and position of the search frame F can be changed later, so the target flower does not necessarily have to be placed in the search frame F. Alternatively, the search frame F may not be displayed in the through image display.

  When the operation of the release button 1 is detected, imaging is performed in step S4 to generate image data, and the image data is recorded in the large capacity memory 14 in step S5. When the operation of the playback button 5 is detected, the latest image data recorded in the large-capacity memory 14 is read into the finder memory 16 and displayed on the liquid crystal monitor 2 in step S6. Although not shown, image data to be displayed can be changed by a predetermined operation.

  When the operation of the search button 6 is detected, it is determined in step S7 whether image data is currently displayed. If the search button 6 is operated after the playback button 5 is operated, step S7 is affirmed and the search process is performed in step S8.

Details of the search processing in step S8 are shown in FIG.
In step S101, a frame selection screen as illustrated in FIG. 9 is displayed on the liquid crystal monitor 2, and a frame selection operation or a determination operation is waited in a loop of steps S102 and S103. The frame selection operation refers to an operation of selecting a desired frame with the four-way button 4 as described above. When the operation of the four-direction button 4 is detected, step S102 is affirmed, and the selection frame (frame to be highlighted) is changed in the direction corresponding to the operation in step S104. When the determination operation is detected, the selection of the highlighted frame is determined, and the process proceeds to step S105.

  In step S105, the image data when the search button 6 is pressed is displayed again, the search frame F selected and determined is displayed superimposed on the image data, and a message prompting the user to match the search frame F to a flower is displayed. Display (for example, FIG. 10). Next, a loop of steps S106 and S107 is performed, and an operation for changing the size (frame size) and position of the search frame F or a determination operation is awaited. As described above, the frame size is changed by operating the zoom up / down buttons 3U and 3D, and the position is changed by operating the four-way button 4. When these operations are detected, the frame size or position is changed in accordance with the operations in step S108. When the determination operation is performed, the frame size and position are determined and the process proceeds to step S109.

  In step S109, the region surrounded by the search frame F in the displayed image data is cut out as analysis target image data. In step S110, an instruction is given to the pixel number conversion circuit 18, and processing for reducing the number of pixels in the cut-out analysis target image data is performed. The contents are as described above. In step S111, the analysis target image data after the reduction in the number of pixels is analyzed to extract the feature of the flower. In step S112, the pictorial book database 17 is searched using the extracted feature data as a search key. In step S113, the search result is displayed and a confirmation message as to whether or not to perform a re-search is displayed (see, for example, FIGS. 4, 7, and 8).

  Next, a loop of steps S114 and S115 is performed, and an operator response to the re-search message is waited for. If a Yes operation (upper operation of the four-way button 4) is detected, the process returns to step S101. If a No operation (lower operation of the four-way button 4) is detected, the process returns to the process of FIG.

  In the above, the shape, size, and position of the search frame F are all variable, but at least one of them may be variable. For example, the shape and position may be fixed and only the size may be changed, or only the position may be changed.

  In the above example, the flower information search is started by pressing the play button 5 and the search button 6 after shooting. For example, when the release button 1 is operated in a state where the search mode is set in advance, the shooting is performed after shooting. The flower information search may be automatically started using the image data. Alternatively, by operating the release button 1 while pressing the search button 6, the shooting and searching may be performed in the same manner.

  The pictorial book database 17 and the search function are not necessarily provided in the camera. For example, a system may be considered in which the picture book database 17 and the search engine are placed on a search server on the network, and when the electronic camera transmits a search key to the search server, the search engine searches and sends the result back to the camera.

  The object of information retrieval is not limited to flowers, and the present invention can be similarly applied to information retrieval such as insects and birds. The present invention is applicable to any device having an electronic camera function (for example, a mobile phone with a camera function).

The figure which looked at the electronic camera which concerns on one Embodiment of this invention from the back. The block diagram which shows the control system of an electronic camera. The figure which shows an example of the analysis image data cut out from the imaged image data. It is a figure which shows the search result in flower information search, and shows the example by which only one candidate was displayed. The figure which shows the imaged image data. The figure which shows the analysis object image data cut out from the image data of FIG. It is a figure which shows the search result in flower information search, and shows the example by which the several candidate was displayed. It is a figure which shows the search result in flower information search, and shows the example of a display when applicable data are not found. The figure which shows an example of a search frame selection screen. The figure which shows an example of the screen prompting to match a search frame with a flower. The figure which shows an example of a screen when a search frame is match | combined with the flower. The figure which shows the analysis object image data (area | region enclosed with the frame) cut out from the image data of FIG. The flowchart explaining operation | movement of the electronic camera containing a flower information search. 14 is a flowchart showing details of the information search process in FIG. 13.

Explanation of symbols

6 Search button 11 CPU
12 Image sensor 17 Picture book database 18 Pixel number conversion circuit F Search frame F1 to F6 Sample frame

Claims (7)

Imaging means for imaging a subject and generating image data;
An area designating unit for designating an arbitrary area of the image data according to an operation;
An imaging apparatus comprising: feature extraction means for extracting a feature of a subject based on image data of the designated area and outputting the feature as a search key in subject information search.   The imaging apparatus according to claim 1, wherein the area specifying unit includes a unit that specifies a size of the area.   The imaging apparatus according to claim 1, wherein the area specifying unit includes a unit that specifies a shape of the area.   The imaging apparatus according to claim 1, wherein the area specifying unit includes a unit that specifies a position of the area.   The area specifying means includes means for selecting a frame representing the shape of the area, and means for specifying the size and position of the selected frame on the image data. The feature extracting means includes the frame The photographing apparatus according to claim 1, wherein the feature of the subject is extracted based on the image data of the area surrounded by.   The area specifying means includes means for selecting any one of a plurality of frames according to the shape of the subject, and the feature extracting means adds image data in an area surrounded by the selected frame on the image data. The photographing apparatus according to claim 1, wherein a feature of the subject is extracted based on the subject.   7. The apparatus according to claim 1, further comprising: search means for searching the subject information from a database based on the extracted features; and notification means for notifying the searched subject information. The imaging device described in 1.

Images