JP6063536B2 - IMAGING DEVICE AND IMAGING DEVICE CONTROL METHOD - Google Patents

Description

  The present invention relates to an imaging apparatus and a control method for the imaging apparatus that can search a wide range of similar images and enjoy them widely.

  In recent years, taking and capturing of digital images and image processing techniques have been improved, and boundaries between digital cameras and video cameras have been lost. In addition, it has become easier for individuals to handle a large number of images. Furthermore, as digitalization progresses, cameras that have traditionally been used only for photography have become easier to exchange image data with other devices such as cameras, personal computers, and televisions. It is also required to function as.

In personal computers and servers on the Internet, it has been generally performed to search for images and the like from an early stage, and reproduce and view these images. Even under such circumstances, the mainstream of search is natural language search, and if a correct keyword is assigned to data such as an image, the search is possible, but the correct keyword is not assigned. In case it was difficult to search. In particular, when searching to overseas sites, there was a problem of language, which was quite difficult.

  Therefore, for example, Patent Document 1 discloses information that allows a user to share a wide range of information by absorbing differences in languages and browsing environments between users in a system for disclosing and sharing information on the Internet. A public system is disclosed. That is, when a browsing request for public information is generated from a viewer, this information disclosure system creates a browsing page with translation data suitable for the viewer and returns it to the viewer. Further, Patent Document 2 discloses a homepage information collecting method that efficiently collects homepage information, and searches are made not only with words but also with attributes such as geographical information.

Japanese Patent Laying-Open No. 2005-92838 Japanese Patent No. 3620996

As described above, various proposals have been made in order to collect information even in different languages, but the language itself is targeted. That is, in the information disclosure system disclosed in Patent Document 1, it is difficult to have a dictionary database for all languages. Further, the homepage information collection method disclosed in Patent Document 2 is aimed at increasing the amount of information and improving the accuracy, and does not correspond to the increase in the number of languages. Therefore, there is a limit as long as an image is searched by the language itself, and it is not enough to search and enjoy a photograph image taken by a foreign user. Also, if the user accidentally assigns a keyword that does not match the captured image, or if the image is received from another user and the keyword is different, a similar image may be searched. Have difficulty.

The present invention has been made in view of such circumstances, and an object of the present invention is to provide an imaging apparatus and a method for controlling the imaging apparatus that can easily search for and enjoy similar images from captured images .

In order to achieve the above object, an image pickup apparatus according to a first aspect of the present invention uses an image pickup unit that photoelectrically converts an optical image and outputs image data, and uses the image data to quantify the feature amount of the image to obtain feature information. a feature quantity quantifying section for calculating, a recording unit for recording the characteristic information in association with the image data, the image similar to the image based on the image data, and search by using the feature information, is the search and if the similar image does not reach the predetermined number, the table Shimesuru display an image retrieval unit for retrieving a similar image until the predetermined number by using a different characteristic information, the retrieved similar image from the feature information Part.

An image pickup apparatus according to a second aspect of the present invention includes the communication unit according to the first aspect, wherein the image data and the feature information are transmitted to the outside of the image pickup apparatus.
In the imaging device according to a third invention, in the first invention, the feature value digitizing unit digitizes at least one feature value of a color and a contour included in the image.
According to a fourth aspect of the present invention, in the first aspect of the invention, the screen is divided and recorded with respect to the discrete feature points representing the edges of the contour points of the image in the screen and the color distribution.

An imaging apparatus according to a fifth aspect of the present invention is the imaging apparatus according to the first aspect, wherein the imaging unit has an operation unit for instructing a shooting operation, and is output from the imaging unit when the shooting unit is instructed by the operation unit. The image data and the feature information calculated by the feature value digitizing unit are associated with the image data and recorded in the recording unit.
An image pickup apparatus according to a sixth invention has the character input unit for inputting character information in the first invention, and is input at the character input unit in association with the image data recorded in the recording unit. Record the character information.
Imaging device according to the seventh invention, in the sixth invention, based on the character information, that having a character search unit for searching for similar images.

An image pickup apparatus according to an eighth aspect of the present invention is an image pickup unit that photoelectrically converts an optical image and outputs image data, and a feature value quantification that calculates feature information by quantifying the feature value of the image using the image data. And a recording unit that records the feature information in association with the image data, and an image similar to the image based on the image data is searched using the feature information, and the searched similar images are in a predetermined number. An image search unit that searches for similar images until the predetermined number is reached using feature information different from the feature information .

According to a ninth aspect of the present invention, there is provided a control method for an image pickup apparatus, photoelectrically converting an optical image, outputting image data, using the image data, quantifying a feature amount of the image, calculating feature information, and calculating the image data. The feature information is recorded in association with the image data, an image similar to the image based on the image data is searched using the feature information, and if the number of similar images searched does not reach a predetermined number, the feature information display the similar image searching similar images until the predetermined number, which is search with different characteristic information from.

According to the present invention, it is possible to provide an imaging apparatus and a method for controlling the imaging apparatus that can easily retrieve and enjoy similar images from captured images .

It is a block diagram which shows the electric circuit of the camera concerning 1st Embodiment of this invention. 2A and 2B show a display on a liquid crystal display unit of a camera according to the first embodiment of the present invention, in which FIG. 1A shows a state in which an electronic keyboard is displayed, and FIG. 2B shows a state in which an image and a comment are input. It is a figure and (c) is a figure which shows the state which displayed the similar image. It is a figure which shows the data structure inside the recording part of the camera concerning 1st Embodiment of this invention. It is a flowchart which shows the operation | movement of the camera control of the camera concerning 1st Embodiment of this invention. It is a flowchart which shows the operation | movement of the image reproduction of the camera concerning 1st Embodiment of this invention. It is a block diagram which shows the structure of the image search system concerning the 1st technique relevant to this invention. It is a figure which shows a search result in the image search system concerning the 1st technique relevant to this invention. It is a flowchart which shows the operation | movement of the image display of the image search system concerning the 1st technique relevant to this invention. It is a flowchart which shows the operation | movement of the reference image display of the image search system concerning the 1st technique relevant to this invention. In the image search system concerning the 1st technique relevant to the present invention, it is a figure explaining how to extract the feature point of an image, (a) shows the picture used as extraction object, and (b) and (c) are one. It is the figure which expanded the part, (d) (e) is a figure which shows a feature point. It is a figure which shows the extracted feature point in the image search system concerning the 1st technique relevant to this invention. It is a figure which shows the feature pattern obtained based on the extracted feature point in the image search system concerning the 1st technique relevant to this invention. In the image search system according to the first technique related to the present invention, other images and feature patterns are shown, (a) shows other images, and (b) shows feature patterns of other images. In the image search system according to the first technique related to the present invention, a feature pattern is shown, (a) is a diagram in which the feature patterns of two images are superimposed, and (b) is a state in which the feature patterns are shifted and matched. (C) is a figure which shows the shift direction of a feature pattern. In the image search system according to the first technique related to the present invention, examples of dissimilar images are shown, (a) shows still another image, and (b) shows a feature pattern of the other image. In the image search system according to the first technique related to the present invention, examples of similar images are shown, (a) shows still another image, and (b) shows a feature pattern of the other image. It is a figure which shows the relevance of a keyword in the image search system concerning the 1st technique relevant to this invention. It is a flowchart which shows the operation | movement of the display image determination of the image search system concerning the 2nd technique relevant to this invention. It is a figure which shows the search screen in the image search system concerning the 2nd technique relevant to this invention.

  Hereinafter, a first embodiment and a preferred embodiment of the present invention will be described using an image search system to which the present invention is applied according to the drawings. The first embodiment of the present invention is an example applied to a camera 10. The camera 10 can search for an image recorded in an external server or the like via a communication unit. In the image search, first, an image is searched by a keyword or the like, common feature information of similar images searched by this is extracted, and a similar image is further searched using the extracted feature information. When searching for similar images using feature information, no keyword or the like is required, so that images with a different language can also be searched.

  FIG. 1 is a block diagram showing a configuration of a camera 10 according to the first embodiment of the present invention. The camera 10 includes a control unit 1, a photographing unit 2, a face detection unit 2 b, a recording unit 3, a time / position determination unit 3 c, a character search unit 4, an image search unit 5, a feature value digitizing unit 5 a, a communication unit 6, and a playback unit. Unit 7, liquid crystal display unit 8, operation unit 9, and character input unit 9a.

  The control unit 1 includes a CPU (Central Processing Unit) and the like, is connected to each unit of the camera 10, and performs overall control according to a built-in program. The operation unit 9 includes operation members such as a release button and a menu button. When the user operates the operation unit 9, the control unit 1 outputs a control signal to each unit according to the operation.

  The photographing unit 2 includes an optical system for forming a subject image, an image sensor that photoelectrically converts the subject image, an image processing unit that processes an image signal output from the image sensor, and the like, and outputs image data. The photographing unit 2 can output image data of still images and moving images (including continuous shooting).

  The face detection unit 2b detects whether or not a face is included in the image data based on the image data acquired by the photographing unit 2, and if a face is included, the position is detected. Is detected. Note that face detection is performed by a pattern matching method or the like. Here, the detected face information is recorded together with the image data in the recording unit 3.

  The recording unit 3 records still image and moving image data output from the imaging unit 2. The reproduction unit 7 reproduces the image data recorded in the recording unit 3, and the liquid crystal display unit 8 displays the image reproduced by the reproduction unit 7. The liquid crystal display unit 8 performs not only reproduction of the image data recorded in the recording unit 3 but also so-called live view display for displaying a subject image for observation during shooting. At the time of reproduction, an image search result is also displayed.

  The time / position determination unit 3c includes a clock unit for measuring the date and time, a GPS (Global Positioning System) for measuring the position, and the like, and determines a shooting time and a shooting position. When recording the image data, the recording unit 3 records the shooting time and the shooting position in association with the image data.

  The character input unit 9a has a touch panel or the like disposed on the display surface of the liquid crystal display unit 8, reads characters written on the touch panel, and displays electronic data as shown in FIG. The keyboard 21 is displayed, and character input is performed by touching the electronic keyboard 21. This character input unit 9a gives keywords, titles, and the like to the photographed photos. When a character or the like is input by the character input unit 9a, for example, an image title or a comment is input to the input display unit 25 as shown in FIG.

  A character information recording unit 3b is provided in the recording unit 3, and the character information recording unit 3b records the type of character. That is, kanji, hiragana, katakana, alphanumeric characters, and the like are selected from the character input unit 9a and characters are input, and the type of the selected character is recorded. In the example shown in FIG. 2A, numerals 0 to 9 are displayed on the electronic keyboard 21, but by touching the “switch” unit 23, the keyboard display is switched to numerals, alphabets, and the like. Further, by touching the “conversion” unit 24, the characters are converted into kanji and the like, and the input characters are displayed on the input display unit 25. If it is difficult to understand with characters, an icon 26 representing the theme is displayed. The icon 26 may display the result of the scene classification automatically performed by the camera 10. In the example of FIG. 2B, the portrait mode is selected. Of course, the shooting mode manually set by the user at the time of shooting may be reflected.

  The feature value digitizing unit 5a digitizes feature values such as colors and contours included in the image recorded in the recording unit 3. The feature information digitized here is recorded together with the image data. The communication unit 6 is a transmission / reception unit for transmitting image data and the like to the outside. Transmission / reception methods include wireless communication, infrared communication, wireless LAN, wired communication using a USB terminal, and the like. Via this communication unit 6, image data recorded in the recording unit 3, the above-described photographing date / time information, characteristic information indicating image characteristics, and the like are transmitted to the outside.

  The character search unit 4 performs an image search based on character data (also referred to as a keyword) attached to an image recorded in the recording unit 3. As described above, the character data is input by the character input unit 9a, and the character data is associated with the image data and assigned to the image data, and is recorded in the recording unit 3. The character search unit 4 searches for an image based on the assigned character data. The image search unit 5 performs an image search based on feature information given to the image recorded in the recording unit 3. As described above, since the feature of the image is digitized by the feature value digitizing unit 5a and recorded together with the image data, an image search is performed based on this feature information.

  The search area 3 a is a search area when the character search unit 4 and the image search unit 5 perform an image search. Since it takes a very long time to search for information in an image for each image, a search area 3a is provided in the recording unit 3, and a search is performed from data in a list format stored therein. ing.

  FIG. 3 shows an internal data structure of the recording unit 3, and an image file 32 to which an address 31 is allocated is recorded in the recording unit 3. In the recording unit 3, there are provided a search area 3a and a character information recording unit 3b.

  In the search area 3a, for the image data recorded in the recording unit 3, an image address and related data of the image recorded therein, for example, shooting information such as a title, a comment, a shooting date and time, a position, Thumbnail images, feature information digitized by the feature value digitizing unit, and the like are stored in a table format. Using the information stored in the search area 3a in the form of a table, the data related to the image can be easily searched.

  The related data described above is called tag data because it can be used for searching and the like. A tag is a small tag attached to an object, and a special character string in which additional information is embedded in an image or the like is called a tag.

  In the character information recording unit 3b, character information is stored in relation to the position of the button displayed on the electronic keyboard and each character, for example. The control unit 1 determines which character is input in which mode based on information recorded in the character information recording unit 3b. In addition, not only a character but the icon 26 may be used together like a pictograph as shown in FIG.

  In this embodiment, as will be described later, an image search is possible. For example, when the keyword “rose” is input, an image related to the rose is displayed on the liquid crystal display unit 8 as shown in FIG. Is done. At this time, not only the images 27a and 27b to which “rose” is assigned as a keyword (character data) but also a similar image 28 is searched based on the feature information even if the images do not match as keywords. And displayed as a reference image.

  As described above, in the present embodiment, when a user mistakenly assigns a keyword that does not match a captured image, or an image received from another user even if the keyword is different, it is similar. If it is an image, it is searched. For this reason, the memory which has been forgotten can be revived, further image search can be performed from unexpected images, and other users can be contacted.

  Next, the operation of the camera 10 in the present embodiment will be described with reference to the camera control flowchart shown in FIG.

  If the camera control flow is entered, it is first determined whether or not the camera is in shooting mode (S101). The camera 10 can be set in three modes: a shooting mode, a playback mode, and a reception mode. If the result of determination in step S <b> 101 is that it is in shooting mode, the flow proceeds to shooting preparation.

  When the preparation for photographing is started, first, an image is captured (S102). In this step, image data is captured by the photographing unit 2, and live view display is performed on the liquid crystal display unit 8 based on the image data. The live view display allows the user to check the shooting range and composition. In addition, the face detection unit 2b uses the captured image data to detect whether or not a face portion exists, and if it exists, the position on the screen. Information about the detected face is recorded together with the shooting.

  Once the image has been captured, it is next determined whether or not a release operation has been performed (S103). The user operates the release button in the operation unit 9 when the subject image intended by the photographer is obtained while observing the live view display. Therefore, in this step, it is determined whether or not the release button has been operated. If the result of determination in step S103 is not release, processing returns to step S101 and the above-described processing is performed.

  On the other hand, if the result of determination in step S103 is that the shutter has been released, shooting is performed (S104). Here, the image data from the photographing unit 2 at the timing when the release operation is performed is recorded in the recording unit 3. Subsequently, the feature of the image is extracted (S105). In this step, the feature quantity digitizing unit 5a digitizes the image features, and the feature information is associated with the image data and recorded in the recording unit 3.

  Subsequently, it is determined whether a comment is input or an icon is set (S106). When shooting a subject and recording an image, a comment regarding the image can be input. In other words, when the release and shooting are finished, the electronic keyboard 21 as shown in FIG. 2A is displayed on the liquid crystal display unit 8. Using this electronic keyboard 21, the user can input comments and keywords as if they were written by a word processor. The icon is automatically set according to the scene classification, shooting mode, etc., but may be set manually.

  Further, by touching the switching unit 23 on the electronic keyboard 21, it is possible to switch to the icon screen. If various themes such as “person”, “flower”, “pet”, etc. are prepared as icons, the arrangement becomes easy. The icon may be automatically selected based on the feature information, but if it is not selected, manual input may be performed.

  As a result of the determination in step S106, if no comment or icon has been input, the process proceeds to step S118. On the other hand, if an input has been made, a comment or icon is input based on this operation state ( S107). Subsequently, thumbnails are created and comments are recorded (S108). In this step, the comments (including keywords) and icons input in step S107 are recorded in association with the image data, and data for thumbnail display is created and recorded.

  Once thumbnail creation / recording has been carried out, it is next determined whether or not it is in transmission mode (S118). Transmission is performed via the communication unit 6. In this step, it is determined whether or not the camera 10 is manually set to the transmission mode and an operation member such as a transmission button is operated. If it is determined that transmission is to be performed, transmission is performed (S119). In this step, the image data recorded in the recording unit 3 and related information associated therewith are transmitted to an external camera or other device via the communication unit 6. Of course, it may be transmitted to an external server via a hot spot.

  If the result of determination in step S118 is not transmission, or if transmission in step S119 is executed, this flow is terminated, and the process is executed again from step S101.

  If the result of determination in step S101 is not shooting mode, it is determined whether or not playback mode is in effect (S111). If the result of this determination is playback mode, image playback is performed (S112). In this step, the selected image is displayed. In addition to this normal reproduction display, an image similar to the selected image is searched using keywords and feature information, and as shown in FIG. Is displayed on the liquid crystal display unit 8. This image reproduction subroutine will be described later with reference to FIG. When the image reproduction is finished, this flow is finished and executed again from step S101.

  If the result of determination in step S111 is not playback mode, it is determined whether or not it is reception mode (S131). This camera 10 can acquire image data from a device such as another camera via the communication unit 6. In this step, it is determined whether or not image data is received from another device.

  If the result of determination in step S131 is not reception mode, various settings are made (S134). In this step, the user can set various shooting modes, such as a mode for shooting a subject beautifully. As described above, when the shooting mode or the like is set, an icon related to the selected mode may be recorded in the tag data. After setting, the process returns to step S101.

  If the result of determination in step S131 is that it is in reception mode, image acquisition is performed (S132). That is, the image data received by the communication unit 9 is recorded in the recording unit 3. Subsequently, feature extraction is performed on the received image data recorded in the recording unit 3 (S133). In this step, feature information is obtained by the feature value digitizing unit 5a for the image data of the received image recorded in the recording unit 3.

  When the feature extraction is performed in step S133, the process proceeds to step S106, and as described above, a comment (including a keyword) can be input and an icon can be selected and recorded in the recording unit 3 in association with the image data.

  As described above, in the camera control flow, a comment including a keyword or the like is input to a captured image or an image obtained from another user, an icon is selected, and the information is associated with the image data. Can be recorded. Further, image feature information can be obtained, and the feature information can also be recorded in association with image data.

  Next, the image reproduction subroutine in step S112 will be described with reference to the flowchart shown in FIG.

  First, a captured image is displayed (S151). In this step, images recorded in the recording unit 3 including images received from other users are displayed on the liquid crystal display unit 8 in a thumbnail format. Subsequently, it is determined whether or not the search mode is set (S152). Since the search mode is set on the menu screen or the like by the operation unit 9, it is determined in this step whether or not the search mode is set.

  If the result of determination in step S152 is search, tag search display is performed (S171). In this step, as shown in FIG. 2A, an input screen for inputting words, numbers, symbols and the like corresponding to the tag is displayed. Using this input screen, the user can input keywords, icons, feature information, etc. to be searched.

  When a search keyword or the like is input on the tag search display screen, it is subsequently determined whether or not there is a similar tag (S172). In this step, the character search unit 4 uses a keyword or the like to determine whether there is an image with a similar tag, and the image search unit 5 uses the feature information to determine whether there is a similar image. If the result of determination in step S172 is that there is no image with a similar tag, a warning is displayed on the liquid crystal display unit 8 (S181). When the warning is displayed, the original flow is restored.

  If the result of determination in step S172 is that there is a similar tag, the retrieved image is displayed as shown in FIG. 2B (S173). Subsequently, it is determined whether there are a plurality of searched images (S174). As a result of the determination, if there are not a plurality of searched images, the process proceeds to step S176. On the other hand, if there is a similar image, features common to a plurality of images including the original image are extracted (S175).

  The common feature extraction performed in step S175 will be described in detail in the first technique related to the present invention, using FIG. 10 to FIG. When images as shown in FIGS. 15A and 16A are searched as similar images, there are feature patterns shown in FIGS. 15B and 16B, and the upper half of the screen is similar to both. Therefore, a pattern distribution (see FIG. 14B) having a high degree of coincidence when overlapped is a common feature. In this way, the determination is made by determining the matching degree of the feature pattern.

  Next, it is determined whether or not a predetermined number of images can be displayed as a result of the search on the liquid crystal display unit 8 (S176). As a result of the determination, if a predetermined number of images are displayed, images that can be enjoyed sufficiently are searched, and the process proceeds to step S153. On the other hand, if a predetermined number of images cannot be displayed, further images are searched to increase the number of displayed images. First, it is determined whether a tag different from the tag searched in step S171 is assigned to the searched image (S177).

  If the result of determination in step S177 is that there is no image to which another tag has been added, processing proceeds to step S181 described above and warning display is performed. On the other hand, if there is another tag, a similar image is searched using the other tag (S178). In this step, if the extracted common feature information exists in step S175, it is determined whether or not a similar image exists using this feature information. In addition, similar images may be searched using tags such as comments and keywords common to the similar images detected in step S174.

  The similar image retrieved in this step is displayed on the liquid crystal display unit 8 (S179). When the display is performed, the process returns to step S176, and steps S177 to S179 are repeated until the predetermined number is reached (however, these steps end when another tag does not exist).

  If the result of determination in step S152 is not retrieval, normal image reproduction is performed. In step S176, a predetermined number of images are retrieved as a result of image retrieval, and search images are retrieved as shown in FIG. When is displayed, the next searched image is reproduced. First, it is determined whether any of a plurality of images displayed on the liquid crystal display unit 8 is selected (S153).

  If the result of determination in step S153 is that the touch panel has been touched and any image has been selected, that image is enlarged and displayed (S154). Subsequently, it is determined whether a return operation has been performed by an operation such as touching the image again (S155). If a return operation has been performed, the image size is returned to the original size (S156). After returning to the original size and displaying thumbnails, the process returns to step S153.

  If any image is not selected as a result of the determination in step S153, it is next determined whether or not the user wants to see another image (S161). In this case, the user touches the next candidate icon. If another image (next candidate) is selected as a result of this determination, the next candidate is displayed as a list in thumbnail display (S162). The next candidate for the already displayed image is displayed, and the process returns to step S151 to execute the above-described operation.

  As described above, in this embodiment, the character search unit 4 or the image search unit 5 searches for an image that the user wants to search using the tag attached to the image (S171). A common feature is extracted from the plurality of images (S175). The feature information extracted here is used to search for an image that the user wants to search again (S178). As described above, the image that the user wants to search can be searched and displayed even other than the keyword, and a large number of images are searched and displayed, and the display screen becomes lively. Using the camera is fun and unexpected images are also displayed, which can contribute to stimulating thoughts and promoting communication.

  In the present embodiment, the digital camera is used as the photographing device. However, the camera may be a digital single-lens reflex camera or a compact digital camera, and may be a mobile phone or a personal digital assistant (PDA). A camera built in Personal Digital Assist) may be used.

  In the present embodiment, the camera 10 has both a photographing function and a playback / search display function. However, among these, only the playback / search display function may be executed by a device such as a personal computer. In this case, the image data and related data recorded in the recording unit 3 of the camera 10 may be installed in a device such as a personal computer and played back or searched. The feature information may also be extracted by a device such as a personal computer.

  Next, a first technique related to the present invention will be described with reference to FIGS. The image search system according to the first embodiment of the present invention is an example applied to the camera 10. The image search apparatus according to the first technique related to the present invention is configured by a system in which a plurality of servers cooperate via the Internet. Different cultures and languages cannot be used to easily search for images with the same keyword, but if the image search system of this technology is configured on a global scale, it will be very effective.

  FIG. 6 shows a configuration of the present technology, which includes a plurality of servers 90, 100, and 110 that can be accessed with each other via the Internet. The server 100 includes a control unit 101, a net access unit 102, an image recording unit 103, and a translation unit 104 therein.

  The control unit 101 is connected to each unit in the server 100 and controls the entire server 100. The net access unit 102 can access external servers 90 and 110 and the like via the Internet. The image recording unit 103 records an image transmitted from a personal computer such as the user 41. The translation unit 104 translates a keyword assigned to a tag or the like when searching for an image recorded in the image recording unit 103 or an external server.

  The server 110 includes a control unit 111, a net access unit 112, and an image recording unit 113 therein. Similar to the control unit 101, the control unit 111 controls the server 110, and the net access unit 112 accesses other servers and the like via the Internet, like the net access unit 102. The image recording unit 113 records an image transmitted from the personal computer of the user 42 or the like.

  The server 90 is a core server in the present technology, and includes a control unit 91, a net access unit 92, a tag search unit 93, a feature extraction unit 94, a similar image search unit 95, an area / language determination unit 96, And a display image determination unit 97.

  The control unit 91 controls the entire server 90 in the same manner as the control units 101 and 111, and the net access unit 92 accesses an external server or the like through the Internet in the same manner as the net access units 102 and 112. Communicates when accessed from other servers. The image recording unit is not explicitly shown in the figure, but is provided in the server 90 and records image data accessed from the outside.

  The tag search unit 93 searches for a tag attached to the image data for the image recorded in the image recording unit in the server 90. As in the first embodiment of the present invention, a search area may be provided, tag data may be arranged in a list format, and this may be searched. The feature extraction unit 94 extracts features from the image recorded in the image recording unit. The similar image search unit 95 searches for similar images based on the features extracted by the feature extraction unit 94.

  The display image determination unit 97 determines a display image such as an image searched by the tag search unit 93, the feature extraction unit 94, the similar image search unit 95, and the like. The area / language determination unit 96 determines the language and installation area mainly used by the other servers 100 and 110. In the present technology, when the region / language determination unit 96 determines that the search by words is not possible, the similar image search unit 95 can search for a similar image based on the feature information of the image.

  The servers 100 and 110 are local servers, and images in the server can be searched according to the language of the country where the servers are installed, but searching in a foreign language is not basically supported. The server 90 can search for images in the external server in various languages, and can search the images themselves by the feature extraction unit 94 and the similar image search unit 95. For example, when a foreign user 42 performs an image search in their own language and displays a search image 43 on a personal computer, an image 44 published on the web by a Japanese user 41 can also be searched and displayed. Of course, the image 44 is not assigned a keyword in the language used by the foreign user 42 (in some cases, no keyword is assigned), and in order to understand this, the Hinomaru (national flag) May be displayed beside the image 44.

  FIG. 7 is an example of the image display retrieved by the server 90. The display position of the image on the world map is determined depending on which server is the image recorded by the world map. Since it is fun to see the images of each region in the world, a display frame is provided so that at least one image can be displayed for each region, and the search images are sequentially displayed there.

  For example, a user who wants to see a rose image in Japan performs a keyword search for “rose”. In recent years, translation technology has also been developed. Therefore, English can be translated as “Rose”, and images in the United States, Australia, etc. can be searched by translating keywords. In other regions, languages are complicated or the number of languages is large. Therefore, images recorded on the server are searched by extracting image features. By searching for images that are similar to the images searched and displayed in Japan, the United States, and Australia, it is possible to display images that match the screen without error.

  Next, the operation of the server 90 for performing image detection and display in the first technique related to the present invention will be described using the flowcharts shown in FIGS.

  When the flow of image display shown in FIG. 8 is entered, it is first determined whether or not a list is displayed (S201). In the image display flow, it is possible to select either an image captured by the user himself / herself from a storage list or a keyword or the like. If the result of this determination is list display, then the recording location is designated (S202). In this step, the address of the location where the user's image is stored is designated.

  Subsequently, it is determined whether or not an image recording location has been designated (S203). As a result of the determination, if no designation is made, the process returns to step S202. On the other hand, if designation is made, the images are sequentially displayed in the new order of designation (S204). Next, it is determined whether or not the list display is completed (S205). If the result of this determination is not the end, the single image display is performed (S231). Subsequently, it is determined whether or not the image is a person other than the person (S232). If the result of the determination is a person, the process returns to step S204 as it is. Perform (S233), and return to step S204.

  That is, the images stored in the location designated in step S202 are displayed one by one until the predetermined number of images are filled in the screen (S231). If the image is other than a person, it is similar to this image. A reference image is retrieved and displayed (S233). The reference image is a similar image such as a flower or landscape, and is displayed side by side with the image 43 taken by the user on the same screen as shown in an image 44 in FIG.

  The reference image display subroutine in step S233 will be described later with reference to FIG. In the case of a person image, displaying a person similar to the displayed person is not interesting from the viewpoint of the displayed person. Therefore, in the case of a person, a similar image search is not performed.

  When the number of display images reaches a predetermined number as a result of the determination in step S205, the list display is terminated. When the list display is completed, it is next determined whether or not to enlarge (S206). In this step, it is determined whether or not an enlargement operation is performed on the personal computer by the user. If the result of this determination is that there is no enlargement operation, processing returns to step S202. On the other hand, if the result of determination is that an enlargement operation has been performed, the selected image is enlarged and displayed (S207).

  Subsequently, it is determined whether or not to end the image display (S208). In this step, it is determined whether or not an end operation has been performed by the user. As a result of this determination, if it is the end, the end process is performed. On the other hand, if it is not the end, it is determined whether another candidate is displayed or the enlarged image is reduced. As a result of this determination, if reduction is selected, the image is reduced (S213). On the other hand, if another candidate is selected, another candidate image is displayed (S212). When these steps are processed, the process returns to step S201.

  If the result of determination in step S201 is not list display, search input is next carried out (S221). In this step, the user inputs a keyword or the like. Based on this search input, next, a reference image is searched and displayed as in step S233 (S222). The subroutine here will be described later with reference to FIG. When the reference image is displayed, the process returns to step S201.

  Next, the reference image display subroutine in steps S222 and S233 will be described with reference to FIG.

  First, character information is read (S251). In this step, character information such as a keyword input in step S221 is read. In this search, as shown in FIG. 3, it is of course possible to separately display a list of relationships between search images and keywords, and search using this list. Next, a similar image is selected based on the character information such as the keyword (S252). In this step, tag data attached to the image is searched using character data, and a similar image is selected.

  Subsequently, in step S252, it is determined whether a similar image has been selected (S253). As a result of the determination, if no image can be selected, a warning is displayed (S259), and the process returns to the original flow. On the other hand, if even a single image can be selected, the similar image is displayed (S254). Here, an image having the same keyword and comment as the input keyword is displayed. In the present technology, a predetermined number of similar images are displayed on the screen in order to make the screen more enjoyable by displaying not only images hit by keywords and similar images but also similar images. Until the display of similar images is repeated.

  If an image is displayed in step S254, it is then determined whether a predetermined number of reproductions have been performed (S255). As described above, this is for determining whether a similar number of similar images have been displayed. If the predetermined number of displays are not finished as a result of this determination, it is next determined whether or not there is a similar image (S261). In this step, it is determined whether or not a plurality of similar images can be displayed in step S254.

  If the result of determination in step S261 is that there are no plurality of similar images, processing proceeds to step S263. On the other hand, if the result of determination is that there are a plurality of similar images, common features are then extracted (S262). The extraction of the common feature will be described later with reference to FIGS. Once the common features have been extracted, it is next determined whether or not another language server can be selected (S263).

  Here, another language server is selected when searching with character data such as a keyword. If the same language is used, there is a possibility that it has already been searched and there is a possibility that it will be troublesome twice. On the other hand, a variety of images can be searched and displayed by selecting a different language server.

  As a result of the determination in step S263, if another language server is not found, the process returns to step S255. On the other hand, if another language server is found, a feature quantity similar image search is performed (S264). The search in this step is performed using the common feature amount (feature information) extracted in step S262. Even if the languages are different, image feature information is used, so that an image search can be performed.

  Subsequently, as a result of the search in step S264, it is determined whether a similar image has been searched (S265). If a similar image is searched as a result of this determination, an image with a high similarity is displayed (S266). When the image display is performed, the process returns to step S255, and the above-described steps are executed until a predetermined number of similar images are displayed.

  If a predetermined number of similar images are reproduced and displayed as a result of the determination in step S255, it is determined whether or not the image is NG (S271). In this step, if the user does not like the similar image, he / she performs an operation indicating that it is NG, and therefore determines whether or not an NG operation has been performed. If the result of this determination is not NG, a predetermined number of similar images have been displayed, and the flow returns to the original flow.

  If the result of determination in step S271 is NG, then NG image determination is performed (S272). Here, it is determined which of the plurality of displayed similar images is NG. Subsequently, it is determined whether or not the search is performed again with characters (S273). In this step, the determination is made based on the operation state of the user. As a result of the determination, when the search is performed again with characters, characters of similar categories are input (S275). When a new search keyword is input, the process returns to step S251 to execute the above-described processing.

  If the result of determination in step S273 is that there is no character input, fine correction of the feature quantity is performed (S274). In the example of FIG. 14B, which will be described later, this feature amount is finely corrected by changing the range in which similar features are obtained, for example, by changing the size of a circle in the drawing. If the feature amount is finely corrected, the process proceeds to step S263, and using the feature amount, a similar image is searched again with another language server, and the search image is reproduced and displayed.

  As described above, in the present technology, at the time of searching, a similar image is searched by inputting characters such as a keyword, feature information of the searched similar image is extracted, and a server in a language different from the keyword is accessed. The image search is performed using the feature information. For this reason, it is possible to search and display a similar image with high accuracy and efficiency from images stored in servers all over the world.

  The user only has to enter a keyword when searching. In other words, keywords are easier to come up with and easier to enter, making searching easier and more convenient.

  Internet such as personal computers and the Internet are good at image search, but there are language and culture barriers, and it is difficult to search for accuracy of translation and worldwide. However, according to the present technology, when the languages are different, it is possible to search from the feature information of the image, to solve the language problem, and to display a more variety of images.

  Next, extraction of feature information of similar images and search of similar images in the present technology will be described with reference to FIGS. 10 to 17.

  Now, feature information is extracted in order to search for similar images of the rose image shown in FIG. FIGS. 10B and 10C show a state where a part of a rose image is enlarged, edge-enhanced at the pixel level, and binarized. In FIGS. 10D and 10E, when a black portion remaining at a predetermined pitch is on an orthogonal straight line, the orthogonal portion (the orthogonal portion 51 surrounded by a circle) is considered as a feature point.

  FIG. 11 shows the distribution of feature points of the image shown in FIG. FIG. 12 shows the distribution of feature points divided into screens according to the density. In this example, an adjacent feature point is close and the color in the vicinity is, for example, a red portion (a portion marked with Δ), and an adjacent feature point is far and the color in the vicinity is a red portion (a portion marked with ○). The adjacent feature points are discrete, and the color in the vicinity can be divided into, for example, a green part (part marked with □) and three parts.

  By comparing the distribution of the feature points in the screen and the color relationship as the feature information as shown in FIG. 3 described above, the numerical values can be compared without having to reproduce all the images. By doing so, it is possible to quickly check the degree of coincidence of images. Similar images are similar in the shape of each of these portions, the relationship between the above-mentioned ○ portion and the Δ portion, and the relationship between the ○ portion and the □ portion.

  Next, the image shown in FIG. 13A is analyzed according to the feature points, and the screen is divided and classified as shown in FIG. 13B. Comparing FIG. 12 and FIG. 13 (b) which analyzed both figures based on the feature points, the portion in which the circled portion circles the portion Δ is similar, and therefore the image of FIG. The image 13 (a) can be determined as a similar image. At the same time, it can be considered that the point where the circle part circles the triangle part is a common feature. Moreover, since the part represented by x of FIG.13 (b) differs from the square part of FIG. 12, this is not regarded as a common feature.

  FIG. 14A is a diagram in which common features of FIG. 12 and FIG. 13B are overlapped, and FIG. 14B shows an average area of both. As shown in FIG. 14C, the characteristic pattern is examined while shifting the screen, and the matching images can be said to be similar images.

  A pattern obtained from the distribution and colors of feature points of the person image as shown in FIG. 15A is as shown in FIG. Even if the pattern shown in FIG. 15B is compared with the pattern shown in FIG. 14C, there is no matching portion. However, the pattern of the flower tree image (see FIG. 16B) as shown in FIG. 16A has a high degree of coincidence with the pattern shown in FIG. 14C, and can be determined as a similar image.

  In addition, when searching for similar images, in addition to using feature information, the degree of matching of keywords is also examined. When looking at the degree of coincidence of keywords, for each related word of each word, as shown in Fig. 17, a database on which the degree of coincidence is quantified is prepared on a server on the network, and the degree of coincidence is determined by referring to this database do it.

  In the example shown in FIG. 17, when searching for “rose”, “rose” has a matching degree of 10 which is the same as “rose”, “Roses” has a slightly lower matching degree of 9, and “flower” Then, the degree of coincidence is low and becomes 5. Further, even if such a database is not prepared, a dictionary possessed by a server on the network may be searched, and if it is a word in the dictionary, it may be used as a word having a high degree of matching. For example, when searching for “roses”, if there is a comment in the dictionary, “Rosaceae shrub, grown for ornamental use. It is also possible to pick up words such as “thorn” and “flower” and display an image having these words as keywords.

  As described above, in the present technology, feature points representing edges of contour points are extracted from an image, feature information is extracted using the distribution shape and color information there, and a similar image is extracted using the feature information. Like to search. Therefore, it is possible to search for similar images quickly.

  As described above, in the first technique related to the present invention, in a server arranged on the net, a similar image is searched for by words such as a keyword, and feature information is extracted from the searched similar image, Similar images are searched without using words such as keywords. For this reason, based on the keyword etc. which were input initially, a similar image can be easily searched from the image from which the language provided to the image differs.

  Next, a second technique related to the present invention will be described with reference to FIGS. In the first embodiment of the present invention and the first technology related to the present invention, the area designation is not particularly performed. However, in the second technique related to the present invention, the area to be searched is designated and similar in this area. It searches for images and displays them.

  For example, consider a case where a user selects an image similar to a photographed image and searches for an image more similar to the similar image in an area of interest. In this case, if the region is a region that uses various languages such as English, Portuguese, Spanish, etc., if the language is understood in many countries such as English, it is possible to search using that language as a keyword. It is. However, there are cases where a user is interested in an image assigned with a keyword other than English, and there is a case where the user is not interested in an image searched with a keyword such as English. Therefore, in the present technology, the search is performed in the designated area according to the feature information of the image in addition to the keyword or the like.

  This technique is also configured by a server 90 or the like as shown in FIG. The search for similar images in the server 90 will be described with reference to the flowchart shown in FIG.

  When a flow for determining a display image for searching for and displaying a similar image is entered, an image is first designated (S301), a region is designated (S302), and a tag of the designated image is determined (S303). Here, as shown in FIG. 19, in the selected image 61 and the region 62 from which the selected image is acquired, the selected image is assigned to the selected image in order to search for a photo that is specific to the region and similar to the selected image 61. The tag that is being used is judged. When a tag is assigned to the designated image in Japanese, the keyword is converted into English or the like using the translation function of the server. Even if it is translated into English, as described above, there are cases where an image to which a keyword in another language is assigned may be interesting. Therefore, an image search is performed to compensate for the lack of the keyword search.

  Subsequently, the search is started based on the information obtained in steps S301 to S303 (S304). When the search is started, it is determined whether or not a predetermined number of similar images have been searched (S305). As a result of this determination, if a predetermined number of searches have not been made, it is determined whether or not the image features match (S306). Whether or not the images are similar to each other is determined based on the pattern matching degree as described with reference to FIGS.

  If the result of determination in step S306 is that the image features do not match, processing returns to step S305. On the other hand, if it is determined that the features of the image match, the image, tag (keyword, etc.), and the matching degree of the image are recorded (S307). When these are recorded, the process returns to step S305.

  While steps S305 to S307 are repeatedly executed, the number of similar images reaches a predetermined number as a result of the determination in step S305. In this case, next, an image with a low matching degree of a tag (keyword or the like) is displayed with priority (S311). That is, tag determination is performed in step S303, and an image having a low matching degree with this tag (keyword or the like) is preferentially displayed. As a result, it is possible to find an image that cannot be searched with a keyword but that has a high degree of similarity.

  There are many examples of searching for images by keyword search, but rather, images that are not easily found are rare images that may be displayed and enjoyable. In this technology, it is possible to effectively use images as a means of communication difficult in words, and it is possible to inflate various imaginations with contents that can be easily seen. In other words, it not only increases the enjoyment of the screen but also has the effect of creating interest in foreign countries.

  As described above, in the first embodiment of the present invention and each technique related to the present invention, first, an image is searched using a keyword or the like, and feature information is extracted from the searched similar images, and this feature is extracted. The similarity is further calculated using the information. For this reason, it is possible to search even an image to which a keyword or the like is assigned in a different language, and even a photographic image taken by a foreign user can be easily searched and enjoyed.

  In the first embodiment of the present invention and each technology related to the present invention, the image search function is concentrated on the camera 10 or the server 90. However, a part or all of the functions are executed by a personal computer. Of course it doesn't matter.

The present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, you may delete some components of all the components shown by embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

DESCRIPTION OF SYMBOLS 1 ... Control part, 2 ... Imaging | photography part, 2a ... Face detection part, 3 ... Recording part, 3a ... Search area, 3b ... Character information recording part, 3c ... Time Position determination unit, 4 ... character search unit, 5 ... image search unit, 6 ... communication unit, 7 ... reproduction unit, 8 ... liquid crystal display unit, 9 ... operation unit, 9a ... Character input unit, 10 ... Camera, 21 ... Electronic keyboard, 23 ... Switching unit, 24 ... Conversion unit, 25 ... Input display unit, 26 ... Icon, 27a 27b ... Image, 28 ... Reference image, 31 ... Address, 32 ... Image file, 41 ... User, 42 ... User, 43 ... Image, 44 ... Image 51 ... Orthogonal part, 90 ... Server, 91 ... Control part, 92 ... Net access part, 93 ... Tag search part, DESCRIPTION OF SYMBOLS 4 ... Feature extraction part, 95 ... Similar image search part, 96 ... Area / language determination part, 97 ... Display image determination part, 100 ... Server, 101 ... Control part, 102 ... Net access unit, 103 ... Image recording unit, 104 ... Translation unit, 110 ... Server, 111 ... Control unit, 112 ... Net access unit, 113 ... Image recording unit

Claims (9)

An imaging unit that photoelectrically converts an optical image and outputs image data;
Using the image data, a feature value quantification unit that quantifies the feature value of the image and calculates feature information;
A recording unit that records the feature information in association with the image data;
The image similar to the image based on the image data, and search by using the characteristic information, when the retrieved similar image does not reach the predetermined number, using different feature information from the feature information described above An image search unit for searching for similar images until a predetermined number is reached;
The retrieved similar image and table Shimesuru display unit,
An imaging device comprising:   The imaging apparatus according to claim 1, further comprising a communication unit that transmits the image data and the feature information to the outside of the imaging apparatus.   The imaging apparatus according to claim 1, wherein the feature value digitizing unit digitizes at least one feature value of a color and a contour included in the image.   The imaging apparatus according to claim 1, wherein the screen is divided, classified, and recorded with respect to the discrete feature points representing the edges of the contour points in the screen and the color distribution. It has an operation unit that gives instructions for shooting operations,
When the photographing operation is instructed by the operation unit, the image data output from the imaging unit and the feature information calculated by the feature value digitizing unit are associated with the image data and recorded in the recording unit. The imaging apparatus according to claim 1, wherein: It has a character input part for inputting character information,
The imaging apparatus according to claim 1, wherein the character information input by the character input unit is recorded in association with the image data recorded in the recording unit.   The imaging apparatus according to claim 6, further comprising a character search unit that searches for similar images based on the character information. An imaging unit that photoelectrically converts an optical image and outputs image data;
Using the image data, a feature value quantification unit that quantifies the feature value of the image and calculates feature information;
A recording unit that records the feature information in association with the image data;
An image similar to the image based on the image data is searched using the feature information. If the searched similar images do not reach a predetermined number, the predetermined information is used using feature information different from the feature information. An image search unit that searches for similar images until the number is reached ,
An imaging device comprising: Photoelectrically convert optical image, output image data,
Using the above image data, the feature amount of the image is digitized to calculate feature information,
Record the feature information in association with the image data,
An image similar to the image based on the image data is searched using the feature information. If the searched similar images do not reach a predetermined number, the predetermined information is used using feature information different from the feature information. Search for similar images until the number is reached,
The search has been similar image table Shimesuru,
And a method of controlling the imaging apparatus.