JP2016033822A - Image search system and image search method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image retrieval system and image retrieval method that are broad in a range of an applicable image, can generate the applicable image in short time, and can give users expectation or enjoyment.SOLUTION: In an image retrieval system retrieving a similarity image similar to a photograph image from a database of an image including characters, the image retrieval system comprises: a similarity image retrieval unit 8 that has a face detection character detection and processing part 8a detecting a location of a face of an image, the number thereof and a size thereof, and performs a first image retrieval at a matching degree of the location of the face, the number thereof and the size thereof in a range having a center part of the photograph image cut out in conformity to an aspect ratio of the image to be included in the database, and retrieves the similarity image from the database by performing a second image retrieval at a matching degree of the image after excluding the character to be included in the image retrieved by the first image search; and a slide show unit 6 that sequentially displays the retrieved images in combination of an auxiliary image and the photograph image.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image playback device and an image playback method, and more particularly to an image search system and an image search method for enjoying an image captured by an imaging device such as a digital camera.

  In recent years, imaging devices such as digital cameras have become widespread. When enjoying an image captured by the imaging apparatus, not only printing but also viewing on a screen of a personal computer (PC), a mobile phone, a television, or the like can be easily performed. In these devices, as a method of appreciating a photographed image, a slide show technique for sequentially reproducing images on a screen is performed in the same manner as sequentially viewing printed photographs. However, a slide show is a situation that is not so popular at present because it takes time to finish watching all of the images simply by sequentially switching the images and is monotonous.

  In order to increase the added value to the slide show technique as described above, Patent Document 1 discloses a slide show reproduction method in which group photos of face images constituting a slide show are combined and inserted into the slide show. That is, in this reproduction method, an order is created from representative images of a group with a large number of face images, and a group scene is generated by determining the image size, and this group scene is inserted into the last scene of the slide show. .

JP 2007-249371 A

  The slide show reproduction method described in Patent Document 1 cannot be applied to images other than face images. In addition, it is necessary to newly generate a representative image, and it takes time and effort to generate the representative image. Furthermore, the representative image is limited to a prediction range of “a group photo of face images”, and does not give the user a sense of excitement and enjoyment.

  The present invention has been made in view of such circumstances, and has a wide range of applicable images, can be generated in a short time, and can provide a user with a sense of expectation and enjoyment. And an image search method.

  To achieve the above object, an image search system according to a first aspect of the present invention is an image search system for searching a similar image similar to a photographed image from a database of images including characters. A face detection unit for detecting the first position of the face in the range where the center portion of the photographed image is cut out in accordance with the aspect ratio of the image included in the database. A search for searching for the similar image from the database by performing a second image search with the matching degree of the image after removing characters included in the image searched by the first image search. And a slide show playback unit that sequentially displays the searched images by combining the auxiliary image and the captured image.

  An image search system according to a second aspect of the invention is an image search system for searching a similar image similar to a photographed image from a database of images including characters, and a face detection unit for detecting the position, number and size of the face of the image The first image search is performed with the degree of coincidence between the position, number, and size of the face in a range obtained by cutting out the central portion of the photographed image in accordance with the aspect ratio of the image included in the database, A search unit that searches for the similar image from the database by performing a second image search with a matching degree of images after removing characters included in the image searched in the first image search. The images included in the database are package images of content including images and music.

  The image search system according to a third aspect of the present invention is the image search system according to the second aspect, wherein the image search system includes a slide show playback unit that uses the searched image as an auxiliary image and sequentially displays the captured image. The playback unit plays back the music data included in the package of the content while performing the sequential display.

  According to a fourth aspect of the present invention, there is provided an image retrieval method for retrieving a similar image similar to a photographed image from a database of images including characters, and a face detection step for detecting the position, number, and size of the face of the image. In accordance with the aspect ratio of the image included in the database, a first image search is performed with the degree of coincidence between the position, number, and size of the face within a range obtained by cutting out the central portion of the captured image. A search step for searching for the similar image from the database by performing a second image search with the matching degree of the image after removing characters included in the image searched for in the first image search; and A slide show reproduction step of sequentially displaying the auxiliary image and the photographed image in combination.

  An image search method according to a fifth aspect of the invention is an image search method for searching for a similar image similar to a photographed image from a database of images containing characters, and a face detection step for detecting the position, number and size of the face of the image. In accordance with the aspect ratio of the image included in the database, a first image search is performed with a degree of coincidence between the position, number, and size of the face within a range in which the central portion of the captured image is cut out. A search step of searching for the similar image from the database by performing a second image search with the matching degree of the image after removing characters included in the image searched in the first image search, and The images included in the database are package images of content including images and music.

  ADVANTAGE OF THE INVENTION According to this invention, the range of the image which can be applied is wide, can produce | generate in a short time, and the image search system and image search method which can give a user a feeling of expectation and pleasure can be provided.

1 is a block diagram illustrating a configuration of an image reproduction system including a camera and a personal computer according to a first embodiment of the present invention. In the image reproduction system concerning 1st Embodiment of this invention, it is a figure which shows the use condition of the image reproduction system by a user, (a) is a mode that it has image | photographed, (b) is a camera's image | photographed image. (C) is a state of enjoying a slide show on a television, and (d) is a view showing a state of reproducing and displaying an auxiliary image in the slide show. FIG. 3 is a diagram showing an example of removing characters in order to search for an auxiliary image similar to a representative image in the image reproduction system according to the first embodiment of the present invention, wherein (a) is an image of a jacket such as a music CD. (B) is the figure which removed the character from the image of (a), (c) shows a representative image. It is a flowchart which shows the operation | movement of the camera control in the camera concerning 1st Embodiment of this invention. It is a flowchart which shows the operation | movement of the image matching degree determination in the camera concerning 1st Embodiment of this invention. It is a block diagram which shows the structure of the image reproduction system which consists of a camera and server concerning 2nd Embodiment of this invention. In the image reproduction system concerning 2nd Embodiment of this invention, it is a figure which shows the relationship between the flow of an image etc., and reproduction | regeneration display. It is a flowchart which shows the operation | movement of the camera control in the camera concerning 2nd Embodiment of this invention. It is a flowchart which shows the operation | movement of the server control in the server concerning 2nd Embodiment of this invention. It is a block diagram which shows the structure of the image reproduction system which consists of a camera and a server concerning 3rd Embodiment of this invention. In the image reproduction system concerning 3rd Embodiment of this invention, it is a figure which shows the relationship between the flow of an image etc., and reproduction | regeneration display. It is a figure which shows the relationship between the flow of an image etc., and a reproduction | regeneration display in the image reproduction system concerning the 1st modification of 3rd Embodiment of this invention. It is a flowchart which shows the operation | movement of auxiliary content determination in the image reproduction system concerning the 2nd modification of 3rd Embodiment of this invention.

  A preferred embodiment will be described below using an image reproduction system to which the present invention is applied according to the drawings. FIG. 1 is a block diagram showing a configuration of an image reproduction system including a camera 10, a personal computer 20, and a television 30 according to the first embodiment of the present invention.

  The camera 10 is a digital camera that captures a subject and acquires an image. The personal computer 20 can be connected to the camera 10 and stores image data of a jacket such as a music CD as will be described later. The television 30 can be connected to the camera 10 and reproduces and displays the slide show based on the image data of the slide show generated in the camera 10. The connection between the camera 10 and the personal computer 20 includes not only a wired connection using a cable or the like, a wireless connection such as radio waves or infrared rays, but also an exchange via a storage medium such as a USB memory.

  The camera 10 includes a photographing unit 1, an image recording unit 2, an image selection unit 3, a communication and control unit 4, a display unit 5, a slide show unit 6, a sequence control unit 7, a similar image search unit 8, a face detection character detection and processing unit. 8 a and an auxiliary content recording unit 9. The communication and control unit 4 performs overall control of the camera 10 and performs communication when connected to the personal computer 20 and the television 30.

  The photographing unit 1 includes an optical system for forming a subject image, an image sensor that photoelectrically converts the subject image, an image processing unit that processes a signal output from the image sensor, and the like, and outputs image data. The image recording unit 2 records the image data output from the photographing unit 1. The display unit 5 performs display such as reproduction display of an image recorded in the image recording unit 2. The image selection unit 3 selects a representative image in response to an instruction from the operation member by the photographer.

  The similar image search unit 8 searches for an auxiliary image similar to the representative image instructed by the image selection unit 3 from images recorded in the personal computer 20 or the like. In the present embodiment, a photographed image is compared with an image of a content jacket such as a music CD, and an image similar to the photographed image is searched.

  The face detection character detection and processing unit 8a performs face detection and character removal processing from the image. Content jackets often include characters such as labels, publisher names, as well as images, titles, authors and artist names. Therefore, the character detection unit in the face detection character detection and processing unit 8a removes characters and the like from the image, and the similar image search unit 8 compares the photographed image of the jacket such as a music CD with the photographed image taken. An image similar to the image (that is, an auxiliary image) is searched.

  The auxiliary content recording unit 9 records the auxiliary image searched by the similar image search unit 8. Also, information related to the auxiliary image, for example, music (audio data) recorded in content such as a music CD, text data of lyrics of the music, and the like are also recorded.

  The slide show unit 6 causes the display unit 5 to perform a slide show by combining the representative image or the captured image recorded in the image recording unit 2 with the auxiliary image recorded in the auxiliary content recording unit 9. The order control unit 7 controls the order of reproduction and display of images when performing a slide show.

  The personal computer 20 includes a communication and control unit 24 and a content jacket recording unit 21. The communication and control unit 24 performs overall control of the personal computer 20, and controls communication when connected to the communication and control unit 4 of the camera 10. The content jacket recording unit 21 stores information related to content such as LP images, content jacket images such as music CDs and DVDs, book cover images, and music and lyrics. That is, the content includes various things such as existing comics, anime, television programs, movies, music, etc., and the content jacket recording unit 21 stores images and related information related to these contents. Yes.

  In the present embodiment, the camera 10 acquires these pieces of information from the content jacket recording unit 21 via the communication and control unit 4 and the communication and control unit 24 of the personal computer 20. However, in addition to this method, it may be obtained from an external server or the like from the personal computer 20 via a network such as the Internet, or the camera 10 directly accesses an external server or the like via a network such as the Internet. You may make it acquire such information. In addition to this method, a recording medium on which such information is recorded, such as a memory card, may be used.

  Next, the operation in this embodiment will be described with reference to FIG. First, as shown in FIG. 2A, shooting is performed using the camera 10. Subsequently, the camera 10 is connected to the personal computer 20 as shown in FIG. At this time, the user selects a representative image while viewing the captured image reproduced on the display unit 8. The similar image search unit 8 or the like of the camera 10 searches for an auxiliary image similar to the representative image, and records the searched auxiliary image in the auxiliary content recording unit 9.

  Images taken with the camera 10 can be played back and displayed in a normal frame by the display unit 5 of the camera 10 or can be enjoyed as a slide show. However, as shown in FIG. You can enjoy watching with many people. In addition, when playing back and displaying on the television 30, as shown in FIG. 2D, the content (auxiliary image) that everyone knows is played back and displayed as an introductory image, and preparations are made for the beginning. I can get it.

  The slide show shown in FIG. 2 (d) does not play back only one frame of the shot image, but when a plurality of shot images are selected and viewed as a slide show, a music CD or the like retrieved as auxiliary content (auxiliary image) is displayed. Display a combination of jackets. That is, as shown in FIG. 2D, the jacket image 110 similar to the captured image 100 is displayed before the captured image 100 as shown in FIG. By displaying such auxiliary content (in this case, the jacket image 110), it is possible to attract the viewer's interest rather than merely displaying the captured image 100.

  If the music recorded on this music CD is used as BGM, it can be used to further enhance the atmosphere. That is, the music on the music CD that is the source of the auxiliary image is the same jacket music as the atmosphere of the photographed image, so it often has a natural and harmonious feeling, and it is possible to save the trouble of selecting the music. .

  Further, when searching for similar images using the jacket image 110 of the music CD jacket as shown in FIG. 3A as a search target, the character portion is removed as shown in FIG. Only the portion 111 is extracted. Then, the photographed image 100 as shown in FIG. 3C is compared with the image portion 111 as shown in FIG. 3B, and an image similar to the photographed image 100 is searched.

  When searching for similar images, information on the position and size of the face in the screen may be used. In recent years, in the field of cameras, research on techniques for specifying the position of a face from within a screen has been performed in order to adjust the focus and exposure, and such techniques may be applied.

  Next, the operation of the camera 10 in the present embodiment will be described using the flowchart shown in FIG. When the camera control flow is entered, it is first determined whether or not the playback mode is set (S1). The camera 10 can use two modes, a playback mode and a shooting mode. If the playback mode is not set as a result of the determination, it is determined that the mode is a shooting mode, and then the operation state of the release button is detected to determine whether or not to perform shooting (S2).

  As a result of the determination, if it is not shooting, the process returns to step S1, whereas if it is shooting, a shooting operation is executed (S3). In the photographing operation, the photographing unit 1 acquires image data of a subject image and records the image data in the image recording unit 2. When the photographing operation ends, the process returns and returns to step S1.

  If the result of determination in step S1 is playback mode, the operating state of the playback button or the like is detected and it is determined whether or not playback is to be performed (S11). As a result of the determination, if reproduction is to be performed, the process proceeds to step S12 and subsequent steps, and the reproduction mode is executed. In the playback mode in this embodiment, in addition to normal playback display and normal slide show, playback display by a slide show combined with auxiliary content is possible.

  Next, it is determined whether to perform database (DB) communication (S13). When the user of the camera 10 obtains an auxiliary image as auxiliary content, the user performs an operation to that effect. In this step, it is determined whether or not the operation has been performed. If DB communication is performed as a result of the determination, it is subsequently determined whether or not to search for similar images (S21). The determination here is whether or not the user has started a search start operation.

  If the result of determination in step S21 is that search has not started, processing returns to step S1. On the other hand, when the search is started, a representative image is designated (S22). In this step, the captured images recorded in the image recording unit 2 are sequentially reproduced and displayed on the display unit 5, and the user designates a representative image from the reproduced and displayed captured images according to his / her preference.

  When the designation of the representative image is completed, an image similar to the selected representative image is searched for (S23). The subroutine for determining the degree of coincidence for searching for similar images will be described later with reference to FIG. Subsequently, with respect to the representative image designated in step S22, the degree of image coincidence with all target images is determined, and it is determined whether or not the processing has been completed (S25). As a result of the determination, if it is not the end, the target image is changed (S27), the process returns to step S23, and the image matching degree is determined for the target image changed in step S27.

  If it is determined in step S25 that the determination of the degree of coincidence with all target images has been completed, an image having the maximum degree of coincidence is then acquired (S26). Since the degree of coincidence is digitized for each target image in the image coincidence degree determination subroutine in step S23, the target image having the highest degree of coincidence among the target images for which the degree of coincidence has been determined is determined in this step. Acquired and used as an auxiliary image. When the matching degree maximum image is acquired, the process returns and is executed from step S1. Note that the representative image that is reproduced and displayed together with the auxiliary image in the slide show is normally one frame, but when there are a plurality of representative images, the selection of the representative image is repeated a plurality of times.

  If the result of determination in step S13 is not DB communication, it is next determined whether or not it is a slide show (S14). If the result of determination is not a slide show, normal playback display, for example, playback display of the selected image from thumbnail playback is performed (S15). On the other hand, in the case of a slide show, by executing the steps after step S31, the auxiliary image (the image acquired in step S26) is reproduced as if it were an opening screen of the slide show, and then the representative image is reproduced and displayed. Then, sequentially, the captured images are reproduced and displayed in the form of a slide show according to the time series of the captured images.

  In the slide show, first, it is determined whether or not an image has been received (S31). That is, in step S13, DB communication is performed, and it is determined whether or not an auxiliary image has been acquired in step S26.

  As a result of the determination, if the image has been received, it is determined whether or not to perform external output (S32). In the case where a television 30 is connected to the camera 10 and a large number of people are watching as shown in FIGS. 2C and 2D, in this step, the slide show image data is externally output to an external device such as the television 30. Do. As a result of the determination, in the case of external output, the image data such as an auxiliary image, a representative image, and a captured image is output from the communication and control unit 4 to an external device such as the television 30. When the external image output ends, the process returns and the process is executed from step 1.

  If the result of determination in step S32 is that there is no external output, auxiliary images are reproduced (S33), and then representative images are reproduced (S34). When the reproduction of the representative image is completed, a slide show of the captured images is sequentially performed in order of the shooting date and time (S35).

  As described above, prior to the normal slide show, the auxiliary image similar to the representative image is reproduced and displayed, and then the representative image is reproduced and displayed. That is, as shown in FIG. 2D, the auxiliary image is reproduced first to enhance the feeling of starting the slide show, and then the reproduction of the captured image is started as shown in FIG. The image that appears after the auxiliary image is a representative image from which a similar image is retrieved. By displaying the representative image, the viewer understands why the auxiliary image is displayed. The order control unit 7 in FIG. 1 performs such rearrangement.

  If the result of determination in step S31 is that the image has not been received, the auxiliary image has not been searched, so steps S33 and S34 are skipped, and the time-series slide show in step S35 is executed. When the time-series slide show ends, the process returns and is executed from step S1.

  Next, the image matching degree determination subroutine in step S23 will be described with reference to the flowchart shown in FIG. When determining the degree of image coincidence, it should be noted that the DB image stored in the database (DB) such as the content jacket recording unit 21 has a different aspect ratio from the photographed image.

  For example, it is difficult to directly compare the degree of coincidence between a square image such as an LP record or a music CD jacket and a 4: 3 image of a normal digital camera. Therefore, in the present embodiment, the comparison is made after cutting out the central portion of the photographed image in accordance with the aspect ratio of the DB image. In addition, the similarity between the images is compared within the cut-out range, the face position is detected, and the comparison is performed using the image from which characters are removed.

  When the image matching degree determination subroutine is entered, first, the number and positions of the faces of the representative image designated in step S22 are determined (S101). Subsequently, based on the determination result in step S101, it is determined whether or not there is a face in the representative image (S102). As a result of the determination, if there is a face, an image having a face is selected from images recorded in the content jacket recording unit 21 as a database (DB) (S103). In this step, one target image is selected from a large number of images. In step S27 described above, the next target image is selected. In step S25, it is determined whether the processing has been completed for all target images.

  Subsequently, the image is compared with the representative image within a range that matches the aspect ratio of the image recorded in the content jacket recording unit 21 (S104). Then, an image having the same face position and the same number of faces is selected within the range in which the aspect ratio is taken into consideration with the image recorded in the content jacket recording unit 21 (S105).

  If the result of determination in step S102 is that there is no face in the representative image, an image recorded in the content jacket recording unit 21 and having no face is selected from the images (S111). Then, it is compared with the representative image within a range that matches the aspect ratio of the image recorded in the content jacket recording unit 21 (S112).

  When the process in step S105 or S112 is completed, it is next determined whether or not a character exists in the selected image (S106). As described above, if characters and the like are mixed in the image, the accuracy of the similarity determination is lowered. Therefore, the degree of coincidence is determined in a portion excluding the characters and the like in advance.

  If the character is present as a result of the determination in step S106, the degree of coincidence is determined in a portion other than the character portion (S107). If the character is not present in step S106, step S107 is skipped, Proceed to step S108.

  Subsequently, the degree of coincidence similar to the representative image is determined over the entire range in accordance with the aspect ratio of the DB image recorded in the content jacket recording unit 21 (S108). When the coincidence determination is completed, the process returns to the original routine.

  As described above, in the first embodiment of the present invention, before the captured image is reproduced and displayed in the slide show, the representative image and the auxiliary image similar to the representative image are reproduced and displayed. The auxiliary image is searched from the content jacket recording unit 21 in which the DB image is recorded. By reproducing and displaying the auxiliary image and the representative image, the viewer's interest can be attracted, and the user has a sense of expectation. And can give fun.

  Further, in the present embodiment, since the face image does not need to exist in the representative image, the range of applicable images is wide, and the representative image is simply specified from the captured image. It can be generated in a short time without image processing for creation.

  Furthermore, in the present embodiment, the degree of coincidence is determined within a range corresponding to the aspect ratio of the DB image, and the degree of coincidence is determined by ignoring the character portion. It is possible to determine a high degree of coincidence.

  In the present embodiment, the DB image is a cover image such as a jacket such as a music CD or a book. However, the DB image is not limited to this. For example, It may be an image taken by a third party recorded in an external server, or other various images such as illustrations and paintings.

  In this embodiment, the representative image and the auxiliary image are reproduced and displayed only at the beginning before the time-series slide show. However, the present invention is not limited to this. The representative image and its auxiliary image may be reproduced and displayed. Furthermore, in the present embodiment, the camera 10 and the personal computer 20 are linked to each other. However, the image selection unit 3, the similar image search unit 8, the face detection character detection and processing unit 8 a, the slide show unit 6, the order of the camera 10. The functions of the control unit 7 and the auxiliary content recording unit 9 may be executed by the personal computer 20 and a program installed therein.

  Next, a second embodiment of the present invention will be described with reference to FIGS. In the first embodiment, similar images are searched in the camera 10, but in the second embodiment, similar images are searched by an external device such as a server via a network such as the Internet. .

  FIG. 6 is a block diagram illustrating a configuration of an image reproduction system including the camera 10 and the server 40 according to the second embodiment. The camera 10 includes a photographing unit 1, an image recording unit 2, an image selection unit 3, a communication and control unit 4, a display unit 5, a slide show unit 6, an order control unit 7, and an auxiliary content recording unit 9. The functions of these units are the same as those of the first embodiment shown in FIG.

  The server 40 that functions as a database includes a content jacket recording unit 21, a similar image search unit 22, a face detection character detection and processing unit 23, a communication and control unit 24, and a representative image temporary storage unit 25. The communication and control unit 24 performs overall control of the server 40 and also communicates with external devices such as the camera 10 through the Internet network or the like.

  Since the similar image search unit 22 and the face detection character detection and processing unit 23 are the same as those in the first embodiment shown in FIG. It functions to detect the position and number, and to extract the character part and remove it from the image. The similar image search unit 22 searches for an image similar to the representative image from the images recorded in the content jacket recording unit 21.

  The representative image temporary storage unit 25 is a storage unit for temporarily storing the representative image designated by the image selection unit 3 of the camera 10. The similar image search unit 22 described above performs a search using the representative image stored in the representative image temporary storage unit 25.

  Thus, on the camera 10 side, the auxiliary content recording unit 9, the slide show order control unit 7, and the slide show unit 6 that sequentially reads out and reproduces data from the image recording unit 2 and the auxiliary content recording unit 9 for the slide show. The server 40 side has a function of temporarily storing a representative image sent from the camera 10 and a function of searching for a similar image. When searching for similar images, as in the first embodiment, the search is performed in consideration of whether the image is a person image, the position and number of faces, and the character portion of the auxiliary content is ignored. .

  Next, the operation in this embodiment will be described with reference to FIG. As shown in FIG. 7, the camera 10 captures the subject 11, designates a representative image 12 a from the photographed images 12 captured at this time (image No. 4 in the figure), and stores the representative image 12 a on the network 19. To the server 40 via The server 40 searches for a similar image similar to the representative image from the content jacket recording unit 21 as a database, and sets the image having the highest degree of matching as the auxiliary image 22.

  When the auxiliary image 22 is searched, the server 40 transmits the auxiliary image 22 to the camera 10. When the camera 10 starts the slide show, first, the searched auxiliary image 22 is reproduced and displayed, and then the representative image 12a is reproduced. 1-No. Four captured images 12 are reproduced.

  Further, since the auxiliary image 22 is a published content, accompanying information (A to D in the figure) such as accompanying information attached by the manufacturer to the content and user's impression and opinion is attached. In the case of a music CD, music data is also included as accompanying information. Therefore, it is possible to perform a slide show while reproducing the music recorded on the music CD along with the reproduction display of the auxiliary image 22. Further, a music piece having the same atmosphere may be searched and reproduced from other information such as a song title, composer, artist, etc. even if it is not recorded on the music CD.

  Next, the operation of the camera 10 in the present embodiment will be described using the flowchart shown in FIG. The same processing steps as those in the camera control in the first embodiment shown in FIG. 4 are given the same step numbers, and detailed descriptions thereof are omitted. In the first embodiment, the search for the auxiliary image is performed in the camera 10, but in the second embodiment, the search is performed in the server 40, and this difference will be mainly described.

  Steps S1 to S15 executed in the camera control flow are the same as those in the first embodiment. If the result of determination in step S13 is DB communication, if it is DB communication, it is determined whether or not acquisition of an auxiliary image and related information is started (S21A). This determination detects whether or not an operation member for starting acquisition (not shown) has been operated by the user.

  If the auxiliary image or the like is not acquired as a result of the determination in step S21A, the acquired information is changed to acquire another information (S29), and the process returns to step S1. In step S29, if the auxiliary image acquired in step S26A, which will be described later, or the related information is not satisfied, a condition for acquiring information is set / changed in order to further acquire information. On the other hand, if acquisition of an auxiliary image or the like is started as a result of the determination in step S21A, a representative image is designated (S22), and the designated representative image is transmitted to the server 40 (S23A). The representative image is designated by the user from the captured images reproduced and displayed on the display unit 5.

  When the transmission of the representative image ends (S25 → Yes), the image (auxiliary image) having the highest degree of coincidence and information related to the auxiliary image are acquired from the server 40 (S26A). That is, since the server 40 searches for auxiliary images and the like, the camera 10 waits for the search in the server 40 to end and the search results to be transmitted. Get related information. As related information, if the auxiliary image is an image of a music CD jacket, it is music recorded on the music CD.

  If the result of determination in step S14 is not a slide show, it is determined whether or not an image has been received (S31). In the second embodiment, the auxiliary image search is performed in the external server 40, and in this step, it is determined whether or not the auxiliary image is received from the server 40 in step S26A.

  If the result of determination in step S31 is that an image has been received, as described with reference to FIG. 7, the auxiliary image is reproduced and displayed, and for example, a music CD is reproduced using the related information (S33A). The representative image is reproduced and displayed (S35A). When the reproduction display of the auxiliary image and the representative image is finished, a normal slide show for sequentially reproducing the photographed images in order of the photographing date and time is executed (S35A).

  If the result of determination in step S31 is that no image has been received, there is no auxiliary image. Therefore, in step S35A, a normal slide show is executed, and when the normal slide show ends, the process returns and executes from step S1. .

  As described above, the camera 10 according to the second embodiment does not perform the image search, transmits the representative image to the server 40 (S24), acquires the auxiliary image retrieved from the server 40 (S26), and this auxiliary image. Is used to execute a slide show (S33 to S35). In the network communication state, an image is transmitted to the outside via the network (S23A), and further different information is acquired from the obtained acquisition information (S29). When the searched auxiliary image or the like is not liked, or when the music is not liked, it can be transmitted in step S29 that further information is acquired.

  Also in the second embodiment, as in the first embodiment, a slide show may be performed in which the television 10 can be captured by the camera 10 and an auxiliary image is reproduced and displayed. In this case, after step S31, it is determined whether to perform external output, and an image is output to the outside according to the determination result.

  Next, the operation of the server 40 in this embodiment will be described using the flowchart shown in FIG. If this flow is entered, it is first determined whether or not a representative image has been acquired (S51). As described above, since the camera 10 transmits the representative image in step S23A, it is determined whether or not the representative image is received in this step.

  When the representative image is acquired as a result of the determination in step S51, the acquired representative image is temporarily stored in the representative image temporary storage unit 25 (S52). Subsequently, the characters are removed from the target image by the face detection character detection and processing unit 23 (S53). That is, characters and the like are described in the jacket image such as a music CD recorded in the content jacket recording unit 21, and by removing these characters and the like, the accuracy of determining the degree of coincidence of the images is improved. ing.

  Subsequently, the face position of the target image is detected by the face detection character detection and processing unit 23, and the degree of coincidence between the representative image and the target image is determined in consideration of the face position (S54). Here, a numerical value corresponding to the degree of coincidence is obtained. When the coincidence determination is completed, it is determined whether or not the coincidence determination has been completed for all the target images in the content jacket recording unit 21 (S55). As a result of the determination, if it has not been completed, the image is changed to the next target image (S57), the process returns to step S53, and the determination is repeated in the same manner as described above.

  As a result of the determination in step S55, when all the target images have been determined, the image with the highest degree of coincidence (auxiliary image) and related information (for example, recorded music) of the auxiliary image are obtained. It transmits to the camera 10 (S56).

  Subsequently, it is determined whether or not an image NG is received from the camera 10 (S71). That is, when the user of the camera 10 is not satisfied with the auxiliary image transmitted from the server 40, the fact is transmitted by changing the acquired information in step S29. If the image NG is received as a result of the determination, the image having the next highest matching degree and its related information are transmitted to the camera 10 (S72), and the process returns to step S71.

  If the result of determination in step S71 is that the image is not NG, it is next determined whether or not the related information is OK (S73). If the user of the camera 10 is not satisfied with the related information transmitted from the server 40, the fact is transmitted in step S29. If the result of determination is that it is not OK, other related information related to the auxiliary image is transmitted (S74), and the process returns to step S73. As a result of the determination in step S73, if the related information is OK, the process returns.

  If the result of determination in step S51 is that a representative image has not been acquired, it functions as a normal image recording and browsing server. First, it is determined whether an image for recording has been acquired (S61). As a result of the determination, if a recording image is acquired, the image is recorded in the recording image recording unit in the server 40 (S62), and the process returns.

  If the result of determination in step S61 is that an image for recording has not been acquired, it is determined whether or not to view the image recorded in the server 40 (S63). If the result of the determination is that the image is not appreciating, the process returns as it is. On the other hand, if the image is appreciating, the image is displayed according to the access (S64), and the process returns.

  As described above, in the second embodiment of the present invention, when the server 40 acquires the representative image, the representative image is temporarily stored, and a music CD having a similar jacket is searched (S51 → S54). The auxiliary image thus transmitted is transmitted to the camera 10 (S56). At this time, related information is also transmitted together. If the user performs an operation not to like the auxiliary image transmitted, if it is determined in steps S71 and S73, another auxiliary image and related information are transmitted (S71 and S74). Further, it has an image bank function for recording a mere photographed image, and stores the transmitted image (S62) and reproduces it in response to access (S63 → S64). The server 40 may omit the image bank function or may have other functions.

  Next, a third embodiment of the present invention will be described with reference to FIGS. 10 and 11. In the first and second embodiments, the auxiliary image is reproduced and displayed at the beginning of the slide show, and a still image is independently displayed and reproduced as the auxiliary image. In the third embodiment, the auxiliary image is reproduced and displayed last, and after the auxiliary image is reproduced, a moving image related to the auxiliary image is continuously reproduced and displayed.

  FIG. 10 is a block diagram illustrating a configuration of an image reproduction system including the camera 10, the television 30, and the server 40 according to the third embodiment. The camera 10 includes a photographing unit 1, an image recording unit 2, an image selection unit 3, a communication and control unit 4, a display unit 5, and an auxiliary data unit 6A. Except for the auxiliary data part 6A, the function of each part is the same as that of the first embodiment shown in FIG. 1, and therefore the detailed description of the same part is omitted.

  The auxiliary data unit 6A is a storage unit that stores the shooting date and time and the shooting location of the representative image in association with the image. The information such as the shooting date and time is transmitted together with the representative image when transmitting the representative image to the server 40, and the information such as the shooting date and time is also taken into consideration when the server 40 searches for the auxiliary image.

  The server 40 serving as a database includes a content jacket recording unit 21, a similar image search unit 22, a communication and control unit 24, a representative image and data temporary storage unit 25A, an image recording unit 26, a related information search unit 27, and a sequential playback. Unit 28 and network distribution unit 29. Since the content jacket recording unit 21 and the like are the same as those in the second embodiment, different parts will be mainly described.

  In the same manner as in the first embodiment, the content jacket recording unit 21 stores a jacket image of a moving image such as a DVD and related video in addition to a music CD or the like. In the case of a moving image such as a DVD, in addition to the jacket image, the top image of the moving image is also stored for retrieval. Similarly, in the case of content with images such as cartoons and animations, the first image is also stored for retrieval in addition to the cover image. The similar image and data search unit 22 searches the content jacket recording unit 21 for an image similar to the representative image and outputs it as an auxiliary image. The representative image and data temporary storage unit 25A stores the representative image transmitted from the camera 10 and information such as the shooting date and time stored in the auxiliary data unit 6 and transmitted in association with the representative image.

  The image recording unit 26 records a captured image or the like that is stored in the image recording unit 2 of the camera 10 and reproduced and displayed in a slide show. The related information retrieval unit 27 retrieves information related to the auxiliary image (a jacket such as a music CD or a video DVD), for example, a composer name or an artist name related to the music CD.

  The sequential playback unit 28 uses the captured image recorded in the image recording unit 26, the similar image, the auxiliary image searched by the data search unit 22, and the related information searched by the related information search unit 27, to generate these images. And the order of playback and display of information. The net distribution unit 29 is connected to the television 30 via a network such as the Internet, and transmits captured images, auxiliary images, and the like sequentially output by the reproduction unit 28 to the television 30. The television 30 is a large-screen television, and reproduces and displays images and information distributed from the network distribution unit 29.

  Next, the operation in this embodiment will be described with reference to FIG. A representative image 12 a (also an image No. 4 in the figure) is designated from the photographed images 12 photographed by the camera 10 and transmitted to the server 40 via the network 19. In addition to the representative image 12a, all the captured images (No. 1 to No. 3 in the figure) used for the slide show are also transmitted. The server 40 searches for similar images from the content jacket recording unit 21 serving as a database, and sets the image with the highest degree of matching as the auxiliary image 22.

  The server 40 searches for similar images in the similar image and data search unit 22 and searches for related information. Then, the sequential playback control unit 28 sequentially transmits captured images to the television 30 via the network distribution unit 29 and the network 19 in the order of photographing date and time, and the television 30 reproduces and displays the captured images. Then, at the end of the slide show, the representative image 12a is transmitted, and then the auxiliary image 22 similar to the representative image 12a is reproduced and displayed.

  In this embodiment, an auxiliary image 22 based on a DVD jacket image similar to the representative image 12a is transmitted, and after the auxiliary image 22 is reproduced and displayed, the moving image 120 recorded on the DVD is reproduced. In this example, after a slide show of photos that are memories of a trip to Vienna, a moving image 120 based on a DVD about Vienna is played.

  In the first and second embodiments, the example in which the auxiliary image 22 corresponding to the representative image 12a is displayed at the top has been described. However, in the present embodiment, the auxiliary image 22 is reproduced last and the viewer has a strong impression. Leave a slide show. Of course, in addition to the last reproduction, the representative image 12a may be reproduced at a place aimed at excitement during the slide show. The reproduction timing may be arbitrarily set by the user. Further, the reproduction order of the auxiliary image 22 may be determined according to the reproduction order of the representative image 12a. Further, instead of being reproduced and displayed on the television 30, it may be reproduced and displayed by the camera 10 as in the first and second embodiments.

  Further, after the last image of the slide show based on the photographed image 12 is displayed, the movie or documentary set in the same place after finishing watching the searched DVD video, for example, a travel photograph. By making it possible to appreciate, it is possible to inspire more travel and indulge in memories.

  Next, the modification of 3rd Embodiment of this invention is demonstrated using FIG.12 and FIG.13. In the first to third embodiments, one image is sequentially displayed on the display unit. However, in the first modification, a plurality of images are displayed. That is, as shown in FIG. 12, when the screen of the television 30 is a large screen, it is possible to display a plurality of images on one screen, so there is no need to stick to time division display. An application in which a plurality of photographed images are arranged on the large screen of the television 30 and a DVD is reproduced at the same time while switching between them is also possible. The configuration of the modified example of the third embodiment is the same as the configuration of the third embodiment shown in FIG.

  In the first to third embodiments, the user designates the representative image and acquires the auxiliary image. However, by executing the auxiliary content determination flowchart shown in FIG. Alternatively, the auxiliary image may be automatically selected.

  In the auxiliary content determination flow, the first photographed image is first selected (S201). Next, an image similar to the selected photographed image is searched from the content jacket recording unit 21 (S203). A content (image) having a similarity equal to or greater than a predetermined value is temporarily recorded (S203). Next, the next image is selected from the captured images (S204).

  If another image is selected, it is next determined whether or not there is a photographed image (S205). As a result of the determination, if there is a next photographed image, the process returns to step S202, the above steps are repeated, and content having a similarity equal to or greater than a predetermined value is recorded. If the result of determination in step S205 is that there are no captured images, that is, all the captured images have been searched, the content that has been searched a predetermined number of times or more and temporarily recorded in step S203 is used as auxiliary content (auxiliary image). Transmit (S206). As for transmission, when playback and display is performed with the camera 10 as in the first and second embodiments, to the camera 10 when being reproduced and displayed over the network as in the third embodiment, to the television 30 when distributed over the network. Send.

  In this way, content that can be searched many times can be said to be auxiliary content that is highly relevant to a captured image. For example, if you are traveling to Vienna, you will be able to select movies and music that are related to the land without fail because the captured images contain many of the features of Vienna. Become.

  As described above, in each embodiment of the present invention, the representative image is compared with the image related to the content, a similar image is searched as an auxiliary image, and the auxiliary image is reproduced and displayed before performing the slide show. The image is being played back. For this reason, even if it is an image other than a face, it is possible to search for an auxiliary image, so that the range of applicable images is wide. Further, since it is not necessary to generate a representative image, it can be processed in a short time. Furthermore, since the auxiliary image is an image related to the content, it is possible to give the user a sense of expectation and enjoyment.

  In the present embodiment, the digital camera is used as the photographing device. However, the camera may be a digital single-lens reflex camera or a compact digital camera, and may be a mobile phone or a personal digital assistant (PDA). Of course, a camera built in Personal Digital Assist) may be used.

  The present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. Moreover, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above embodiments. For example, you may delete some components of all the components shown by embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

DESCRIPTION OF SYMBOLS 1 ... Imaging | photography part, 2 ... Image recording part, 3 ... Image selection part, 4 ... Communication and control part, 5 ... Display part, 6 ... Slide show part, 6A ... Auxiliary data section, 7 ... Order control section, 8 ... Similar image search section, 8a ... Face detection character detection and processing section, 9 ... Auxiliary content recording section, 10 ... Camera, 12. ..Photographed image, 12a ... representative image, 20 ... personal computer, 21 ... content jacket recording unit, 22 ... auxiliary image, 23 ... face detection character detection and processing unit, 24 ... Communication and control unit, 25 ... representative image temporary storage unit, 25A ... representative image and data temporary storage unit, 26 ... image recording unit, 27 ... related information search unit, 28 ... sequentially Reproduction part, 29 ... Net distribution part, 30 ... Television, 40 ... Server , 100 ... photographed image, 110 ... jacket image, 111 ... image portion, 120 ... Video

Claims (5)

In an image retrieval system for retrieving a similar image similar to a captured image from a database of images including characters,
It has a face detection unit that detects the position, number and size of faces in the image,
In accordance with the aspect ratio of the image included in the database, a first image search is performed with a degree of coincidence between the position, number, and size of the face within a range in which the central portion of the captured image is cut out. A search unit that searches for the similar image from the database by performing a second image search with a matching degree of the image after removing characters included in the image searched by the image search;
A slide show playback unit for sequentially displaying the searched images by combining the auxiliary image and the captured image,
An image search system comprising: In an image retrieval system for retrieving a similar image similar to a captured image from a database of images including characters,
It has a face detection unit that detects the position, number and size of faces in the image,
In accordance with the aspect ratio of the image included in the database, a first image search is performed with a degree of coincidence between the position, number, and size of the face within a range in which the central portion of the captured image is cut out. A search unit that searches for the similar image from the database by performing a second image search with a matching degree of the image after removing characters included in the image searched by the image search;
Comprising
An image search system, wherein the image included in the database is a package image of content including images and music. The image search system includes a slide show playback unit that uses the searched image as an auxiliary image and sequentially displays it in combination with the captured image.
The image search system according to claim 2, wherein the slide show reproduction unit reproduces music data included in the package of the content while performing the sequential display. In an image retrieval method for retrieving a similar image similar to a photographed image from a database of images including characters,
A face detection step for detecting the position, number and size of faces in the image;
In accordance with the aspect ratio of the image included in the database, a first image search is performed with a degree of coincidence between the position, number, and size of the face within a range in which the central portion of the captured image is cut out. A search step of searching for the similar image from the database by performing a second image search with the matching degree of the image after removing characters included in the image searched by the image search;
A slide show playback step for sequentially displaying the searched images by combining the auxiliary image and the captured image;
An image search method comprising: In an image retrieval method for retrieving a similar image similar to a photographed image from a database of images including characters,
A face detection step for detecting the position, number and size of faces in the image;
In accordance with the aspect ratio of the image included in the database, a first image search is performed with a degree of coincidence between the position, number, and size of the face within a range in which the central portion of the captured image is cut out. A search step of searching for the similar image from the database by performing a second image search with the matching degree of the image after removing characters included in the image searched by the image search;
Comprising
An image search method, wherein the image included in the database is a package image of content including an image and music.