JP2009065324A - Image providing device, image providing method and image providing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a new technology for providing an image that a user desires to view, by retrieving the image among large amounts of moving images and still images photographed at a specific location. <P>SOLUTION: When images photographed at the specific place are inputted and stored in a storing means, a photographing position and a photographing angle of the inputted images are deduced, by collating the inputted images with an environmental model of the specific place, and information of the estimated photographing position and the estimated photographing angles are recorded, while being associated with the images stored in the storing means. When there is a request from a user for browsing an image that designates the position of the visual point and the angle of a visual line, images which are seen from the position of the visual point and the angle of the visual line designated by the user are retrieved, by retrieving the photographing positions and the photographic angles stored in the storing means; an image of an environmental model of a particular place which is seen from the position of the visual point and the angle of the visual line designated by the user is displayed on a display; and a list of summary images showing a summary of the retrieved images is displayed on the image. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention provides an image providing apparatus and method for searching for and providing to a user an image that the user wants to see from a vast number of moving images and still images taken at a specific location, and the image provision The present invention relates to an image providing program used for realizing the apparatus, and in particular, an image providing apparatus and method for searching for an image that the user wants to see and providing the user with a clue of a photographing position, a photographing direction, and a photographing time. And an image providing program used for realizing the image providing apparatus.

  There is a need to shoot sports events such as soccer and baseball with a large number of cameras, and to search for a scene in which the target player is reflected from the direction and position of the captured video.

  Furthermore, it is also possible to search for and view highlight scenes that have been missed by archiving and sharing videos and photos taken by many people, such as athletic meet, in the community.

  Against this background, there are also provided websites that share digital information about the same subject to be photographed by many users.

  On the other hand, the storage media of digital products is growing, and it takes too much time to view all the photos and videos that are shared.

  Therefore, there is a demand for searching in a quick and intuitive manner. For example, it is a search that wants to see a group of photos taken from a specific place during a specific time period.

  As a conventional technique for image retrieval, there is a technique for assigning a tag to each image after the image is shared and performing retrieval based on the tag.

  For example, you can set “Evaluation” and “Popularity” for each photo and video, and after viewing the photo, you can enter the rating or calculate the popularity based on the number of times it was viewed. There are technologies that use popularity to search for items that are highly rated and popular.

  Such image search technology presupposes the action of the viewer, but as the premise of the action of the uploader, by adding comments to each photo or video, from those comments There is also a technique for searching for an image by searching for a specific term.

  Also, by recording the user's behavior, we can extract a group of photos and videos that many people continue to see, and introduce the photos and videos belonging to the same group to those who have seen a specific photo. There is also technology to do.

In addition, a site that provides a service of displaying a photograph of a community site that can share a digital photograph on a map and performing a search using the map is also provided (see, for example, page 76 of Non-Patent Document 1).
Ambient Findability, Publisher: Oreilly & Associates Inc (2005/10), ISBN-10: 0596007655, ISBN-13: 978-0596007652

  As described above, there are two types of conventional image search technologies based on the browsing action and the uploading action.

  Among these services, the service based on the action on the browsing side can be used only after an unspecified number of users actively use it. Therefore, it is difficult to search for information immediately after publishing an image or video, and the number of users is limited for information that is disclosed only to specific communities, so that these technologies can be used for searching. It is difficult to do.

  On the other hand, in a service that presupposes an action on the uploading side, information such as a shooting position is manually input. However, a storage medium that can store several hundred photos is not uncommon. When participating in a specific event, it may be possible to take more than 100 images per day, but it is not practical to manually add information to these photos.

  It is also possible to automatically detect the shooting position using the GPS function and use it as metadata, but the camera attached to the mobile phone with the GPS function has a limited resolution, carry a digital camera with high resolution with GPS, In the first place, the service of assigning a shooting position to a photograph is accompanied by the trouble of carrying the GPS with the camera.

  Moreover, the GPS service currently used with the camera has an error of about 10 [m], and the accuracy is insufficient to identify the location in the specific event venue.

  In addition, metadata called exif (Exchangeable Image File Format) is added to digital photos, whereas there are formats in which video time cannot be stored as metadata. Such a file cannot be searched by shooting time.

  The present invention has been made to solve these problems, and searches for an image that the user wants to view from a vast number of moving images and still images taken at a specific location. In order to realize the provision to the user, a new action that does not require the active action of the browsing user and the posting user, obtains shooting position information with higher accuracy than GPS, and thereby enables an intuitive search. The purpose is to provide image providing technology.

  In order to achieve this object, the image providing apparatus of the present invention searches for an image satisfying the user's browsing request from images taken at a specific location and provides the user with the image. (1) An input unit that inputs a moving image or a still image shot at a specific place and stores it in the storage unit; (2) An image input by the input unit and an environmental model of the specific place are collated. , An estimation unit for estimating the shooting position and shooting angle of the image input by the input unit; and (3) information on the shooting position and shooting angle of the image estimated by the estimation unit in association with the image stored in the storage unit. (4) When there is an image browsing request specifying information on the viewpoint position and / or line-of-sight angle from the user, information on the shooting position and shooting angle stored in the storage means is retrieved. The user's finger Search means for searching for an image that can be seen from the viewpoint position and / or line-of-sight direction, and (5) an environment model image of a specific place that is visible from the viewpoint position and / or line-of-sight direction specified by the user Display means for displaying a list of summary images indicating an outline of the image searched by the search means on the image.

  When this configuration is adopted, the similarities of the voices of the images stored in the storage means are detected, and based on the detected similarities, the shooting times of the images having no shooting time information stored in the storage means are estimated. Second estimation means may be provided.

  When the second estimating unit is provided, the recording unit further records the shooting time information for the image having the shooting time information in the image input by the input unit, and has the shooting time information. For images that do not exist, the information of the photographing time estimated by the second estimating means is further recorded. Then, when there is an image browsing request that also specifies the shooting time information from the user, the searching means also searches for the shooting time information stored in the storage means, thereby obtaining the shooting time specified by the user. The photographed image is searched.

  Each of the above processing means can also be realized by a computer program. This computer program is provided by being recorded on an appropriate computer-readable recording medium or provided via a network, and is used when implementing the present invention. The present invention is realized by being installed and operating on a control means such as a CPU.

  In the image providing apparatus of the present invention configured as described above, when a moving image or a still image taken at a specific place is input and stored in the storage means, the input image is collated with an environmental model at the specific place. Thus, the shooting position and shooting angle of the input image are estimated, and information on the estimated shooting position and shooting angle is recorded in association with the image stored in the storage unit. At this time, with respect to an image having shooting time information in the input image, the shooting time information is also recorded.

  Here, in order to ensure the estimation of the shooting position and shooting angle of an image, when inputting a moving image, a specific place is selected using the most zoomed-out image selected from the moving images. By collating with the environmental model, it is possible to estimate the shooting position and angle of the moving image, or to collate with the environmental model at a specific location using the panoramic image created from the moving image. The shooting position and shooting angle of the moving image may be estimated.

  In this way, the storage means stores moving images and still images shot by a large number of posting users at a specific location, and records information on the shooting positions and shooting angles of those images. It will be done.

  At this time, the image having the shooting time information is further recorded with respect to the shooting time information. Therefore, for the image having no shooting time information, the image stored in the storage means is stored. The similarities of the speech possessed are detected, and based on this, the shooting time is estimated and recorded.

  From now on, in the storage means, moving images and still images taken by a large number of posting users in a specific place will be stored, and information on the shooting positions and shooting angles of those images will be stored. The shooting time information of these images is recorded.

  Upon receiving images stored in the storage means and attribute information of those images, and when there is a browsing request for an image designating information on the viewpoint position and line-of-sight angle from the viewing user, the shooting position and the shooting stored in the storage means By searching for the angle information, an image that can be seen from the viewpoint position or line-of-sight direction specified by the viewing user is searched.

  Then, in order to provide the search result to the browsing user, an image of the environmental model of a specific place that is visible from the viewpoint position and the line-of-sight direction specified by the browsing user is displayed on the display, and the search is performed on the image. A summary image list showing an overview of the selected images is displayed.

  At this time, if there is an image browsing request that also specifies the shooting time information from the browsing user, the shooting time information stored in the storage means is also searched to capture the shooting time specified by the browsing user. The retrieved image will be searched.

  In addition, when collating the input image with the environmental model of a specific location, based on the degree of collation, calculate the certainty of estimating the shooting position and shooting angle of the input image, and Thus, in order to provide an image that more closely matches the browsing request, a final search result may be obtained by selecting an image with a higher certainty factor from the searched images in priority.

  In addition, in order to make it possible for the viewing user to know which shooting location image is provided, the image of the environmental model associated with the shooting location indicated by the shooting position and shooting angle of the searched image A summary image of search results may be displayed at the upper position.

  In addition, in order to be able to grasp what kind of inclination the image is provided to the browsing user, the state when the browsing user browses according to the shooting angle of the searched image. In some cases, the search result summary image is tilted and displayed.

  Also, in order to make it possible for the viewing user to grasp whether the image was taken at a distance or an image taken at a distance, the image is taken at a distance according to the shooting position of the searched image. The size of the summary image of the search result may be changed and displayed in a form that becomes smaller as the image is displayed.

  According to the present invention, by having a step of registering a plurality of moving images and still images and constructing an environmental model, there is an effect of acquiring shooting position information with higher accuracy than GPS, and a shooting angle can also be determined. effective.

  In addition, it automatically assigns the shooting position, shooting angle, and shooting time to each of a large number of moving images and still images, and by using them, it is necessary for the viewer / poster to take an active action for searching. There is no effect.

  Moreover, there is an effect that an intuitive search can be provided to a viewer by creating an environment model and displaying it.

  In addition, even for a moving image having no time information, there is an effect that the time information is estimated by attempting to synchronize sound with an already registered image, and the search target is further expanded.

  Hereinafter, the present invention will be described in detail according to embodiments.

  FIG. 1 illustrates a system configuration of an image providing system including the present invention.

  The image providing system including the present invention is a moving image or a still image (hereinafter referred to as a moving image or a still image) that satisfies a browsing user's browsing request from a moving image or a still image taken at a specific event venue. In order to perform this process, an environment model calculation information input device 10 and an environment model calculation device are provided as shown in FIG. 11, environmental model storage device 12, image input device 13, image storage device 14, estimation device 15, image attribute information storage device 16, search condition input device 17, image search device 18, and output device 19.

  The environmental model calculation information input device 10 inputs information necessary for calculating an environmental model (three-dimensional solid model) of a specific event venue to be photographed. The environmental model calculation device 11 calculates the environmental model of the event venue to be photographed based on the information input by the environmental model calculation information input device 10. The environmental model storage device 12 stores the environmental model calculated by the environmental model calculation device 11.

  The image input device 13 inputs a moving image or a still image taken at a specific event venue. The image storage device 14 stores the image input by the image input device 13.

  The information stored in the image storage device 14 will be further described. If the image input by the image input device 13 is a still image, in addition to storing the still image file, FIG. ), Information such as an image ID, an image file type, a contributor ID, and a file name is stored. When the image input by the image input device 13 is a moving image, in addition to storing the moving image file and the representative image file extracted from the moving image, FIG. As shown in FIG. 4, information such as a moving image ID, a moving image file type, a representative image file type, a contributor ID, a moving image file name, and a representative image file name is stored.

  The estimation device 15 collates the image stored in the image storage device 14 with the environment model stored in the environment model storage device 12, thereby obtaining the shooting position and shooting direction (shooting angle) of the image input by the image input device 13. ) Is detected, and the similarities of the voices of the images stored in the image storage device 14 are detected. On the basis of the detected similarities of the images stored in the image storage device 14, Estimate the shooting time.

  As described in Reference Document 1 below, a technique for estimating not only an image shooting position but also an image shooting direction using image processing is disclosed. In this technique, an environmental model is actually photographed, feature points are extracted from the photographed image sequence, a 3D model of the subject is restored, and a photographed place is estimated.

  In this estimation device 15, for example, by using the technique described in this reference 1, the image stored in the image storage device 14 and the environmental model stored in the environment model storage device 12 are collated, thereby obtaining an image. Processing is performed so as to estimate the shooting position and shooting direction of the image input by the input device 13.

[Reference 1] Carlo Tomasi, Takeo Kanade: "Shape and Motion from Image Streams", International Journal of Computer Vision Volume 9, Number 2, pp.137-154 (1992).
The image attribute information storage device 16 stores information on the shooting position and shooting angle of the image estimated by the estimation device 15 while corresponding to the image stored in the image storage device 14 and information on the shooting time of the image ( The information on the photographing time of the image input by the image input device 13 and the information on the photographing time estimated by the estimating device 15) are stored.

  Here, the image storage device 14 and the image attribute information storage device 16 correspond to storage means described in the claims.

  If the information stored in the image attribute information storage device 16 is further described, when the image stored in the image storage device 14 is a still image, it corresponds to the image ID as shown in FIG. In addition, information such as the shooting time, the shooting position, the shooting angle, the certainty of estimation, the number of subjects, and the popularity is stored. When the image stored in the image storage device 14 is a moving image, as shown in FIG. 3B, the shooting start time, the shooting end time, the shooting position, and the shooting angle are associated with the moving image ID. , Information such as synchronization list, estimation confidence, popularity.

  The search condition input device 17 receives a search request issued by a browsing user, and inputs an image search condition that specifies information on the viewpoint position and the line-of-sight angle. The image search device 18 uses the search condition input by the search condition input device 17 as a key to search for information on the shooting position, shooting angle, and shooting time of the image stored in the image attribute information storage device 16. The image indicated by the search condition input by the input device 17 is searched, and three-dimensional drawing data representing the search result (3D map drawing data indicating the image of the examination result in association with the 3D event venue) is generated. . The output device 19 outputs three-dimensional drawing data representing the search result generated by the image search device 18.

  FIG. 4 shows a flowchart executed by the image providing system having the present invention configured as described above.

  Next, processing executed by the image providing system including the present invention configured as described above will be described according to this flowchart.

  As shown in the flowchart of FIG. 4, in the image providing system including the present invention, first, in step S100, the management user obtains the environmental model of the event venue to be photographed by calculation and stores it in the environmental model storage device 12. sign up.

  Specifically, there are the following methods for building this environmental model.

(B) Method 1
An administrative user prepares software capable of handling CAD and 3D in the environmental model calculation device 11 and inputs information written in the design drawing of the event venue desired to be 3D from the environmental model calculation information input device 10. The result is assigned to the environmental model calculation device 11, and the resulting event venue environmental model is transferred to the environmental model storage device 12 and stored therein.

(B) Method 2
A video camera is used to shoot an event venue to be restored to 3D from two or more locations that are sufficiently separated by the management user. The captured moving image is encoded in a format such as avi (Audio Video Interleaving) or mpg (Moving Picture Experts Group), and the data is transferred from the environmental model calculation information input device 10 to the environmental model calculation device 11. The environmental model calculation device 11 extracts feature points, detects immovable objects, and constructs a 3D environment model of the event venue by matching the feature points. The obtained environmental model is transferred to and stored in the environmental model storage device 12.

  Here, for extracting feature points, a Harris operator, a Moravec operator, or the like described in Reference Document 2 below can be used. Further, immovable objects can be detected by tracking feature points, and feature points can be extracted using the KLT method or the like. In addition, the Tomasi and Kanade factorization method described in the above-mentioned Reference 1 can be used for 3D conversion from feature point matching. Further, data is transferred from the environmental model calculation information input device 10 to the environmental model calculation device 11 via the Internet. It is also possible to upload a moving image shot with a mobile phone in a format such as mpg. Of course, the environmental model calculation information input device 10 and the environmental model calculation device 11 may be a single device, and in that case, it is not necessary to connect via a network.

[Reference 2] Jun Kanazawa, Kenichi Kanaya: “Extraction of image feature points for computer vision” Journal of IEICE Vol.87, No.12 pp.1043-1048 (2004).
(C) Method 3
The management user takes a sufficient number of still images of the event venue that he / she wants to restore in 3D, encodes it in a format such as jpg (Joint Photographic Experts Group), and transfers the data from the environmental model calculation information input device 10 to the environmental model calculation device 11 . The environment model calculation device 11 extracts feature points and constructs a 3D environment model of the event venue by matching the feature points. The obtained environmental model is transferred to and stored in the environmental model storage device 12.

  Here, when the environment model of the event venue is constructed by the method 2 or the method 3, it is necessary to photograph the event venue with a camera or a video camera and extract the feature points for each image. At this time, in order to improve the accuracy of the environmental model, it is necessary to prevent subjects other than the event venue from being included in the image, and if subjects other than the event venue are included in the image, zoom It is preferable to use a device such as using an out image or generating and using a panoramic image.

  Using these methods, in step S100, the environmental model of the event venue to be photographed is obtained and registered in the environmental model storage device 12. Subsequently, in step S101, the image taken by the posting user at the event venue and the user himself / herself are recorded. Is input from the image input device 13 and transferred to the image storage device 14.

  Thereby, as shown in FIG. 2, the image storage device 14 stores information such as an image taken by the posting user at the event venue and the name of the poster.

  Subsequently, in step S102, a feature point is extracted for each image stored in the image storage device 14 in accordance with the method described in Reference Document 1 described above, and the feature point coordinates stored in the environment model storage device 12 are extracted. And the location where each image was shot and the shooting direction are estimated. Furthermore, the image capturing time is estimated for an image of which the image capturing time is unknown by extracting the similarities of the sound possessed by the image. Then, information such as the estimated coordinates (location), direction, and time is stored in the image attribute information storage device 16 together with the image ID and the moving image ID.

  Thus, as shown in FIG. 3, the image attribute information storage device 16 corresponds to each image stored in the image storage device 14, and the shooting time, shooting location, shooting angle, etc. of those images. Is stored.

  The process of step S102 will be described in detail according to the flowcharts of FIGS.

  Subsequently, in step S103, a search condition such as whether the viewing user is limited to a viewpoint position, an angle, a time range, or a photograph showing a face is input from the search condition input device 17, and the search condition 3D is satisfied. A map is generated and displayed on the output device 19.

  That is, based on the feature point information stored in the environmental model storage device 12, the event venue image specified by the search condition (the event venue image that will be seen by the viewing user) is output and displayed on the output device 19. Based on the information stored in the image attribute information storage device 16, an image satisfying the search condition is determined, and the determined image (representative image in the case of a moving image) is read out from the image storage device 14 to create a thumbnail to create an event venue. By outputting and displaying the image on the output device 19 in a superimposed manner, a 3D map indicating the search result image is output and displayed in association with the 3D event venue.

  The process of step S103 will be described in detail according to the flowchart of FIG.

  Next, the process executed in step S102 will be described in detail according to the flowcharts of FIGS.

  Before entering the process of step S102, as shown in the flowchart of FIG. 5, first, in step S200, information such as a photograph (still image) taken by the posting user at the event site, a moving image, and his / her name is displayed. The information input from the image input device 13 is transferred to the image storage device 14. That is, the process of step S101 in the flowchart of FIG. 4 is executed.

  Thereafter, the process of step S201 is entered, and the process described below is repeatedly executed for the input file (photograph or moving image) in steps S201 to S214. Here, this processing is executed by the estimation device 15.

  That is, in step S202, it is determined whether the input file is a still image file or a moving image file from the extension of the input file and the file header.

  When it is determined that the input file is a moving image file in accordance with the determination process in step S202, the process proceeds to step S203 to verify whether a panoramic image of the moving image file can be created. A panoramic image is an image generated by synthesizing an image sequence from a section including camera operations such as panning, tilting, and zooming. Whether a panoramic image can be created by the method described in Reference 3 below. It is possible to determine.

[Reference 3] Yukinobu Taniguchi, Akihito Akutsu, Yoshinobu Tonomura: “Panorama Excerpts: Video List by Automatic Generation and Layout of Panoramic Images” IEICE Transactions D-II Vol.J82-D-II, No .3 pp.390-398 (1999).
When it is determined that it is possible to create a panoramic image of the input moving image file according to the determination process of step S203, the process proceeds to step S204, and the method described in Reference 3 is used to input the panoramic image from the input moving image file. A panoramic image is created and stored in the image storage device 14 as a representative image. By using a panoramic image as a representative image in this way, a large number of event venues are included as a background, thereby enabling more features to be extracted and capturing images according to the processing described below. When estimating a place and a shooting direction, the estimation accuracy can be increased.

  On the other hand, when it is determined in step S203 that it is impossible to create a panoramic image of the input moving image file, the process proceeds to step S205, and the moving image is in focus. The most zoomed-out image is selected from the images, and is stored in the image storage device 14 as a representative image. By using the most zoomed-out image as a representative image in this way, a lot of event venues are included as a background, and more features can be extracted, which will be described later. When estimating the shooting location and shooting direction of an image according to the processing, the estimation accuracy can be improved.

  Subsequently, in step S206, an image captured at the same time from the already registered moving images is estimated and detected. If detected, it is registered in the synchronization list (the synchronization list shown in FIG. 3B). A method for synchronizing time of moving images will be described in detail with reference to a flowchart of FIG.

  Subsequently, in step S207, according to the time synchronization process, it is determined whether or not the information of the shooting time taken at the same time can be detected. If it is determined that the information cannot be detected, the process proceeds to step S208. Give up the automatic detection of time and set the variable time_s = 0 (not set).

  On the other hand, when it is determined that the input file is a still image file (photograph) in accordance with the determination process in step S202, the process proceeds to step S209 to check whether the still image file holds exif information.

  When it is confirmed that the input still image file holds exif information according to the confirmation processing in step S202, the process proceeds to step S210, and the shooting time recorded in the exif information is held in the variable time. If it is determined that the input still image file does not hold the exif information, the process proceeds to step S208 to give up the automatic detection of the shooting time and set the variable time = 0 (not set).

  Subsequently, in step S211, if the input image is a still image, a human face is extracted from the still image and the number of the extracted faces is held, while the input image is a moving image. If it is an image, a human face is extracted from the representative image of the moving image, and the number of extracted faces is held.

  In step S212, if the input image is a still image, feature points of the still image are extracted. On the other hand, if the input image is a moving image, the representative image of the moving image is extracted. Extract feature points. Then, by collating with the feature points stored in the environment model storage device 12, it is estimated from which position and from which direction the input image was taken, and held together with the certainty factor. Here, the certainty factor is calculated as a function of the number of feature points used for shooting position estimation and the number not used.

  Subsequently, in step S213, the value of information obtained by the operation from step S200 (information excluding popularity in the information shown in FIG. 3) is stored in the image attribute information storage device 16.

  Next, the moving image time synchronization processing executed in step S206 will be described with reference to the flowchart of FIG.

  In step S206, as shown in the flowchart of FIG. 6, first, in step S300, a moving image file to be processed (moving image file input in step S200) is input from the posting user.

  In step S301, it is determined whether or not the input moving image file has shooting time metadata (metadata to be recorded for shooting start and end times).

  When it is determined that the input moving image file does not have the shooting time metadata in accordance with the determination process in step S301, the process proceeds to step S302 to indicate that the shooting time is not set. On the other hand, when it is determined that it has metadata of shooting time, the process proceeds to step S303, and the start and end times of shooting are substituted into time_s and time_e, respectively.

  Subsequently, in step S304, the sound of the input moving image file is extracted, and in steps S305 to S315, the processing described below is repeated for all moving image files registered in the past with the extracted sound as a processing target. Execute.

  That is, in step S306, the sound of the moving image file registered in the past is extracted, and a portion (synchronized portion) that matches the sound of the input moving image file is searched.

  Specifically, for example, both the sound of the input moving image file and the sound of the moving image file registered in the past are cut at intervals of 500 [ms], and each cut has a sound pressure exceeding a certain threshold. If yes, it is checked whether the sound of the moving image file registered in the past does not match. Specifically, the method described in Reference Document 4 below can be used. Then, when the number of matching parts exceeds the threshold, it is assumed that the two sounds are synchronized.

[Reference 4] Kunio Kanno, Gavin Smith, Hiroshi Murase: "Fast Search Method for Acoustic Signals Using Histogram Features" IEICE Transactions D-II Vol.J82-D-II, No.9 pp.1365- 1373 (1999).
Subsequently, in step S307, it is determined whether or not the synchronized sound has been detected in accordance with the search process in step S306, and when it is determined that the synchronized sound has not been detected, the process proceeds to step S315. The process is transferred to the moving image file registered in the past.

  On the other hand, when it is determined that the sound to be synchronized has been detected according to the determination process in step S307, the process proceeds to step S308, and each ID (moving image ID) is displayed in each other's synchronization list (the synchronization list shown in FIG. 3B). ).

  Subsequently, in step S309, it is determined whether or not the moving image file registered in the past that is currently processed does not have time information and the time information of the input moving image file is fixed. to decide.

  According to the determination process in step S309, it is determined that the currently registered moving image file currently processed does not have time information, and the time information of the input moving image file is not in a fixed state. If so, the process proceeds to step S310, in which whether or not the time information of the moving image file registered in the past that is currently processed is fixed and the input moving image file has no time information. Judging.

  According to the determination process in step S310, it is determined that the time information of the previously registered moving image file being processed is fixed and the input moving image file does not have the time information. In some cases, the process proceeds to step S315, and the process proceeds to the next moving image file registered in the past.

  On the other hand, according to the determination process in step S310, it is determined that the time information of the previously registered moving image file being processed is fixed and the input moving image file has no time information. In step S311, the shooting start time and end time of the input moving image file are set from the synchronization point between the sound of the input moving image file and the sound of the currently registered moving image file. After estimating and substituting them into time_s and time_e of the input moving image file, the process proceeds to step S315, and the process proceeds to the next past moving image file.

  On the other hand, according to the determination processing in step S309, it is confirmed that the currently registered moving image file currently processed does not have time information, and that the time information of the input moving image file has been confirmed. When the determination is made, the process proceeds to step S312, and the previously registered moving image file currently processed from the synchronized portion of the sound of the input moving image file and the sound of the previously registered moving image file. The shooting start time and end time are estimated and substituted for time_s and time_e of the previously registered moving image file currently processed.

  Subsequently, in step S313, it is determined whether or not there is an unknown shooting time in the synchronization list of the moving image file registered in the past.

  If it is determined in accordance with the determination processing in step S313 that there is no unknown shooting time in the synchronization list of the previously registered moving image file, the process proceeds to step S315, and the next registered moving image file is registered in the past. Migrate processing.

  On the other hand, when it is determined that there is an unknown shooting time in the synchronization list of the moving image file registered in the past according to the determination processing in step S313, the process proceeds to step S314, and the moving image files connected in the synchronization list are processed. In step S315, the shooting time is recursively estimated and substituted for all the shooting times unknown. Then, the process proceeds to step S315, and the process proceeds to the next moving image file registered in the past.

  In the moving image time synchronization processing executed in the flowchart of FIG. 6 described above, since the moving images were taken at the same event venue, for example, cheering or athletic meet at a baseball cheer song or soccer goal Focusing on the fact that the ambient sound is also picked up by other moving images, such as the announcement in the field, and it is possible to estimate the shooting time by synchronizing the sound from now on. The image capturing time is estimated.

  For example, as shown in FIG. 7, when the shooting time of the moving image A is known and the shooting time of the moving image B is not known, a matching portion between the sound of the moving image A and the sound of the moving image B is obtained. Based on this, the shooting time (shooting start time and shooting end time) of the moving image B is estimated. Further, for example, as shown in FIG. 8, when posting is made in the order of moving image A, moving image B, moving image C, moving image D and only the shooting time of moving image D is known, A matching portion between the sound of the image A and the sound of the moving image B, a matching portion of the sound of the moving image B and the sound of the moving image C, and a matching portion of the sound of the moving image C and the sound of the moving image D. Then, based on this, the shooting times (shooting start time and shooting end time) of the moving images A, B, and C are estimated.

  In this way, the image providing system including the present invention obtains the environmental model of the event venue to be photographed and registers it in the environmental model storage device 12 in accordance with the flowchart of FIG. When information such as the image taken at the venue and the name of the contributor is stored in the image storage device 14 (step S101), the process enters step S102, and the flowcharts of FIGS. 5 and 6 are executed to store the image. A feature point is extracted for each image stored in the device 14 and collated with the feature point coordinates stored in the environmental model storage device 12 to estimate the location where the image was captured and the direction of the capture. Further, by extracting similarities of the voices of the images, the shooting time of the image whose shooting time is unknown is estimated, and information such as the estimated coordinates (location), direction, and time, together with the image ID and moving image ID, is estimated. It is stored in the image attribute information storage device 16.

  Thus, as shown in FIG. 3, the image attribute information storage device 16 is associated with each image stored in the image storage device 14 while taking the image capturing time, image capturing location and image capturing. Information such as the angle is stored.

  As described in step S103 of the flowchart of FIG. 4, when information such as the shooting time, shooting location, and shooting angle of an image shot by the posting user is stored in the image attribute information storage device 16, the information is stored. And processing to provide the browsing user with an image requested to be viewed.

  Next, the process executed in step S103 will be described in detail according to the flowchart of FIG. Here, the image search device 18 executes the processing of this flowchart.

  In step S103, as shown in the flowchart of FIG. 9, first, in step S400, the viewpoint position, the angle, the time range, and the face are captured from the browsing user via the search condition input device 17. Enter search conditions that describe whether you want to limit the images to

  Subsequently, in step S401, based on the feature point information stored in the environmental model storage device 12, an easy-to-see 3D map of the event venue viewed from the input viewpoint position and angle is created and rendered on the output device 19 Output.

  Here, the visibility index is a function of the viewpoint position and angle. From the viewpoint position, determine whether you are standing on the ground or looking down from above, and decide whether to make a perspective view.

  Subsequently, in step S402, by searching the information stored in the image attribute information storage device 16, (1) an image that can be seen from the viewpoint position and angle specified by the search condition, and (2) specified by the search condition. If the image is taken within the time range and (3) it is limited that the face is captured by the search condition, it is an image that satisfies that, and (4) the image was taken on the 3D map. The ID of an image satisfying the condition that the thumbnail can be drawn at the place is extracted.

  If the ID of the image satisfying the above condition cannot be extracted in step S402, the process ends. On the other hand, if the ID of the image satisfying the above condition can be extracted, the process proceeds to step S403. The image with the highest evaluation is selected from the extracted images (a representative image is used in the case of a moving image). Here, the evaluation of the image is a function of the certainty of the shooting location, the popularity, the number of faces in the image, and the like.

  Subsequently, in step S404, when the thumbnail is drawn at the shooting position on the 3D map, it is determined whether or not the thumbnail of the new image can be displayed more than half of the thumbnail of another image already displayed. to decide.

  For example, when the current output is as shown in FIG. 10 and the next image is the image G shown in FIG. 11, more than half of the area of the image G is hidden by the images B and C and cannot be displayed. . It is determined whether or not such a state occurs. Here, the size of the thumbnail is a function of how to write the 3D map and the distance from the viewpoint position.

  If it is determined in step S404 that the image selected in step S403 is an image that cannot be displayed by more than half, the process proceeds to step S405, and the image selected in step S403 is regarded as being undrawn and the drawing is given up. Then, the condition is removed so that the image is not selected in S402, and the process returns to step S402.

  On the other hand, when it is determined in the determination process in step S404 that the image selected in step S403 is an image that can be displayed by more than half, the process proceeds to step S406, and the image ID selected in step S403 is indicated by the image storage device 14. An image (a representative image in the case of a moving image) is read out, a thumbnail of the read image is generated, and the image is drawn and output at the shooting position of the image on the 3D map of the output device 19, and then the process returns to step S402. .

  In this way, the image providing system including the present invention executes the flowcharts of FIGS. 5 and 6 to thereby obtain information such as the shooting time, shooting location, and shooting angle of the image shot by the posting user. Is stored in the image attribute information storage device 16, the image of the event venue that will be seen by the viewing user is output and displayed on the output device 19 by executing the flowchart of FIG. An image satisfying the search condition is determined based on the information to be read, the image (the representative image in the case of a moving image) is read from the image storage device 14, a thumbnail is created, and output in a form superimposed on the event venue image By outputting and displaying on the device 19, the output device 19 displays a search result image in association with the 3D event venue. It is to output display the flop.

  FIGS. 10 to 13 show an example of the output display of the 3D map showing the search result image.

  FIG. 10 shows a bird's-eye view of the event venue. By defining only the time range and overlooking the entire event venue, you can specify the location you want to investigate in more detail. The image F in the figure is a panoramic image created from a moving image.

  FIG. 11 is a diagram for explaining that when the display of FIG. 10 is performed and the image G is searched, the image G is not a drawing target.

  FIG. 12 is a view from the ground. Images taken from a distance are output smaller. In addition, when it is desired to perform a search using the photographed angle, the image may be given an angle. FIG. 13 shows an example using an angled image.

  By enabling such switching of the display form, it is possible to grasp the rough image layout by inputting the time and overlooking the event venue. Then, if you switch to the output seen from the ground viewpoint, you can search for images as if you were actually standing there. Furthermore, if the output is switched to the output using the estimated shooting angle, the image can be limited to a photograph of a specific subject and the search becomes easier.

  When the browsing user selects one thumbnail from the thumbnails displayed in this way and requests detailed display, processing is performed to display the original image from which the selected thumbnail was created.

  The present invention can be applied to a case where an image that a user wants to see is retrieved from a vast number of moving images and still images captured at a specific location and provided to the user. Using the shooting direction and shooting time as clues, it is possible to search for and provide the user with an image that the user wants to see.

1 is a system configuration diagram of an image providing system including the present invention. It is explanatory drawing of the information which an image memory | storage device memorize | stores. It is explanatory drawing of the information which an image attribute information storage device memorize | stores. It is a flowchart which the image provision system which comprises this invention performs. It is a flowchart which the image provision system which comprises this invention performs. It is a flowchart which the image provision system which comprises this invention performs. It is explanatory drawing of the time synchronization process of a moving image. It is explanatory drawing of the time synchronization process of a moving image. It is a flowchart which the image provision system which comprises this invention performs. It is a figure which shows an example of the output display of a search result. It is a figure which shows an example of the output display of a search result. It is a figure which shows an example of the output display of a search result. It is a figure which shows an example of the output display of a search result.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Environmental model calculation information input device 11 Environmental model calculation device 12 Environmental model storage device 13 Image input device 14 Image storage device 15 Estimation device 16 Image attribute information storage device 17 Search condition input device 18 Image search device 19 Output device

Claims (10)

An image providing device for searching for an image satisfying a user's browsing request from images taken at a specific location and providing the image to the user,
An input means for inputting a moving image or a still image taken at the place and storing it in the storage means;
An estimation unit that estimates an imaging position and an imaging angle of the image input by the input unit by comparing the image input by the input unit with the environmental model of the place;
A recording unit that records information of a shooting position and a shooting angle of the image estimated by the estimation unit in association with an image stored in the storage unit;
When there is an image viewing request specifying information on the viewpoint position and / or line-of-sight angle from the user, the information on the shooting position and shooting angle stored in the storage means is searched, and the viewpoint position specified by the user and And / or search means for searching for an image that will be seen from the line-of-sight direction;
An image of the environmental model that is visible from the viewpoint position and / or line-of-sight direction specified by the user is displayed on the display, and a summary image list showing an overview of the image searched by the search means is displayed on the image. Providing display means for
An image providing apparatus. The image providing apparatus according to claim 1,
Second estimation for detecting a similar point of the sound stored in the storage unit and estimating the shooting time of an image having no shooting time information stored in the storage unit based on the detected similarity With means,
The recording means further records the information of the photographing time for the image having the information of the photographing time among the images input by the input means, and the second means for the image having no information of the photographing time. Further record the information of the shooting time estimated by the estimation means,
When there is an image browsing request that also specifies the shooting time information from the user, the search means also searches the shooting time information stored in the storage means to obtain the shooting time specified by the user. Searching for images that were taken
An image providing apparatus. The image providing device according to claim 1 or 2,
When the input unit inputs a moving image, the estimation unit performs matching with the environmental model using the most zoomed-out image selected from the moving images, thereby obtaining the moving image. Estimating the shooting position and shooting angle,
An image providing apparatus. The image providing device according to claim 1 or 2,
When the input unit inputs a moving image, the estimation unit performs collation with the environmental model using a banorama image created from the moving image, so that a shooting position and a shooting angle of the moving image are obtained. To estimate
An image providing apparatus. The image providing apparatus according to any one of claims 1 to 4,
The estimation means calculates the certainty factor of the estimation based on the degree of matching,
The search means obtains a final search result by selecting an image in a form giving priority to the image with a high certainty factor from the searched images.
An image providing apparatus. The image providing apparatus according to any one of claims 1 to 5,
The display means displays the summary image at a position on the image of the environmental model associated with the shooting position of the image searched by the search means and the shooting position indicated by the shooting angle.
An image providing apparatus. The image providing apparatus according to any one of claims 1 to 6,
The display means tilts and displays the summary image in a form that matches the state when the user browses according to the shooting angle of the image searched by the search means.
An image providing apparatus. The image providing apparatus according to any one of claims 1 to 6,
The display means displays the summary image by changing the size of the summary image in a form that becomes smaller as the image taken farther according to the shooting position of the image searched by the search means.
An image providing apparatus. An image providing method executed by an image providing apparatus that searches an image that satisfies a user's browsing request from images taken at a specific location and provides the image to the user,
A process of inputting a moving image or a still image captured at the place and storing it in the storage means;
The process of estimating the shooting position and shooting angle of the input image by collating the input image with the environmental model of the place;
In association with the image stored in the storage means, the process of recording the estimated shooting position and shooting angle information of the image,
When there is an image viewing request specifying information on the viewpoint position and / or line-of-sight angle from the user, the information on the shooting position and shooting angle stored in the storage means is searched, and the viewpoint position specified by the user and And / or the process of searching for an image that will be visible from the line of sight,
Displaying the image of the environmental model to be seen from the viewpoint position and / or line-of-sight direction specified by the user on the display, and displaying a summary image list indicating the outline of the searched image on the image; Having
An image providing method as a feature. An image providing program used for realizing an image providing apparatus that searches for an image that satisfies a user's browsing request from images taken at a specific location and provides the image to the user,
Computer
An input means for inputting a moving image or a still image taken at the place and storing it in the storage means;
An estimation unit that estimates an imaging position and an imaging angle of the image input by the input unit by comparing the image input by the input unit with the environmental model of the place;
A recording unit that records information of a shooting position and a shooting angle of the image estimated by the estimation unit in association with an image stored in the storage unit;
When there is an image viewing request specifying information on the viewpoint position and / or line-of-sight angle from the user, the information on the shooting position and shooting angle stored in the storage means is searched, and the viewpoint position specified by the user and And / or search means for searching for an image that will be seen from the line-of-sight direction;
An image of the environmental model that is visible from the viewpoint position and / or line-of-sight direction specified by the user is displayed on the display, and a summary image list showing an overview of the image searched by the search means is displayed on the image. An image providing program for functioning as display means.

Images