JP2017033382A - Image information retrieval server, image information retrieval method and user terminal - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide means that calls up a memory of past behaviors taken by users by means of images.SOLUTION: An image information retrieval server 20 is connected to an imaging terminal 10 having imaging means via a network, and the image information retrieval server 20 comprises: image storage means 28 that store a photographing image photographed by photograph means from the imaging terminal 10 as a retrieval object image (useful in calling up a memory of past behaviors by users); means that receives an image retrieval request including identification information relevant to the user and a retrieval word from a user terminal 30 of the user; image extraction means 23 that extracts an image relevant to the retrieval word by the identification information relevant to the user with reference to the image storage means 28; and means that transmits the image extracted by the image extraction means 23 to the user terminal 30. The image information retrieval server further comprises means that retrieves an image from the images in which not only an image keyword but also a retrieval image relevant to an image to be retrieved are registered.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an image information search server, an image information search method, and a user terminal for image information search.

  Humans often forget their past actions. For example, if you have forgotten to lock your front door when you go out, whether you've closed the kitchen gas tap, where you put your cell phone, and where you parked your car, you met before For example, you can't remember the name of a person, or forget the appointment time or place. In particular, there is a tendency for forgetfulness to become more severe as people get older.

  Among these, regarding the misplacement of goods, in the case of an electronic device such as a mobile phone, when it is determined that the electronic device has been misplaced (when the owner and the device are separated from a predetermined distance), an alarm is issued or registered. There is a known method of contacting location information or the like first.

  However, according to the above method, the target electronic device requires a function of measuring the distance between the owner and the corresponding electronic device, or such a device must be installed. There is a disadvantage of becoming high.

  Thus, for example, Patent Document 1 proposes a lost object search apparatus that recognizes an object that a person has left behind without adding the above function to the object. Here, a camera unit that captures a human visual area, an image storage unit that stores a template, and whether or not there is a template stored in the image storage unit in the image data captured by the camera unit are sequentially detected. A technique relating to an object retrieval apparatus including a control circuit that performs the above and a monitor that outputs image data is disclosed.

  By applying the technique disclosed in Patent Document 1, it is possible to recognize an object that a person has left behind without adding a special function to the object that may be lost. However, only pre-registered images are subject to search, and cannot be applied to unregistered images. Of course, forgetting things other than misplaced items cannot be dealt with.

JP 2002-329201 A

  In view of the above-described problems, the present invention provides an image information search server, an image information search method, and a user terminal that can recall a memory of actions taken by a user in the past. With the goal.

  In order to solve the above problems, the present invention provides the following solution.

  (1) An image information retrieval server connected to an imaging terminal having a photographing unit via a network, wherein the image information retrieval server stores a user's memory among the photographed images photographed by the photographing unit from the imaging terminal. Photographed image sorting means for sorting out images useful for evoke, and image storage means for storing the sorted images together with identification information related to the user as a search target image that is a search target from the user. Means for receiving an image search request including identification information related to the user and a search word from the user terminal of the user, and the identification information related to the user with reference to the image storage means. An image extracting means for extracting an image related to the search word from the search target image, and a method for transmitting the image extracted by the image extracting means to the user terminal. Characterized in that it comprises a and.

  (2) Further, in the image information search server according to (1), the image information search server further includes search image registration means for registering an image taken by the user as a search image, wherein the image extraction means includes the user terminal. When an image search request using the search image is received, an image related to the search image may be extracted from the search target image.

  (3) Moreover, in the image information search server according to (1) or (2), when the image element of interest is included in the captured image, the image storage unit stores the entire captured image, If the image element of interest is not included in the captured image, an image that is different from the previously stored image may be stored.

  (4) Further, in the image information search server described in (2), a target image or an image keyword added to the image based on the search word and / or the search image transmitted from the user terminal is selected. You may make it extract.

  (5) The image information search server according to (2) or (4) may further include means for selecting the registered search image from the user terminal when the image search request is made. Good.

  (6) Further, in the image information search server according to any one of (1) to (5), the imaging terminal includes glasses with a camera, a hat with a camera, accessories with a built-in camera, and other wearables having a camera function. It may be a terminal.

  (7) Moreover, in the image information search server according to any one of (1) to (6), according to the deletion request from the user terminal or the search results of the user, the captured image or the search target image You may make it have a picked-up image selection means to delete at least one part.

  (8) Moreover, in the image information search server according to any one of (1) to (7), the photographed image selection unit is configured as the search target image except for an image having no screen change among the photographed images. You may make it memorize | store.

  (9) According to a second aspect of the present invention, there is provided an image information search method in which an imaging terminal having a photographing unit and an image information search server are connected via a network, wherein the image information search server includes the image information search server, Receiving a photographed image photographed by the photographing means from an imaging terminal, selecting an image useful for invoking a user's memory among the photographed images, and providing the selected image to the user. Storing as a search target image to be searched from the user together with related identification information; receiving a search request based on the identification information and search word related to the user from the user terminal; When an image search request including identification information related to the user and the search word is received from the user terminal, the stored image for preventing forgetting of the user is detected. And extracting the identification information related to the user and the image related to the search word from the search target image, and transmitting the extracted image to the user terminal. .

  (10) According to a third aspect of the present invention, there is provided a user terminal connected via a network to an imaging terminal having an imaging unit, and is used among the captured images taken by the imaging unit from the imaging terminal. A picked-up image selecting means for selecting an image useful for invoking a user's memory, and an image for storing the selected image as a search target image to be searched by the user together with identification information related to the user A storage unit, a unit for obtaining an image search request including identification information and a search word related to the user, and a reference to the search word by the identification information related to the user with reference to the image storage unit. Image extraction means for extracting an image from the search target image; and means for displaying the image extracted by the image extraction means.

(11) According to a fourth aspect of the present invention, there is provided a user terminal having a photographing unit, wherein an image useful for evoking a user's memory is selected from the photographed images photographed by the photographing unit. Captured image sorting means, image storage means for storing the sorted image together with identification information related to the user as a search target image to be searched from the user, and identification information related to the user And an image extraction means for extracting an image related to the search word from the search target image based on identification information related to the user with reference to the image storage means. And means for displaying the image extracted by the image extraction means.

  ADVANTAGE OF THE INVENTION According to this invention, the image information search server, the image information search method, and user terminal which can call up the memory | storage of the action which the user took in the past by an image can be provided.

It is an image figure of the system containing the image information search server which concerns on the 1st Embodiment of this invention. It is an image figure which shows another form of the system containing the image information search server which concerns on the 1st Embodiment of this invention. It is a figure which shows the function structure of the image information search server which concerns on the 1st Embodiment of this invention. It is a figure which shows an example of the data structure memorize | stored in picked-up image DB of FIG. 3, and search object image DB. It is a figure which shows an example of the data structure memorize | stored in index information storage DB of FIG. It is a flowchart which shows the processing operation of the image information search server which produces | generates index information. It is a flowchart which shows the detail of the operation | movement which memorize | stores a picked-up image. It is a flowchart which shows the processing operation of the image information search server which registers the image for a search, and an image keyword. It is a flowchart which shows the image search process operation | movement of the image information search server which concerns on embodiment of this invention. It is a figure which shows an example which displays a search imaging screen. It is a figure which shows an example which displays the search imaging screen at the time of using a composite keyword. It is a flowchart which shows the processing operation of the image information search server which concerns on this embodiment when a user makes an image search request with an image. It is a figure which shows an example which displays a search imaging screen. It is a figure which shows the function structure of the user terminal which concerns on the 2nd Embodiment of this invention.

  DESCRIPTION OF EMBODIMENTS Hereinafter, embodiments for carrying out the present invention (hereinafter referred to as embodiments) will be described in detail with reference to the accompanying drawings. In the subsequent drawings, the same numbers or symbols are assigned to the same elements throughout the description of the embodiments. Further, the arrows between the functional blocks represent the data flow direction or the process flow direction.

(First embodiment)
FIG. 1 is an image diagram of a system including an image information search server according to the first embodiment of the present invention. Hereinafter, the above system will be referred to as “the present system”.

  This system includes an imaging terminal having an imaging function, a user terminal that displays an image, an image information search server (hereinafter referred to as an image information search server) that is connected to the imaging terminal and the user terminal via a network and searches for an image requested by the user terminal. Simply called a search server).

  In the following description, it is assumed that the user makes an inquiry to the search server as to whether or not the user has forgotten to close the entrance key. The user wears an imaging terminal, for example, a wearable terminal (such as glasses with a camera) that has a photographing function, links the user's past actions, photographs the field of view, and retrieves a captured image obtained by photographing. Send to server. The captured image may be a moving image (including sound), or may be continuous shooting of still images taken at predetermined intervals. Note that the user registers his / her identification information in the search server in advance. Here, “identification information” is information that identifies an imaging terminal or a user terminal used by the user, such as a unique address. Specifically, the user registers the service in the search server, so that the identification number of the imaging terminal and the user terminal (for example, a personal computer (PC), a smart phone, or a mobile phone (garage)) that browses the image. ) And register the identification number. In this way, viewing of images from a third party can be locked, leading to privacy protection.

  The search server stores the received captured image in a database managed for each user, further selects an image that is useful for evoking the user's memory among the captured images, and uses the selected image It is stored in the database as a search target image that is a search target from a person. The search target image, for example, “forgetting to close the entrance key” means an image of the entrance door or the like in which the opening and closing of the door is imaged. Further, in the case of a moving image, the search server generates and stores index information for each scene (scene) of the captured image. The scene of the captured image is stored as one scene by detecting a change in contrast and a change in motion of the captured image. Since the method for detecting a scene change is known, for example, from Japanese translation of PCT publication No. 2000-513848, a detailed description thereof will be omitted.

  In order to evoke the memory of forgetting to close the front door key when going out, the user inputs a search word such as “entrance” or “key” to the search server and makes an inquiry. The input method is performed by using a key input from a mobile phone, a PC, or the like, or a voice input function mounted on the wearable terminal. A search request may be made using an image stored in a mobile phone or a PC. In FIG. 1, description will be made assuming that a target image is searched from a search word. Here, the “target image” refers to an image related to a search word or a search image. The “search image” is an image taken by the user or possessed by the user for searching (extracting) a target image.

  The user inputs, for example, “entrance” and “key” as search terms for searching for a target image. In the search target image, an image keyword to be added to the image is registered. The image keyword is generated by the image analysis function of the search server, or is directly input by the user.

  In response to this, the search server extracts a target image from the search word and / or search target image and transmits it to the user. When a plurality of target images are detected, the latest search target image is usually extracted. Note that it is possible to extract the oldest search target image according to the user's setting. By doing in this way, the last state that he / she saw or the first state that he / she saw can be confirmed as “It was like this”.

  The user receives the target image and browses it, so that the user can recall the action taken by the user in the past. In the above description, the example in which the glasses with a camera are used as a wearable terminal for acquiring a photographed image has been described. However, other devices can be used as long as they can be worn near the visual region. A hat, accessories with a built-in camera, or other camera functions may be used.

  In addition, as shown in FIG. 2, it is possible to use a fixed camera (such as a monitoring camera or a fixed point observation camera) that is already installed without using a wearable terminal whose imaging range is the user's field of view. . For example, if fixed cameras for crime prevention or watching are installed at a plurality of locations in the house, images taken by these fixed cameras may be transmitted to the search server. The search method is the same as in FIG. 1. When the user wants to search for past images, the search result image is transmitted to the user terminal by keying in the search word from the user terminal. Is done.

(Configuration of image information retrieval system)
FIG. 3 is a diagram illustrating a functional configuration of a system including the image information search server according to the first embodiment of the present invention. As shown in FIG. 3, the present system includes an imaging terminal 10 having a photographing function (which may include a recording function), a user terminal 30 for searching and browsing a photographed image, the imaging terminal 10 and use. An image information search server 20 connected to the person terminal 30 via a network. Note that the imaging terminal 10 and the user terminal 30 may be integrated.

(Configuration of image information search server)
As shown in FIG. 3, the image information search server 20 includes, for example, a communication unit 21, an index information generation unit 22, an image extraction unit 23, a scene change detection unit 24, and a captured image selection unit as one configuration example. 25, a search image registration unit 26, a control unit 27, and an image storage unit 28. The index information generation unit 22, the scene change detection unit 24, the captured image selection unit 25, the search image registration unit 26, and the control unit 27 are various units for storing a search target image for search in the DB. These means may be collectively considered as image storage means in a broad sense.

  The communication unit 21 functions as a communication interface when communicating with the imaging terminal 10 and the user terminal 30. In addition, the communication unit 21 has a function of receiving an image search request including identification information related to the user and a search word from the user terminal 30 of the user. “Identification information related to the user” here refers to an imaging terminal that the user owns or owns, as well as an entrance door or a room such as an apartment house in which the user resides or a user's office. This refers to the identification information of the fixed camera, the imaging terminal possessed by the user's family and friends. Therefore, the photographing range can be further expanded. In the case of an imaging terminal possessed by another person, the user's consent is acquired in advance and browsing is permitted.

  The index information generation unit 22 has a function of generating index information for each scene of a captured image when the captured image is a moving image. For example, the index information generation unit 22 associates, as an index item, a frame number in which a contrast or motion greater than a predetermined value detected by the scene change detection unit 24 is detected, and stores the index information stored in the image storage unit 28 together with its start time. Store in DB300.

  As will be described in detail later, the image extraction unit 23 receives an image search request using a search image from the user terminal 30 and refers to the image storage unit 28 on the basis of the search word to obtain a desired (stored memory). It has the function of extracting images that can be recalled. In addition, when receiving an image search request using a search image from the user terminal 30, the image extraction unit 23 also has a function of extracting an image related to the search image from the search target image. Specifically, the image extraction unit 23 extracts an image related to (matched with) the search image based on the search image registered in advance by an image analysis technique.

  Whether to use a search word or a search image when searching for a target image depends on the user's selection, but the search word and the search image may be used together. The purpose of the search is not limited to the image, and text information (referred to as “image keyword”) that is added to the image and indicates the keyword or content of the image can be extracted.

  Note that the image extraction unit 23 may extract an image that the user wants to confirm in advance by setting and transmit the image to the user terminal before a request from the user.

  When the captured image received from the imaging terminal 10 is a moving image, the scene change detection unit 24 compares the captured image in units of frames, and detects a contrast or movement greater than a predetermined value between the current frame and the immediately preceding frame. The information is transmitted to the index information generating means 22.

  The photographed image sorting means 25 sorts out images useful for evoking the user's memory among the photographed images from the imaging terminal 10 (images stored in the photographed image DB 100), and is a search target image for later retrieval. Is stored in the search target image DB 200. At this time, the above-described image keyword is added to the search image in order to facilitate the search for the image. Of course, an image keyword may also be added to the search target image. Further, the captured image sorting unit 25 determines the similarity with the search image in the search target image DB 200 in which the captured image has already been registered, and images whose similarity is lower than a predetermined threshold (substantially overlapping images). Is determined to be an image useful for evoke the user's memory, and is stored in the search target image DB 200.

  In addition, the photographed image sorting unit 25 does not store in the search target image DB 22 images that are clearly determined not to be useful for revoking the user's memory among the photographed images. For example, an extremely unclear image, an image that is judged to be a shooting mistake (an image that only shows the ground, the sky, or a wall, an image that was taken at a time other than the user's life time such as midnight, etc.) is stored. Do not. The above image selection may be automatically performed using a known image analysis technique. Further, when the picked-up image selecting means 25 cannot automatically select, the user may be asked to determine whether or not to store.

  Further, the photographed image sorting means 25 may determine whether the image is an image that the user is gazing at the time of photographing (gazing image) or an image that is unconsciously photographed (unconscious image). At this time, the captured image sorting unit 25 may determine that the subject existing near the center of the captured image is a gaze image that the user is gazing at, and may determine other subjects as unconscious images. Further, in the case where the captured image is a person, for example, all the persons who are facing the front may be determined as gaze images, and the other images may be determined as unconscious images. Since an image keyword is not expected to be added by the user to the unconscious image, the image keyword may be automatically generated and added by analyzing what the subject is by image analysis.

  It is also possible to leave the user with a decision as to whether the image is a gaze image or an unconscious image. In that case, the captured image sorting means 25 transmits an inquiry as to whether it is a gaze image or an unconscious image based on the captured image to the user terminal 30. When the user replies that the image is a gaze image, the image is registered in the search target image DB 200 as a search image. On the other hand, when the user replies that it is an unconscious image, it is separately registered in the search target image DB 200 as an unconsciously photographed image.

  Thereafter, if the captured image is an overlapping image (an image having a high degree of similarity) that is substantially the same as the unconscious image, the captured image sorting means 25 determines that the image is not useful for revoking the user's memory, and Duplicate images are deleted. The reason why the unconscious image is stored in the search target image DB 200 in this manner is that the unconscious image remains in the user's subconscious, and the image that the user wants to search for is the gaze image. This is not always the case. In some cases, you may want to look for things that remain in your subconscious mind (something that was somewhere in your sight). For example, if a person you want to know is one of five people in a meeting on a certain day and you did not explain it, or a shop where you wanted to know someday This is the case. Of course, registering all unconscious images causes a shortage of memory in the image storage means 28, and therefore, duplicate images are deleted.

  Further, the photographed image sorting unit 25 erases at least a part of the photographed image or the search target image in accordance with an erasure request from the user terminal 30 or a user search result. In addition, the captured image sorting unit 25 stores, as a search target image, an image that has no screen change among captured images. That is, only the first frame is stored when there is no frame change in the moving image.

  Further, the photographed image sorting means 25 stores only the first still image when the same still image continues. This is because it is meaningless to memorize the same image. Note that the user himself / herself may select a photographed image or a search target image from the user terminal 30, or may be automatically deleted after a predetermined period (for example, three years) depending on the setting. The photographed image sorting means 25 makes it possible to effectively use the memory resources of the image storage means 28 having a finite storage capacity.

  Also, the photographed image sorting means 25 rechecks the images once stored under the instruction of the user or periodically under the control of the control means 27, and images that are considered unnecessary are retrieved from the search target image DB 200 as needed. Delete (this is called “image inventory”). At this time, when registering the image in the search target image DB 200, the photographed image sorting unit 25 uses an object that is easy to forget (for example, an entrance door and a key, a gas main valve, etc.) as an element to be noticed. In principle, the image to be included is controlled to store the entire image. On the other hand, for an image that does not include the element of interest, only the difference from the previous image may be saved to effectively use the memory resources of the image storage means 28. This process will be described later in detail.

  In addition, for images including the same elements, the degree of similarity is determined again every time a predetermined time elapses, and the images with high similarity are preferentially reduced. As a result, for example, when 3 days have passed, the image is reduced by 50%, and after 1 week, the image is further reduced by 50%, further after 1 month, further 50%, and reduced by 50% every year. Save the image for 10 years. Note that an image for which processing in the photographed image DB 100 has been completed is unnecessary and is deleted. As a result, the memory resources of the image storage means 28 can be further effectively used.

  The search image registration unit 26 has a function of registering a search image related to a search word based on the search word and the search image input or selected from the user terminal 30. Here, the “search word” is a keyword input by the user when searching for an image, and is different from the image keyword added to the image. The search term may be the name of the search object, or the location and time (position information and time information) of the search object. Also, an image designated by the user is registered as a search image together with the image keyword. The association between the image keyword and the search image is performed by automatically analyzing the search image or by performing an operation in which the user associates and registers the search image and the search word in advance. The search image registration means 26 can define a composite keyword for registering a plurality of search images for one keyword. Details of this will be described later.

  The control unit 27 stores the captured image received from the imaging terminal 10 in the image storage unit 28 as an image for preventing the user from forgetting, and upon receiving an image search request including a search word from the user terminal 30, the image storage unit. 28, the above-described functional blocks are controlled in order to realize a function as the image information search server 20 that extracts a captured image related to the search term and transmits the extracted captured image to the user terminal 30. Take a role. The image storage means 28 includes a captured image DB 100, a search target image DB 200, and an index information storage DB 300.

  The image storage means 28 stores the image selected by the photographed image selection means 25 as a search target image that is a search target from the user together with identification information related to the user.

  The images stored in the search target image DB 200 may be organized as part of image inventory. For example, if a person image and a non-person image are separated and the image is a non-person image, only the latest image among the images related to the same keyword may be left. Taking “forgetting to close the entrance key” as an example, there is a low possibility that an image other than the latest entrance door will be used. However, if the image is a person image, an image at that time including the oldest image among the images related to the same keyword may be left. This is because the face of a person may change, and it is not sufficient to leave the latest image. The image organization setting may be manually set by the user, or may be automatically organized according to the user settings in advance.

  As an example of the data structure is shown in FIG. 4A, in the captured image DB 100, at least “user ID” numbered for each user and “photographed” numbered for each captured image. “Information No”, “captured image”, “capturing date / time”, and “location information” of the location where the image was captured are registered.

  According to FIG. 4A, the photographed image of the photographing information No “1” of the user ID “00100” includes the photographed entrance image and the photographing date and time “2015/05/05 13:05:30”. Yes, position information “35 ° 30′7.16 ″, 139 ° 8′8.12 ″” is registered.

  As an example of the data structure is shown in FIG. 4B, the search target image DB 200 includes at least “user ID” numbered for each user, automatically recognized by the search server, or by the user. The registered “keyword”, “search image” associated with the keyword, and “registration date” are registered as data items.

  According to FIG. 4B, the image keyword “doorknob”, “entrance”, and “key” are registered in the first image of the user ID “00100”, and the search associated with the image keyword is performed. The work image is registered in “2015/05/06”. In addition, image keywords such as “car” and “automobile” are registered in the fourth image in FIG. If you also register the number of the license plate, it will help you to search for an image of your car at the time of image search.

  As shown in FIG. 5 as an example of the data structure, the index information storage DB 300 includes at least “user ID” numbered for each user and “shooting information No.” numbered for each captured image. ”,“ Index information ”generated for each scene of the shooting information,“ scene start date and time ”, and“ position information ”acquired from the GPS or the like at the start of scene shooting are registered. Although illustration is omitted, the end time of the scene, the length of the scene, and the like may be registered.

  According to FIG. 5, shooting information No “1” is registered for the user ID “00100”. Index information A to D is registered in the shooting information No. “1”. For example, the index item “A” includes the scene start date and time “2015/05/05 13:05:30” and the position information “35”. A photographed image of “30 ° 7.16 ″, 139 ° 8′8.12 ″” is registered.

  The above-described functional configuration of the present system is merely an example, and one functional block (database and function processing unit) may be divided, or a plurality of functional blocks may be configured as one functional block. Each function processing unit reads a computer program stored in a storage device such as a ROM (Read Only Memory), flash memory, SSD (Solid State Drive), or hard disk by a CPU (Central Processing Unit) built in the device. This is realized by a computer program executed by the CPU. That is, in each function processing unit, this computer program reads and writes necessary data such as a table from a database (DB; Data Base) stored in a storage device or a storage area on a memory, and may be related in some cases. This is realized by controlling hardware (for example, an input / output device, a display device, and a communication interface device). Further, the database (DB) in the embodiment of the present invention may be a commercial database, but it simply means a collection of tables and files, and the internal structure of the database itself is not questioned.

  FIG. 6 is a flowchart illustrating the processing operation of the image information search server that generates index information based on the captured image received from the imaging terminal 10. Note that the processing described below does not necessarily have to be performed in the order shown in this flowchart, and the processing order may be changed as long as the relationship between input data and output data of each processing block is not impaired.

  As shown in FIG. 6, when the control unit 27 of the image information search server 20 according to the present embodiment receives a captured image from the imaging terminal 10 (step S11), the captured image is captured by the image storage unit 28 in time series order. It memorize | stores in image DB100 (step S12). Details of this processing will be further described with reference to FIG.

  When a predetermined amount of photographed images are stored in the photographed image DB 100, the photographed image sorting means 25 sorts out images that are useful for invoking the user's memory among the photographed images photographed by the photographing means from the imaging terminal 10, The search target image is stored in the search target image DB 200 as a search target image (step S13). Note that step S13 may be performed after storing index information of step S17 described later.

  Next, the control means 27 determines whether the stored captured image is a still image or a moving image (step S14). If the stored captured image is a moving image (step S14 “moving image”), the scene change detection unit 24 compares the captured images in units of frames based on the time-series captured images stored in the search target image DB 200. Specifically, the current frame in which a contrast change or a motion vector change (scene change) greater than or equal to a predetermined value is detected between the current frame and the immediately preceding frame, and a contrast or motion greater than the predetermined value has occurred. Is extracted (step S15). Note that still images and moving images are not particularly distinguished, and still images may be considered as moving images with extremely long frame shooting intervals. For example, in the case of a normal moving image, the shooting speed is 30 frames / second, but it may be considered that a still image is one frame that is about several frames to several hours.

  Subsequently, the index information generating unit 22 generates index information by associating, as an index item, the number of a frame in which a contrast or movement greater than a predetermined value detected by the scene change detecting unit 24 has occurred (step S16). A stamp is attached and stored in the index information storage DB 300 of the image storage means 28 (step S17). On the other hand, when the captured image stored in step S12 is a still image (step S14 “still image”), the control unit 27 ends the process without sending an instruction to generate index information.

  FIG. 7 is a flowchart showing details when the captured image is stored in step S12 of the previous figure. First, in step S121, the control unit 27 analyzes the captured image transmitted from the imaging terminal 10. The image analysis here is to analyze what the object is photographed in the image, and uses a known technique (for example, JP-A-2015-056787). When the image analysis is completed, it is determined in step S122 whether or not the captured image includes an element of interest. The element of interest is an object that may be forgotten by the user. Specifically, when a user forgets an object, it is a search image registered in advance to search for the image, such as a front door, a gas faucet, a bath faucet, a room window, an air conditioner and a stove switch. , An image of a safe, own car, acquaintance's face, etc. (hereinafter also referred to as “image element”). For example, in order to remember whether the user has locked the entrance door of the home, if the image of the entrance door of the home is registered in advance, whether the image of the entrance door of the home is included in the image taken after that Can be determined.

  If it is determined in step S122 that the captured image includes the image element of interest, the entire received captured image (however, if the same image continues, only the first image of the scene may be used). ) Is stored in the photographed image DB 100 (step S124). If it is determined that the captured image does not include the image element of interest, it is further determined in step S123 whether there is a change from the previously stored image (referred to as the background or the previous image). If there is a change, only the difference image from the previous image is stored in step S125. For example, if the photographed image is a video showing the street in front of the house, the previous image (background) is “a building in front of the house”, and if a cat passes there, the image of the cat Is newly stored as a difference.

  Next, in step S126, it is determined whether or not the image once determined to be a difference is integrated with the background for a predetermined time or more (for example, about several weeks to several months or more). If the image is integrated with the background, the entire image is stored as a new previous image (background) including the image determined to be the difference (step S128). Also, for example, if someone stands a billboard in front of a “building in front of the house”, it is stored as a difference once, but the billboard does not move after that, so it is stored as a new previous image (background).

  Returning to step S123, if the current captured image has not changed from the previous image, the process proceeds to step S127, where it is determined whether a predetermined time has elapsed since the captured image was started (date and time when the same landscape was first captured). . Here, “no change” includes not only the state in which the entire image has not changed at all, but also the state in which there is no substantial change. For example, analog clock hands, digital clock displays, shadows that change according to the movement of the sun, etc. may not be regarded as changes described above. If the predetermined time has not elapsed, nothing is done, but if the specified time has elapsed, the entire captured image at that time is stored as the previous image (background) (step S128). Even if it is determined that there is no change from the previous image, it is preferable to update the previous image to the current image if a certain amount of time has passed.

  The above steps are repeated until the processing is completed for the entire photographed image received this time (step S129). By performing such processing before storing the image in the captured image DB 100, the image data stored in the DB can be significantly reduced as compared with the case of storing the image received from the imaging terminal 10 as it is. The above process may be performed as a pre-process of the step of selecting a captured image and storing it as a search target image (step S13 in the previous figure), or may be performed in step S13.

  Next, the processing operation of the image information search server for registering the search image will be described with reference to the flowchart of FIG. As shown in FIG. 8, when the image information search server 20 receives a search image registration request from the user terminal 30 (step S <b> 21), the search image registration means 26 sends a search request to the user terminal 30. Screen information that prompts the transmission of image data is transmitted, and the search image input by the user terminal 30 is registered (step S22).

  Subsequently, when the image information search server 20 receives an image keyword registration request from the user terminal 30 (step S23), the search image registration unit 26 searches the image keyword of the search image corresponding to the user terminal 30. Is transmitted, and the image keyword input by the user terminal 30 is registered (step S24). As described above, the image keyword is a keyword added to the image to make it easy to search for the image, and is different from the search word that the user inputs at the time of the search. Note that the processing in step S23 and step S24 may extract an image keyword related to the search image received from the user terminal 30 by using an automatic recognition function by the image information search server 20. Then, the search image registration unit 26 associates the search image with the image keyword and stores them in the search target image DB 200 (step S25).

  Next, the image search processing operation of the image information search server according to the embodiment of the present invention will be described with reference to FIG. As shown in FIG. 9, when the control means 27 of the image information search server 20 according to the present embodiment receives an image search request from the user terminal 30 (step S31 “YES”), the search condition is addressed to the user terminal 30. Information that prompts the user to input (for example, location, date, and other keywords) is transmitted (step S32).

  When “other keywords” is transmitted as a search condition from the user terminal 30 (step S33 “other keywords”), the image extraction means 23 is received with reference to the index information storage DB 300 of the image storage means 28. Then, search images related to other keywords are extracted (step S34). Then, the image extraction unit 23 performs matching processing (collation operation) between the extracted search image and the search target image, and extracts a target image from the search target image (step S35).

  In addition, when both a still image and a moving image are stored as search target images, the image extraction unit 23 may transmit information prompting the user terminal 30 to acquire which image. In this way, for example, if you want to confirm with an image whether you have closed the gas main valve, if it is a still image, you must browse multiple images, whereas in the case of a movie By browsing one index information (with sound), it is possible to confirm the gas main stopper. Moreover, you may make it acquire both as needed.

  On the other hand, when “place” is transmitted as a search condition from the user terminal 30 (step S33 “place”), the image extraction unit 23 calculates position information of the received place (step S36), and the image storage unit 28. Referring to the index information storage DB 300, the specified position information is compared with the position information registered in the index information storage DB 300, and an image located in the vicinity is extracted (step S37).

  On the other hand, when “date and time” is transmitted as a search condition from the user terminal 30 (step S33 “date and time”), the image extraction unit 23 refers to the index information storage DB 300 of the image storage unit 28 and stores it in the index information storage DB 300. The image including the registered time zone is compared with the corresponding date, and an image near the corresponding date is extracted (step S38). In this way, you can search by "location" and "date and time" alone, so even if you don't come up with a search term for your actions ("other search terms"), if you remember the location or date and time, You can search for images. For example, in the case of voice input, it is only necessary to input “Display an image from today 9:00 to 10:00!”.

  Then, the image extraction unit 23 transmits an image corresponding to the search condition of the search request from the user terminal 30 to the user terminal 30 that requested it (step S39).

  By the image search request process described above, for example, as shown in FIG. 10, a search condition selection field is displayed on the user terminal 30, and the user selects the type of selection condition. As an example, when “other keywords” is selected as the selection condition, a column for inputting keywords is displayed. When the user inputs a keyword and presses the search start button, the number of hits of the image corresponding to the search condition from the registered search target images is displayed on the screen, and the latest image is displayed. Further, by displaying position information (such as a place name) on the screen, it becomes clear where the image was taken. For example, taking FIG. 10 as an example, it can be confirmed that this is the entrance of the house. In addition, by setting (pressing the setting button) from the user terminal 30, for example, by adding position information, date and time to the search condition, not only the latest image but also an image hit in the corresponding place and time zone can be selected. It is also possible to search.

  It is also possible to define a composite keyword for registering a plurality of search images for one keyword. Specifically, for example, for the keyword “going out”, the items to be checked before going out are “forgetting to close the entrance key”, “forgetting to close the gas main valve”, “forgetting to turn off the lighting in the room” ", Forgetting to turn off the air conditioner", etc., and register it as a compound keyword. By doing so, the user terminal 30 displays all the associated items as shown in FIG. 11 by inputting the search term “outing” into the image information search server 20 and performing a search. Is done.

  Next, the processing operation of the image information search server according to the present embodiment when an image search request is made with images held by the user terminal will be described with reference to the flowchart shown in FIG. As shown in FIG. 12, when the control means 27 receives an image search request including an image from the user terminal 30 (step S41 “YES”), the image extraction means 23 searches the search image associated with the image. Is matched with the index information registered in the index information storage DB 300 (step S42). Note that the matching process between images is known, for example, from Japanese Patent Application Laid-Open No. 2004-341940, and detailed description thereof is omitted.

  If the search image matches the index information as a result of the matching process (step S42 “match”), the image extraction unit 23 extracts the index information including the target image from the search target image, and requests the extracted image. Is transmitted to the user terminal 30 where the error occurred (step S47).

  By the above-described image search request processing, for example, as shown in FIG. 13, a search condition selection field is displayed on the user terminal 30, and the user inputs the selection condition. By selecting “person image” as the selection condition and pressing the reference button, a list of person images stored in the user terminal 30 is displayed. When the user selects the corresponding image and presses the search start button, the number of hits of the image that matches the search condition from the registered search target images is displayed on the screen, and the latest image is displayed. Is done. As in the case of FIG. 10, for example, by adding position information and date / time to the search condition by setting from the user terminal 30 (pressing the setting button), not only the latest image but also the corresponding location. It is also possible to search for images that hit in the time zone.

  On the other hand, when the search image and the index information do not match (step S42 “mismatch”), the image extraction unit 23 acquires a search term attached to the image search request from the user terminal 30 (step S43). Then, the search target image DB 200 is referred to using the acquired search word as a condition to determine whether an image keyword corresponding to the search word is registered (step S44). When the corresponding image keyword is not registered (step S44 “NO”), the image extracting means 23 sends an error notification indicating that the corresponding search word does not exist in the image information search server 20 to the user terminal 30. Transmit (step S46).

  On the other hand, if the corresponding search word is registered (step S44 “YES”), the image extraction means 23 searches the search image linked with the search word and the index information registered in the index information storage DB 300. The index information including the target image is extracted from the search target image, and the extracted image is transmitted to the requested user terminal 30 (step S47).

(Second Embodiment)
FIG. 14 is a diagram showing a functional configuration of a user terminal according to the second embodiment of the present invention. In the second embodiment, the user terminal 30 has the function of the image information search server 20 according to the first embodiment. Hereinafter, descriptions of various functions of the user terminal 30 that are the same as those in the first embodiment will be omitted.

  The captured image memory 100 ′, the search target image memory 200 ′, and the index information storage memory 300 ′ of the user terminal 30 according to the second embodiment are image storage means of the image information search server 20 according to the first embodiment. 28 has the same functions as the photographed image DB 100, the search target image DB 200, and the index information storage DB 300. Therefore, in the second embodiment, data stored in each DB is stored in each memory of the user terminal 30. Since the memory capacity of each memory of the user terminal 30 is limited compared to each DB of the server, it is more important to select, organize, and reduce images stored in the memory.

  The communication unit 31 of the user terminal 30 functions as a communication interface when communicating with the imaging terminal 10.

  When acquiring an image search request using the search image from the user, the image extraction unit 33 extracts an image related to the search image from the search target image.

  The photographed image sorting means 35 sorts out the images to be stored according to the erasure request from the user or the search results of the user, and erases at least a part of the photographed image or the search target image.

  The control unit 37 stores the captured image received from the imaging terminal 10 in the image storage unit 38 as an image for preventing the user from forgetting, and when the image search request including the search word is acquired from the user, the image storage unit 38 is stored. In order to realize a function as the image information search server 20 that extracts a captured image related to the search term and transmits the extracted captured image to the user terminal 30, the function of controlling each functional block described above is provided. Bear.

  The user terminal 30 shown in FIG. 14 has been described as a substitute for all the functions of the image information search server 20 shown in FIG. 3, but only a part can be substituted. Even in this case, the processing load of the image information search server 20 can be reduced.

  Note that, as a modification of the second embodiment, a form in which the imaging terminal 10 is integrated (integrated) with the user terminal 30 is also possible. However, in this case, the fixed camera is not supported as an imaging terminal.

(Effect of embodiment)
As described above, according to the image information search server 20 according to the present embodiment, images that are useful for invoking the memory of the user are selected from the captured images captured from the imaging terminal, and the search from the user is performed. An image search request including a search word is received from the user terminal of the user, and stored as a search target image. Then, an image related to the search term is extracted from the search target image, and the extracted image is transmitted to the user terminal, so that the memory of the actions taken by the user in the past regardless of whether there is prior registration or not is stored in the image. It can be recalled. That is, the image information search server 20 can be used as a “self-dedicated action memory”, that is, the user can store his / her actions and retrieve them at any time.

  Further, when an image captured by the user is registered as a search image and an image search request using the search image is received from the user terminal, an image related to the search image is extracted from the search target image. That is, an image can be retrieved from the image.

  If the image element of interest is included in the captured image, the entire captured image is stored. If the image element of interest is not included in the captured image, only the difference image from the previously stored image (background) is stored. Therefore, it is possible to avoid the accumulation of unnecessary images and to effectively use memory resources.

  Further, by extracting a target image or an image keyword added to the image based on the search word and / or the search image, it is possible to perform a search by combining the search word and the search image. You can use different search methods. Conversely, keywords of an image related to the image can be extracted from the image. For example, when the name of a person who has recently met cannot be remembered, if there is an image of the person, it is possible to search the name and attributes of the person from the image.

  Further, by erasing a part of the photographed image or the search target image in accordance with the erasure request from the user terminal or the user's search results, the memory resource of the image storage unit is effectively used.

  In addition, by storing, as a search target image, images that have no screen change among captured images, the memory resources of the image storage unit are effectively used.

  Note that the present invention can be understood not only as an image information search server but also as an invention of an image information search method. Moreover, it can also be regarded as the invention of the combination of a user terminal and an imaging terminal, or the user terminal integrated with an imaging terminal, without requiring an image information search server. In this case, since communication with the server is not required, the mobility and operability can be further utilized by making the user terminal a wearable terminal.

  As mentioned above, although this invention was demonstrated using embodiment, it cannot be overemphasized that the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be added to the above embodiment. Further, it is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

DESCRIPTION OF SYMBOLS 10 Imaging terminal 20 Image information search server 21 Communication means 22 Index information generation means 23 Image extraction means 24 Scene change detection means 25 Photographed image selection means 26 Search image registration means 27 Control means 28 Image storage means 30 User terminal 100 Photographed image DB
200 Search target image DB
300 Index information storage DB
100 'Captured image memory 200' Search target image memory 300 'Index information storage memory

Claims (11)

An image information search server connected via a network to an imaging terminal having imaging means,
The image information search server
Photographed image sorting means for sorting out images useful for evoking a user's memory among the photographed images photographed by the photographing means from the imaging terminal;
Image storage means for storing the selected image as a search target image that is a search target from the user together with identification information related to the user;
Means for receiving, from the user terminal of the user, an image search request including identification information and a search word related to the user;
Referring to the image storage means, image extracting means for extracting an image related to the search word from the search target image by identification information related to the user;
Means for transmitting the image extracted by the image extraction means to the user terminal;
An image information search server comprising: A search image registration means for registering the image taken by the user as a search image;
The image extracting means includes
2. The image information search server according to claim 1, wherein when an image search request using the search image is received from the user terminal, an image related to the search image is extracted from the search target image. .   The image storage means stores the entirety of the captured image when the image element of interest is included in the captured image, and stores the previously stored image when the image element of interest is not included in the captured image. The image information search server according to claim 1, wherein the difference image is stored. The image extracting means includes
The image information search server according to claim 2, wherein a target image or an image keyword added to the image is extracted based on the search word and / or the search image transmitted from the user terminal. .   5. The image information search server according to claim 2, further comprising means for causing the registered search image to be selected from the user terminal when the image search request is made. The imaging terminal is
The image information search server according to any one of claims 1 to 5, wherein the image information search server is a wearable terminal equipped with glasses with a camera, a hat with a camera, accessories with a built-in camera, and other camera functions.   The photographed image selection means for erasing at least a part of the photographed image or the search target image according to an erasure request from the user terminal or a search result of the user. The image information search server according to claim 1. The photographed image sorting means includes
The image information search server according to any one of claims 1 to 7, wherein an image having no screen change among the photographed images is excluded and stored as the search target image. An image information search method in which an imaging terminal having a photographing means and an image information search server are connected via a network,
The image information search server is
Receiving a photographed image photographed by the photographing means from the imaging terminal;
Selecting an image useful for evoking a user's memory among the captured images;
Storing the selected image as a search target image to be searched from the user together with identification information related to the user;
Receiving, from a user terminal, a search request based on identification information and search terms related to the user;
When an image search request including identification information related to the user and the search word is received from the user terminal, the stored image for preventing forgetting of the user is searched, and the identification related to the user Extracting information and an image related to the search term from the search target image, and transmitting the extracted image to the user terminal;
A method for retrieving image information, comprising: A user terminal connected via a network to an imaging terminal having imaging means,
Photographed image sorting means for sorting out images useful for evoking a user's memory among the photographed images photographed by the photographing means from the imaging terminal;
Image storage means for storing the selected image as a search target image to be searched from the user together with identification information related to the user;
Means for acquiring an image search request including identification information and a search term related to the user;
Referring to the image storage means, image extracting means for extracting an image related to the search word from the search target image by identification information related to the user;
Means for displaying the image extracted by the image extraction means;
A user terminal comprising: A user terminal having photographing means,
Photographed image sorting means for sorting out images useful for evoking the user's memory among the photographed images photographed by the photographing means;
Image storage means for storing the selected image as a search target image to be searched from the user together with identification information related to the user;
Means for acquiring an image search request including identification information and a search term related to the user;
Referring to the image storage means, image extracting means for extracting an image related to the search word from the search target image by identification information related to the user;
Means for displaying the image extracted by the image extraction means;
A user terminal comprising:

Images