WO2020188924A1 - 情報処理装置、検索方法、及びプログラムが格納された非一時的なコンピュータ可読媒体 - Google Patents

情報処理装置、検索方法、及びプログラムが格納された非一時的なコンピュータ可読媒体 Download PDF

Info

Publication number
WO2020188924A1
WO2020188924A1 PCT/JP2019/049299 JP2019049299W WO2020188924A1 WO 2020188924 A1 WO2020188924 A1 WO 2020188924A1 JP 2019049299 W JP2019049299 W JP 2019049299W WO 2020188924 A1 WO2020188924 A1 WO 2020188924A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
search condition
search
information processing
images
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2019/049299
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
テイテイ トウ
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
NEC Corp
Original Assignee
NEC Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by NEC Corp filed Critical NEC Corp
Priority to US17/436,299 priority Critical patent/US20220179899A1/en
Priority to JP2021506166A priority patent/JP7238963B2/ja
Publication of WO2020188924A1 publication Critical patent/WO2020188924A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/53—Querying
    • G06F16/532—Query formulation, e.g. graphical querying
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/53—Querying
    • G06F16/538—Presentation of query results
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/54—Browsing; Visualisation therefor
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/94—Hardware or software architectures specially adapted for image or video understanding
    • G06V10/945—User interactive design; Environments; Toolboxes

Definitions

  • the present invention relates to an information processing device, a search method, and a program.
  • Patent Document 1 discloses a technique for generating a search condition from a search key image and searching for an image in order to reduce the burden of inputting search conditions such as features and imaging conditions by the user.
  • a plurality of different search conditions are generated based on the feature amount or the imaging condition acquired from the search key image.
  • the technique searches for images that match or resemble each search condition and presents the search results to the user.
  • the user selects an image from the presented search results and sets the selected image as a new search key image. In this way, the search is repeated so as to find an image that satisfies the features and imaging conditions intended by the user.
  • Patent Document 2 discloses a technique for searching for a color specified by a user or a portion having a color and a shape in an image of a subject displayed on a monitor screen of an electronic device. Then, in this technique, the search result is displayed by displaying only the part corresponding to the specified condition or by displaying the part other than the corresponding part in a semi-transparent state.
  • Non-Patent Document 1 discloses a technique for generating a realistic image that matches a text from a user by using a machine learning technique. This technology aims to produce images that are faithful to the text.
  • Patent Document 1 when searching, the user only selects a search key image, and the user does not input information about a specific search target into the device. Therefore, it is not possible to determine the search conditions that reflect the user's intention in detail.
  • Patent Document 2 when searching for a portion matching the color or shape specified by the user, the search intention of the user is not confirmed in more detail. Therefore, with this technique, it is not possible to determine search conditions that reflect the user's intention in detail.
  • Non-Patent Document 1 is a technique for generating a high-quality image that meets the user's conditions, and it is not possible to determine search conditions that reflect the user's intention in detail.
  • one of the purposes to be achieved by the embodiments disclosed in the present specification is to provide an information processing device, a search method, and a program capable of determining search conditions that reflect the user's intention in detail. There is.
  • the information processing device is Search condition acquisition means to acquire the entered search conditions, An image displaying one or more types of images representing an image of an object specified by the search condition acquired by the search condition acquisition means and representing a variation of the object or a variation of an aspect of the object specified by the search condition.
  • Display means and A selection receiving means for receiving an instruction to select one or more types of images displayed by the image displaying means, and a selection receiving means. It has a search condition determining means for determining a search condition based on the image selected according to an instruction received by the selection receiving means.
  • the search condition is determined based on the image selected according to the received instruction.
  • the program according to the third aspect is Search condition acquisition step to acquire the entered search condition, and An image display step of displaying one or more types of images representing the variation of the object or the variation of the aspect of the object specified by the search condition, which is an image of the object specified by the acquired search condition.
  • a selection acceptance step that accepts an instruction to select one or more types of the displayed images, and The computer is made to execute the search condition determination step of determining the search condition based on the image selected according to the received instruction.
  • FIG. 1 is a block diagram showing an example of the configuration of the information processing apparatus 1 according to the outline of the embodiment.
  • the information processing device 1 includes a search condition acquisition unit 2, an image display unit 3, a selection reception unit 4, and a search condition determination unit 5.
  • the search condition acquisition unit 2 acquires the search conditions input to the information processing device 1.
  • the search condition acquired by the search condition acquisition unit 2 is, for example, a search condition input by the user. This search condition at least specifies an object to be searched. Further, this search condition may specify not only the object to be searched but also the mode of the object (for example, the color of the object, the position of the object, the orientation of the object, the movement of the object, etc.).
  • the information processing device 1 does not use the search condition acquired by the search condition acquisition unit 2 as it is in the search process, but determines the search condition that reflects the user's intention in more detail than the search condition by the search condition determination unit 5. To do.
  • the image display unit 3 is an image of an object specified by the search condition acquired by the search condition acquisition unit 2, and is one or more types representing a variation of the object or a variation of the mode of the object specified by the search condition. Display the image of. For example, when the object to be searched specified by the search condition acquired by the search condition acquisition unit 2 is a "car", the image display unit 3 displays one or more types of images representing variations of the car. More specifically, for example, the image display unit 3 displays an image of an ordinary car, an image of a small car, an image of a bus, and the like. In the following description, an image representing a variation may be simply referred to as a variation image.
  • the selection reception unit 4 receives an instruction to select one or more types of images among the images displayed by the image display unit 3.
  • the user who inputs the search condition selects an image that reflects his / her intention from the displayed images. This selection is accepted by the selection reception unit 4.
  • the search condition determination unit 5 determines the search condition based on the image selected according to the instruction received by the selection reception unit 4. That is, the search condition determination unit 5 sets the search condition corresponding to the content of the selected image as the search condition used in the search process.
  • the information processing device 1 displays the variation image and accepts the user's selection for the variation image. Then, the search condition is determined according to the selection. Therefore, it is possible to determine a search condition that reflects the user's intention in detail.
  • FIG. 2 is a block diagram showing an example of the configuration of the information processing apparatus 10 according to the embodiment.
  • the information processing apparatus 10 includes a thesaurus storage unit 11, a search condition acquisition unit 12, an image generation unit 13, an image display unit 14, a control unit 15, a search condition determination unit 16, and the like. It has an image search unit 17.
  • the thesaurus storage unit 11 stores information in which keywords that can be used for a search are systematically summarized in advance. In the following description, this information will be referred to as thesaurus information.
  • the thesaurus information is, for example, tree-structured information showing the relationship between a keyword that is a superordinate concept and a keyword of the subordinate concept.
  • the thesaurus storage unit 11 stores the thesaurus information regarding the object and the thesaurus information regarding the mode of the object.
  • FIG. 3 is a schematic diagram showing an example of thesaurus information regarding an object.
  • 4 to 7 are schematic views showing an example of thesaurus information regarding the mode of the object, respectively.
  • the thesaurus information shown in FIGS. 3 to 7 is hierarchically and repeatedly configured to associate keywords that classify keywords of higher-level concepts. For example, in the example shown in FIG. 3, the concept (keyword) of "object” is associated with the concept (keyword) of "person” and “other”. Furthermore, regarding "person”, the concepts (keywords) of "male”, “female”, and “unknown” are associated.
  • FIG. 4 is a schematic diagram showing an example of thesaurus information regarding the color of an object.
  • Known color classifications may be used in the thesaurus information about color.
  • FIG. 4 shows a part of a classification system of 147 colors, which is an "extended basic color" used in HTML.
  • FIG. 5 is a schematic diagram showing an example of thesaurus information regarding the position of an object.
  • the information about the "next" position is classified into “left” next and “right” next. Further, “left” is further classified into “upper left” and “lower left”, and “right” is classified into “upper right” and “lower right”.
  • FIG. 6 is a schematic diagram showing an example of thesaurus information regarding the orientation of the object.
  • the information about the position of "front” is “left front” (a state in which the left side is slightly visible instead of the front) and “right front” (front). However, it is categorized as (a state in which the right side can be seen to some extent instead of directly in front).
  • FIG. 7 is a schematic diagram showing an example of thesaurus information regarding the movement of an object.
  • the information about the motion of "standing” is classified into the motion of "standing still", the motion of "moving the head", and the motion of "moving the arm”. ..
  • the information about the motion of "moving the head” is classified into the motion of "moving the head left and right” and the motion of "moving the head up and down”.
  • the particle size of classification in the thesaurus information and the depth of layering may be decided arbitrarily.
  • the thesaurus information may be created by the designer or automatically based on an existing knowledge base or algorithm.
  • the search condition acquisition unit 12 corresponds to the search condition acquisition unit 2 in FIG.
  • the search condition acquisition unit 12 acquires the search condition input by the user.
  • the user specifies an object (that is, a subject) depicted in the image to be searched as a search condition. That is, the search condition acquired by the search condition acquisition unit 12 includes the designation of the object to be searched. Further, the search condition acquired by the search condition acquisition unit 12 may include the designation of the mode of the object described in the image to be searched. That is, the search condition acquisition unit 12 acquires the condition for the subject of the image to be searched as the search condition.
  • the search condition acquisition unit 12 may acquire the text input by the user to the information processing device 10 as the search condition, or may acquire the search condition specified by the input other than the text.
  • the search condition may be acquired based on the voice data input to the information processing device 10.
  • the search condition acquisition unit 12 acquires the search condition by converting the voice data into text by using a known voice analysis technique for the voice data.
  • the user may also select options such as a predetermined object or an icon representing a predetermined mode.
  • the search condition acquisition unit 12 acquires the search condition corresponding to the selected option.
  • the search condition acquisition unit 12 may present the text "person" as one of the options, and when this option is selected by the user, acquire "person” as the search condition.
  • the search condition acquisition unit 12 may present an illustration figure of a person as one of the options, and when this option is selected by the user, may acquire "person” as the search condition.
  • the search condition acquisition unit 12 analyzes the text and extracts the information of the search condition by using a known text analysis technique such as syntax analysis and morphological analysis.
  • a known text analysis technique such as syntax analysis and morphological analysis.
  • known words are stored in a dictionary in advance, and the text is divided into appropriate word strings by referring to the dictionary.
  • part of speech types of words such as nouns and verbs
  • readings to words in a dictionary various information can be added to words.
  • the search condition acquisition unit 12 acquires the search condition by extracting the words appearing in the dictionary from the input text.
  • This synonym list is data indicating words having the same meaning as the keywords (words) defined in the thesaurus information.
  • the search condition acquisition unit 12 can acquire not only the word defined in the thesaurus information but also the word of the synonym as the search condition.
  • the image generation unit 13 and the image display unit 14 correspond to the image display unit 3 of FIG. That is, the image generation unit 13 and the image display unit 14 may be collectively referred to as an image display unit.
  • the image display unit 14 presents the image to the user by displaying the image generated by the image generation unit 13 on the display.
  • the image generation unit 13 generates an image representing the search condition according to the search condition acquired by the search condition acquisition unit 12.
  • the image generation unit 13 generates a variation image of the object specified by the search condition acquired by the search condition acquisition unit 12 or a variation image of the mode specified by the search condition acquired by the search condition acquisition unit 12.
  • the image generation unit 13 generates a variation image to be displayed as follows.
  • the image generation unit 13 specifies a keyword corresponding to the search condition acquired by the search condition acquisition unit 12 in the thesaurus information. That is, the image generation unit 13 specifies which keyword defined in the thesaurus information corresponds to the object specified in the search condition. In addition, the image generation unit 13 specifies which keyword defined in the thesaurus information corresponds to the mode of the object specified in the search condition. Then, the image generation unit 13 generates an image corresponding to the keyword defined in the thesaurus information as a subordinate concept of the specified keyword. That is, the image generation unit 13 generates an image representing a concept (keyword) related to the concept (keyword) specified in the search condition.
  • the image generation unit 13 generates the following image, for example.
  • "car” is acquired as a search condition
  • "ordinary car”, “small car”, and “bus” are defined as subordinate concepts of "car” according to the thesaurus information shown in FIG. Therefore, the image generation unit 13 generates three types of images: an image of a "normal car”, an image of a "small car”, and an image of a "bus".
  • the image generation unit 13 may generate an image representing the concept itself specified in the search condition instead of an image of the concept related to the concept specified in the search condition. For example, when "male" is acquired as a search condition, the image generation unit 13 may generate one type of image representing "male".
  • the image generation unit 13 may generate only one type of image, or may generate a plurality of types of images.
  • search conditions include multiple keywords (concepts)
  • a search condition including "red” and “car” a variation image for "red” can be generated, and a variation image for "car” can also be generated.
  • the predetermined priority is the order of the object, the position of the object, the orientation of the object, the color of the object, and the movement of the object.
  • the designated order of the objects or modes in the search conditions acquired by the search condition acquisition unit 12 may be used.
  • objects or modes may be specified in order of importance.
  • the variation image of the previously specified object or mode may be preferentially displayed. Therefore, the image generation unit 13 may preferentially generate a variation image of the previously specified object or mode. Further, the image generation unit 13 generates variation images of all the designated objects or modes, and the image display unit 14, which will be described later, gives priority to the variation images of the previously specified object or mode among those images. It may be displayed in.
  • the image display unit 14 may determine the priority in displaying the image according to the designated order of the object or the mode in the search condition acquired by the search condition acquisition unit 12. According to such a configuration, it is possible to preferentially present a variation image of a concept that the user emphasizes, so that the user can easily select a variation image that reflects his / her intention.
  • the content of the image to be generated is determined using the thesaurus information, but the content of the image to be generated may be determined by another method.
  • the variation image to be generated may be determined by referring to the hierarchical structure of the index defined in advance for the image data set to be searched.
  • the default settings may be used for modes that are not specified in the search conditions acquired by the search condition acquisition unit 12. For example, when "red car" is acquired as a search condition, the mode regarding the orientation of the object and the position of the object is not specified in this search condition.
  • the image generation unit 13 generates an image in which an object having a predetermined orientation exists at a predetermined position in the image. For example, the image generation unit 13 generates an image in which the red car viewed from the front is depicted at the center of the image.
  • the image generation unit 13 When the image generation unit 13 specifies the content of the image to be generated, the image generation unit 13 generates an image corresponding to the content by using any known technique. For example, the image generation unit 13 selects image data that matches the content of the image to be generated from a group of image data prepared in advance that represents a keyword defined in the thesaurus information (see FIG. 3) regarding the object.
  • the image data group representing the keyword defined in the cissolus information about the object is, for example, image data of a figure representing a car, image data of a figure representing an ordinary car, image data of a figure representing a small car, and an image of a figure representing a bus. Data etc. Note that these image data do not necessarily have to be prepared in advance.
  • the image generation unit 13 may generate an image of the object from the keyword of the object by a known image generation technique. Then, the image generation unit 13 uses the image data of the object to generate an image in which the object is represented in a mode determined based on the search condition or the default setting. For example, the image generation unit 13 generates an image in which an object is colored with a color determined based on the acquired search conditions or default settings. Any drawing software, including computer graphics software, may be used to generate the image.
  • the generated image may be a still image or a moving image.
  • the image generation unit 13 When the generated image is a moving image, the image generation unit 13 generates the moving image by, for example, combining a plurality of continuous still images representing the movement of an object. Examples of still images include paintings, figures, clip art, and illustrations, and examples of moving images include video images and animations, but the types of images are not limited to these.
  • the user may specify image data of a figure created by himself / herself using a drawing tool or the like.
  • the image generation unit 13 may generate an image in which the object is represented in a mode determined based on the search condition or the default setting by using the image data of the graphic created by the user.
  • the control unit 15 corresponds to the selection reception unit 4 in FIG.
  • the control unit 15 receives an instruction from the user to select one or more types of images from the images displayed by the image display unit 14. Further, the control unit 15 receives an instruction for determining the search condition from the user. In addition, the control unit 15 performs control processing including control for requesting the user to select an image, control for re-input of search conditions, and the like.
  • the user confirms whether or not the image group displayed by the image display unit 14 includes an image of the content reflecting his / her intention, and if there is an image of the content reflecting his / her intention, the image of the content reflecting his / her intention is displayed. Select one or more.
  • the user can re-enter the search condition after confirming the image displayed by the image display unit 14.
  • the image generation process by the image generation unit 13 and the display process by the image display unit 14 are performed again. These processes are repeated until an instruction for determining a search condition is received from the user.
  • the search condition determination unit 16 corresponds to the search condition determination unit 5 of FIG. 1, and when the search condition determination instruction is received, the search condition determination unit 16 is based on the image selected by the image selection instruction received by the control unit 15. Determine the search conditions. That is, the search condition determination unit 16 sets the search condition corresponding to the content of the selected image as the search condition used for the search process. Specifically, the object and the mode of the object represented by the selected image are specified as the search target, and the object and the mode are set as the search conditions.
  • the image search unit 17 searches for an image corresponding to this search condition according to the search condition determined by the search condition determination unit 16. That is, the image search unit 17 searches for an image that matches the search conditions from the image data set.
  • FIG. 8 is a block diagram showing an example of the hardware configuration of the information processing device 10.
  • the information processing device 10 includes, for example, a network interface 50, a memory 51, a processor 52, an input device 53, and a display device 54.
  • the network interface 50 is used to communicate with other devices. For example, it is used when the information processing device 10 receives the input from the user via another device, or when the image is presented to the user via the other device.
  • the network interface 50 may include, for example, a network interface card (NIC).
  • the memory 51 is composed of, for example, a combination of a volatile memory and a non-volatile memory.
  • the memory 51 is used to store software (computer program) or the like including one or more instructions executed by the processor 52.
  • Non-temporary computer-readable media include various types of tangible storage media. Examples of non-temporary computer-readable media are magnetic recording media (eg flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (eg magneto-optical disks), CompactDisc ReadOnlyMemory (CD-ROM), CD-ROM. Includes R, CD-R / W, and semiconductor memory (eg, mask ROM, Programmable ROM (PROM), Erasable PROM (EPROM), flash ROM, Random Access Memory (RAM)).
  • the program may also be supplied to the computer by various types of temporary computer readable media. Examples of temporary computer-readable media include electrical, optical, and electromagnetic waves.
  • the temporary computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire and an optical fiber, or a wireless communication path.
  • the processor 52 may be, for example, a microprocessor, an MPU (Micro Processor Unit), a CPU (Central Processing Unit), or the like.
  • the processor 52 may include a plurality of processors.
  • the processor 52 reads a computer program from the memory 51 and executes it to process the search condition acquisition unit 12, the image generation unit 13, the image display unit 14, the control unit 15, the search condition determination unit 16, and the image search unit 17. I do.
  • the thesaurus storage unit 11 is realized by a memory 51 or a storage device (not shown).
  • data required for processing such as an image data set, is also stored in the memory 51 or a storage device in advance.
  • the input device 53 is a device such as a keyboard that accepts input from the user.
  • the display device 54 is a device such as a display that displays information.
  • FIG. 9 is a flowchart showing an operation flow of the information processing device 10.
  • the operation of the information processing apparatus 10 will be described with reference to FIG.
  • step S100 the search condition acquisition unit 12 acquires the search condition input by the user.
  • step S101 the image generation unit 13 refers to the thesaurus information and specifies a keyword corresponding to the search condition acquired in step S100 in the thesaurus information. Further, the image generation unit 13 specifies a keyword defined in the thesaurus information as a subordinate concept of the specified keyword.
  • step S102 the image generation unit 13 generates a variation image corresponding to the specific result in step S101.
  • step S103 the image display unit 14 displays the image generated in step S102 on the display.
  • step S104 the control unit 15 outputs a message to select an image whose content matches the user's search intention from the images displayed in step S103, and prompts the user to select the image.
  • the user can modify the search condition with or without the selection of the image.
  • step S105 the control unit 15 determines whether or not the instruction for selecting an image and the instruction for determining the search condition have been accepted. When these instructions are received, the process proceeds to step S107. On the other hand, if there is no instruction to determine the search condition, the process proceeds to step S106. If there is no instruction to determine the search condition, the above-mentioned process will be repeated again. In this case, the image generation unit 13 may generate a new variation image based on the modified search condition, or may generate a new variation image based on the selected image.
  • step S106 the control unit 15 determines whether the search condition has been modified.
  • the process returns to step S100, and the search condition is acquired again. That is, in step S102, an image is generated based on the new search condition. If the search condition is not modified, the process returns to step S101.
  • step S102 when a new variation image is generated based on the selected image, the image generation unit 13 generates, for example, a variation image corresponding to a further subordinate concept of the keyword corresponding to the selected image. ..
  • step S107 the search condition determination unit 16 determines the search condition based on the selected image, and the image search unit 17 searches the image data set for an image that matches the search condition.
  • the information processing device 10 displays the variation image and accepts the user's selection for the variation image. Then, the search condition is determined according to the selection, and the search is performed using this search condition. According to such a configuration, it is possible to determine a search condition that reflects the user's intention in detail. Therefore, it is possible to provide search results according to the user's intention.
  • the information processing device 10 provides a search condition modification function and an image display function corresponding to the modification condition. That is, the search condition acquisition unit 12 newly acquires the search condition after the image is displayed by the image display unit 14.
  • the image display unit 14 is an image of an object specified by the newly acquired search condition, and is one or more types representing a variation of the object or a variation of the mode of the object specified by the search condition. Display the image. Therefore, the intention of the user can be appropriately grasped.
  • the information processing device 10 further generates a variation image based on the selected image. That is, the image display unit 14 displays one or more types of images representing variations in the mode of the object represented by the image selected according to the instruction received by the control unit 15. Therefore, the user's intention can be grasped in more detail.
  • FIG. 10 is a schematic diagram showing a flow of a search example of an image in which a person is drawn.
  • the information processing apparatus 10 acquires a search condition from the input text, generates an image based on the thesaurus information, and presents the image to the user.
  • step 1 it is assumed that the user has entered "male, red clothes" as a search condition.
  • the information processing device 10 refers to the thesaurus information regarding the object of FIG. 3 and the thesaurus information regarding the color of FIG. 4, and generates, for example, the following three types of images representing a masculine body.
  • the first image is an image of a man wearing red clothes.
  • the second image is an image of a man wearing dark red clothes.
  • the third image is an image of a man wearing light coral clothes.
  • the information processing device 10 displays these images and allows the user to select an image that suits his / her search intention. Suppose the user chooses an image of a man wearing dark red clothes. Further, it is assumed that the user who sees the displayed image and feels that the intention is not correctly transmitted to the information processing apparatus 10 changes the search condition "red clothes" to "upper body is red and lower body is gray”.
  • step 2 the information processing device 10 generates a new image based on the image selected in step 1 and the modified search condition.
  • three types of images are newly generated.
  • the first image is an image of a man wearing dark red upper body and gray lower body.
  • the second image is of a man wearing clothes with a brown upper body and a gray lower body.
  • the third image is an image of a man wearing a firebrick upper body and a gray lower body.
  • the information processing device 10 displays these images and allows the user to select an image that suits his / her search intention. On the other hand, it is assumed that the user selects an image of a man wearing clothes with a dark red upper body and a gray lower body, and does not change the search conditions.
  • step 3 the information processing device 10 generates a new image based on the image selected in step 2.
  • the information processing device 10 refers to the thesaurus information regarding the color shown in FIG. 4, and generates, for example, the following image.
  • the first image is an image of a man wearing dark red upper body and gray lower body.
  • the second image is of a man wearing dark red upper body and silver lower body.
  • the third image is an image of a man wearing dark red upper body and dark ash lower body.
  • the information processing device 10 displays these images and allows the user to select an image that suits his / her search intention.
  • the user has two types of images: an image of a man whose upper body is dark red and his lower body is gray, and an image of a man whose upper body is dark red and whose lower body is dark ash. Suppose you select. Further, it is assumed that the user adds "sunglasses" to the search condition.
  • step 4 the information processing device 10 generates a new image based on the image selected in step 3 and the added search condition.
  • the information processing device 10 generates an image of the person selected in step 3 with sunglasses attached.
  • the image of a person wearing sunglasses is generated here, the information processing device 10 may generate an image in which a figure representing sunglasses and a figure representing a person are drawn side by side.
  • an image depicting a person wearing sunglasses is generated instead of an image drawn by arranging sunglasses and a person side by side.
  • the information processing device 10 displays these images and allows the user to select an image that suits his / her search intention.
  • the user selects an image in which the lower half of the body is dark ash.
  • the user adds a condition that "the head is moving" to the search condition.
  • the information processing device 10 generates a new image based on the image selected in step 4 and the added search condition.
  • the information processing apparatus 10 refers to the thesaurus information related to the operation shown in FIG. 7 and generates an image.
  • the information processing apparatus 10 has these two types. Generates an image that represents the subordinate concept of.
  • the first set of generated images is a set of images showing how the head moves from side to side.
  • the first set consists of, for example, an image with the head facing left, an image with the head facing forward, and an image with the head facing right.
  • the second set of generated images is a set of images showing how the head moves up and down.
  • the second set consists of, for example, an image with the head facing up, an image with the head facing forward, and an image with the head facing down.
  • the information processing device 10 displays these two types of sets and allows the user to select a set that suits his / her search intention. On the other hand, it is assumed that the user selects the first set. Then, it is assumed that the user inputs an instruction for determining the search condition. In this case, the search condition determination unit 16 determines, for example, the search condition shown in FIG. 11 as the final search condition.
  • the search condition determination unit 16 uses the object specified by the selected image and its mode as the final search condition. Then, the image search unit 17 searches for an image based on the determined search conditions. As a result, the search that reflects the user's intention can be performed more than when the search is performed based on the search conditions input in step 1. In the example shown in FIG. 11, the default setting value is used as the search condition for the mode not specified by the user.
  • FIG. 12 is a schematic diagram showing a flow of a search example of an image depicting a car.
  • the information processing apparatus 10 acquires a search condition from the input text, generates an image based on the thesaurus information, and presents the image to the user.
  • step 1 it is assumed that the user has entered "car" as a search condition.
  • the information processing device 10 refers to the thesaurus information about the object of FIG. 3 and generates, for example, the following three types of images representing a car.
  • the first image is an image of an ordinary car.
  • the second image is an image of a small car.
  • the third image is an image of a bus.
  • the information processing device 10 displays these images and allows the user to select an image that suits his / her search intention. Suppose the user chooses an image of an ordinary car. Further, it is assumed that the user who sees the displayed image and feels that the intention is not correctly transmitted to the information processing device 10 modifies the search condition to "red car".
  • step 2 the information processing device 10 generates a new image based on the image selected in step 1 and the modified search condition.
  • the information processing apparatus 10 refers to the thesaurus information regarding colors shown in FIG. 4 and newly generates the following three types of images.
  • the first image is an image of an ordinary car whose color is red.
  • the second is an image of an ordinary car whose color is dark red.
  • the third is an image of an ordinary car whose color is light coral.
  • the information processing device 10 displays these images and allows the user to select an image that suits his / her search intention.
  • the user selects an image of an ordinary car whose color is red and does not change the search condition.
  • step 3 the information processing device 10 generates a new image based on the image selected in step 2.
  • the information processing device 10 refers to the thesaurus information regarding the color shown in FIG. 4, and generates, for example, the following image.
  • the first image is an image of an ordinary car whose color is red.
  • the second is an image of an ordinary car whose color is crimson.
  • the third is an image of an ordinary car whose color is orange-red.
  • the information processing device 10 displays these images and allows the user to select an image that suits his / her search intention. On the other hand, it is assumed that the user selects an image of an ordinary car whose color is orange-red and modifies the search condition to "red car in front".
  • step 4 the information processing device 10 generates a new image based on the vehicle image selected in step 3 and the added search condition.
  • the information processing apparatus 10 refers to the thesaurus information regarding the orientation of FIG. 5 and generates the following three types of images.
  • the first image is an image of an ordinary car in which the left side is slightly visible instead of the front, although the color is orange-red.
  • the second image is an image of an ordinary car facing straight ahead and whose color is orange-red.
  • the third image is an image of an ordinary car in which the right side can be seen to some extent instead of the front, and the color is orange-red.
  • the information processing device 10 displays these images and allows the user to select an image that suits his / her search intention. On the other hand, it is assumed that the user selects an image of an ordinary car facing straight ahead and whose color is orange-red. Further, it is assumed that the user adds the condition "there is a person next to" to the search condition.
  • step 5 the information processing device 10 generates a new image based on the image selected in step 4 and the added conditions.
  • the information processing apparatus 10 refers to the thesaurus information regarding the position shown in FIG. 5 and generates the following two types of images. That is, it is assumed that the following two types of images are generated based on the subordinate concepts of "next", "left” and "right".
  • the first image is an image in which a person is added to the left of the car selected in step 4.
  • the second image is an image in which a person is added to the right of the car selected in step 4.
  • the information processing device 10 displays these images and allows the user to select an image that suits his / her search intention.
  • the search condition determination unit 16 determines, for example, the search condition shown in FIG. 13 as the final search condition. That is, for example, the search condition determination unit 16 uses the object specified by the selected image and its mode as the final search condition. Then, the image search unit 17 searches for an image based on the determined search conditions. As a result, the search that reflects the user's intention can be performed more than the case where the search is performed based on the search conditions input in step 1. In the example shown in FIG. 13, the default setting value is used as the search condition for the mode not specified by the user.
  • (Appendix 1) Search condition acquisition means to acquire the entered search conditions, An image displaying one or more types of images representing an image of an object specified by the search condition acquired by the search condition acquisition means and representing a variation of the object or a variation of an aspect of the object specified by the search condition.
  • Display means and A selection receiving means for receiving an instruction to select one or more types of images displayed by the image displaying means, and a selection receiving means.
  • An information processing device having a search condition determining means for determining a search condition based on the image selected according to an instruction received by the selection receiving means.
  • the search condition acquisition means newly acquires a search condition after displaying an image by the image display means.
  • the image display means is a newly acquired image of an object specified by the search condition, and is one or more types of images representing a variation of the object or a variation of an aspect of the object specified by the search condition.
  • the information processing device according to Appendix 1.
  • Appendix 3 The information processing device according to Appendix 1 or 2, wherein the image display means displays one or more types of images representing variations of the mode of the object represented by the image selected according to an instruction received by the selection receiving means.
  • Appendix 4 The information processing according to any one of Supplementary note 1 to 3, wherein the image display means determines a priority in displaying an image according to a designated order of an object or an aspect in the search condition acquired by the search condition acquisition means. apparatus.
  • (Appendix 10) Get the entered search criteria and Display one or more types of images representing the variation of the object or the variation of the mode specified by the search condition of the object, which is an image of the object specified by the acquired search condition. Accepts instructions to select one or more of the displayed images, A search method for determining search conditions based on the image selected according to the received instruction. (Appendix 11) Search condition acquisition step to acquire the entered search condition, and An image display step of displaying one or more types of images representing the variation of the object or the variation of the aspect of the object specified by the search condition, which is an image of the object specified by the acquired search condition.
  • a selection acceptance step that accepts an instruction to select one or more types of the displayed images
  • Information processing device Search condition acquisition unit 3 Image display unit 4 Selection reception unit 5 Search condition determination unit 10 Information processing device 11 Sisorus storage unit 12 Search condition acquisition unit 13 Image generation unit 14 Image display unit 15 Control unit 16 Search condition determination Unit 17 Image search unit 50 Network interface 51 Memory 52 Processor 53 Input device 54 Display device

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Library & Information Science (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Multimedia (AREA)
  • Human Computer Interaction (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
PCT/JP2019/049299 2019-03-20 2019-12-17 情報処理装置、検索方法、及びプログラムが格納された非一時的なコンピュータ可読媒体 Ceased WO2020188924A1 (ja)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US17/436,299 US20220179899A1 (en) 2019-03-20 2019-12-17 Information processing apparatus, search method, and non-transitory computer readable medium storing program
JP2021506166A JP7238963B2 (ja) 2019-03-20 2019-12-17 情報処理装置、検索方法、及びプログラム

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2019053045 2019-03-20
JP2019-053045 2019-03-20

Publications (1)

Publication Number Publication Date
WO2020188924A1 true WO2020188924A1 (ja) 2020-09-24

Family

ID=72519058

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2019/049299 Ceased WO2020188924A1 (ja) 2019-03-20 2019-12-17 情報処理装置、検索方法、及びプログラムが格納された非一時的なコンピュータ可読媒体

Country Status (3)

Country Link
US (1) US20220179899A1 (https=)
JP (1) JP7238963B2 (https=)
WO (1) WO2020188924A1 (https=)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP6590329B1 (ja) * 2019-06-26 2019-10-16 株式会社ラディウス・ファイブ 画像表示システム及びプログラム
US12045735B1 (en) 2023-02-08 2024-07-23 Typeface Inc. Interactive template for multimodal content generation
US12411893B2 (en) 2023-02-08 2025-09-09 Typeface Inc. Proactively generated content and personalized feeds available for audiences

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH10289245A (ja) * 1997-04-15 1998-10-27 Canon Inc 画像処理装置及びその制御方法
JP2008217117A (ja) * 2007-02-28 2008-09-18 Fujifilm Corp 画像検索方法及び画像検索システム
JP2009009461A (ja) * 2007-06-29 2009-01-15 Fujifilm Corp キーワードの入力支援システム、コンテンツ検索システム、コンテンツ登録システム、コンテンツ検索・登録システム、およびこれらの方法、並びにプログラム
JP2014002493A (ja) * 2012-06-18 2014-01-09 Konica Minolta Inc 画像処理装置、画像処理方法およびプログラム

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6598054B2 (en) * 1999-01-26 2003-07-22 Xerox Corporation System and method for clustering data objects in a collection
US7099860B1 (en) * 2000-10-30 2006-08-29 Microsoft Corporation Image retrieval systems and methods with semantic and feature based relevance feedback
US9152624B1 (en) * 2003-12-04 2015-10-06 Retail Optimization International, Inc. Systems and methods for visual presentation and navigation of content using data-based image analysis
US7526476B2 (en) * 2005-03-14 2009-04-28 Microsoft Corporation System and method for generating attribute-based selectable search extension
US8190604B2 (en) * 2008-04-03 2012-05-29 Microsoft Corporation User intention modeling for interactive image retrieval
JP2011070412A (ja) * 2009-09-25 2011-04-07 Seiko Epson Corp 画像検索装置および画像検索方法
US20110202543A1 (en) * 2010-02-16 2011-08-18 Imprezzeo Pty Limited Optimising content based image retrieval
US9384216B2 (en) * 2010-11-16 2016-07-05 Microsoft Technology Licensing, Llc Browsing related image search result sets
US10664515B2 (en) * 2015-05-29 2020-05-26 Microsoft Technology Licensing, Llc Task-focused search by image
US10042866B2 (en) * 2015-06-30 2018-08-07 Adobe Systems Incorporated Searching untagged images with text-based queries
WO2019133849A1 (en) * 2017-12-29 2019-07-04 Ebay Inc. Computer vision and image characteristic search

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH10289245A (ja) * 1997-04-15 1998-10-27 Canon Inc 画像処理装置及びその制御方法
JP2008217117A (ja) * 2007-02-28 2008-09-18 Fujifilm Corp 画像検索方法及び画像検索システム
JP2009009461A (ja) * 2007-06-29 2009-01-15 Fujifilm Corp キーワードの入力支援システム、コンテンツ検索システム、コンテンツ登録システム、コンテンツ検索・登録システム、およびこれらの方法、並びにプログラム
JP2014002493A (ja) * 2012-06-18 2014-01-09 Konica Minolta Inc 画像処理装置、画像処理方法およびプログラム

Also Published As

Publication number Publication date
JPWO2020188924A1 (https=) 2020-09-24
JP7238963B2 (ja) 2023-03-14
US20220179899A1 (en) 2022-06-09

Similar Documents

Publication Publication Date Title
US12265655B2 (en) Moving windows between a virtual display and an extended reality environment
CN114612290B (zh) 图像编辑模型的训练方法和图像编辑方法
CN114187633B (zh) 图像处理方法及装置、图像生成模型的训练方法及装置
CN113362263B (zh) 变换虚拟偶像的形象的方法、设备、介质及程序产品
US7003140B2 (en) System and method of searching for image data in a storage medium
WO2020188924A1 (ja) 情報処理装置、検索方法、及びプログラムが格納された非一時的なコンピュータ可読媒体
US12271417B2 (en) Multi-image search
CN105096353A (zh) 一种图像处理方法及装置
CN120428866B (zh) 一种应用于古典名画展示的虚拟现实交互方法及系统
CN105580050A (zh) 提供图像中的控制点
WO2024088100A1 (zh) 特效处理方法、装置、电子设备和存储介质
Mattheij The eyes have it
CN119971503B (zh) 数据处理方法、装置、电子设备及计算机可读存储介质
JP7418709B2 (ja) コンピュータプログラム、方法及びサーバ装置
CN120451514B (zh) 图像处理方法、装置、设备及存储介质
KR102743852B1 (ko) 안면 데이터 관리 장치 및 방법
CN114170070A (zh) 对象属性编辑方法、装置、电子设备及可读存储介质
JP2026021163A (ja) システム
JP2026045668A (ja) システム
JP2026045169A (ja) システム
JP2026051195A (ja) システム
JP2026070226A (ja) システム
JP2026021068A (ja) システム
CN119624755A (zh) 图像风格迁移方法、装置、电子设备和存储介质
JP2025049030A (ja) システム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19920353

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021506166

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19920353

Country of ref document: EP

Kind code of ref document: A1