WO2013084422A1 - Information processing device, communication terminal, information search method, and non-temporary computer-readable medium - Google Patents

Information processing device, communication terminal, information search method, and non-temporary computer-readable medium Download PDF

Info

Publication number
WO2013084422A1
WO2013084422A1 PCT/JP2012/007342 JP2012007342W WO2013084422A1 WO 2013084422 A1 WO2013084422 A1 WO 2013084422A1 JP 2012007342 W JP2012007342 W JP 2012007342W WO 2013084422 A1 WO2013084422 A1 WO 2013084422A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
target image
information
information processing
display
Prior art date
Application number
PCT/JP2012/007342
Other languages
French (fr)
Japanese (ja)
Inventor
光洋 渡邊
Original Assignee
日本電気株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 日本電気株式会社 filed Critical 日本電気株式会社
Publication of WO2013084422A1 publication Critical patent/WO2013084422A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
    • G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
    • G06F3/04883—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures for inputting data by handwriting, e.g. gesture or text
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
    • G06F3/04842—Selection of displayed objects or displayed text elements

Definitions

  • the present invention relates to a technique for searching for related information of a subject displayed in a moving image.
  • Patent Document 1 discloses a system that specifies a constellation by performing a similar image search based on a night sky photographed image and provides information on the specified constellation.
  • Patent Document 2 discloses a system that searches for similar images based on diagnostic images in the medical field and provides case data associated with the searched similar images.
  • Patent Document 3 discloses a technique for retrieving attribute information (eg seller, price) of a subject (eg ⁇ costume worn by an actor) displayed in a video such as a television broadcast.
  • attribute information eg seller, price
  • the search system disclosed in Patent Document 3 uses moving picture identification information (eg channel number and broadcast date and time) and a selected position on the screen in order to identify the subject selected by the viewer. And information indicating the range is transmitted from the viewer side terminal to the server. And a server searches the attribute information linked
  • Patent Documents 1 and 2 uses an image matching technique. That is, the similarity is evaluated by comparing the feature amounts of two images.
  • Patent Document 3 performs a search using the identification information (eg channel number and broadcast date and time) of the moving image and information indicating the selection position and range on the screen as a key. Not something to do.
  • the moving image browsed by the viewer is, for example, a television broadcast, a movie, or a video of a sports competition or concert.
  • the moving image may be an image obtained by displaying a photographed image by a camera mounted on a mobile communication terminal (eg smart phone, tablet computer, notebook PC (Personal Computer)) substantially in real time on the display of the terminal.
  • the moving image may be a reproduced image of encoded moving image data (e.g. MPEG-2 data, MPEG-4 data) acquired from a recording medium (e.g.
  • the subject is, for example, a person, an animal, a plant, a product, an anime character, or the like.
  • the subject attribute information includes, for example, a person profile such as name, nationality, birthplace, date of birth, character name, product name, manufacturer, release date, price, URL (Uniform Resource Locator), and the like.
  • Patent Documents 1 to 3 disclose a technique that contributes to the solution of this problem.
  • the present invention has been made on the basis of the above-mentioned knowledge and consideration by the present inventor, and when searching for attribute information of a subject displayed in a moving image, an operation for designating a target image showing the subject from the moving image.
  • An object is to provide an information processing apparatus, a communication terminal, an information search method, and a program that can be facilitated.
  • the first aspect of the present invention includes an information processing apparatus.
  • the information processing apparatus includes a designation control unit and a search control unit.
  • the designation control unit accepts designation of a target image included in the moving image by an operation of an input device by a user in order to search for attribute information of a subject shown in the moving image displayed on the display.
  • the search control unit transmits the target image or a substitute image thereof to a search system, and receives attribute information searched based on the target image or the substitute image from the search system. Further, the designation control unit determines, as the target image, an image that has been displayed on the display before a reference time point that is before the completion time of the operation in order to compensate for a delay time required for execution of the operation. To do.
  • the second aspect of the present invention includes a communication terminal.
  • the communication terminal includes the information processing apparatus, the display, the input device, and the communication unit according to the first aspect of the present invention described above.
  • the communication unit is used for transmission of the target image and reception of the attribute information by the search control unit.
  • a third aspect of the present invention includes an information search method by an information processing device.
  • the information retrieval method includes the following steps (a) to (c).
  • the accepting in (a) means that an image displayed on the display before a reference time point before the completion time of the operation is compensated for in order to compensate for a delay time required for execution of the operation. Including determining as an image.
  • the third aspect of the present invention includes a program for causing a computer to perform the information search method according to the third aspect of the present invention described above.
  • an information processing apparatus capable of facilitating an operation of designating a target image in which a subject is displayed from the video when searching for attribute information of the subject displayed in the video,
  • a communication terminal, an information search method, and a program can be provided.
  • FIG. 1 is a diagram showing a network configuration including communication terminals according to the present embodiment.
  • the communication terminal 1 has a wireless or wired transceiver and can communicate with the search system 9 via the network 8.
  • Specific examples of the communication terminal 1 include a mobile phone terminal, a smartphone, a tablet computer, a notebook PC, a desktop PC, and a television broadcast receiver having a communication function.
  • the network 8 is a data transfer network such as an IP (Internet Protocol) network.
  • the network 8 may be a wired network, a wireless network, or a combination thereof.
  • the network 8 includes, for example, a radio access network and packet core network of a communication carrier, an IP leased line, and the public Internet.
  • the communication terminal 1 transmits the target image selected from the video by the viewer to the search system 9 in order to search for the attribute information of the subject shown in the video displayed on the display.
  • the target image may be an image for one screen (one frame) or a partial image corresponding to a part of the screen. Details of the method of selecting a target image in the communication terminal 1 will be described later.
  • the search system 9 specifies a subject shown in the target image by performing a similar image search using the target image received from the communication terminal 1. Further, the search system 9 transmits attribute information related to the identified subject to the communication terminal 1.
  • the subject is, for example, a person, an animal, a plant, a product, an animation character, or the like.
  • the subject attribute information includes, for example, a person profile such as name, nationality, birthplace, or date of birth, character name, product name, manufacturer, release date, price, URL (Uniform Resource Locator), and the like. .
  • the search system 9 may be a general-purpose image search server arranged on the Internet.
  • the search system 9 may be a search system specialized for a specific moving image.
  • the search system 9 may be a system specialized in searching for a person (performer) displayed on a television broadcast.
  • the search system 9 may determine an image to be preferentially collated with the target image by using information on a television program broadcast in the time zone in which the search is performed.
  • the search system 9 may preferentially collate the image of the performer of the television program broadcasted during the search time zone with the target image.
  • the communication terminal 1 sends the target image selected from the moving image by the viewer (user of the i.e. terminal 1) to the search system 9. Therefore, the viewer needs to perform an operation on the communication terminal 1 to specify a target image in which a subject desired to be searched is shown.
  • the display content of the video changes from when the viewer decides to search for the subject until the viewer finishes this operation, and the subject that the viewer wants to search moves to a different position on the screen. It is assumed that it no longer exists in the screen. That is, since the moving image display screen changes every moment, there is a problem that it is difficult to perform the selection operation of the target image.
  • a viewer operates a touch panel as the input device 13 at a timing when a desired subject is displayed on the display of the communication terminal.
  • the subject may be specified by touching the viewer so that the viewer surrounds the range on the touch panel where the desired subject is displayed on the display below (with an e.g. blue circle).
  • the display content of the moving image changes with time. For this reason, the display content of the video changes from when the viewer decides to search for the subject to when the operation on the touch panel is finished, and the subject that the viewer tries to search moves to a different position on the screen, It is assumed that they no longer exist within.
  • the communication terminal 1 has a function of supporting an operation for designating the target image. That is, the communication terminal 1 determines, as a target image, an image displayed on the display before a reference time determined based on the viewer's operation in order to compensate for the delay time required for the viewer to perform the operation. It is configured.
  • the reference time point is before the time point when the viewer's operation is completed.
  • FIG. 2 is a block diagram illustrating a configuration example of the communication terminal 1.
  • the communication terminal 1 shown in FIG. 2 has a wireless communication function.
  • the processor 10 performs control, monitoring, and information processing of the terminal 1.
  • the processor 10 may be a combination of a plurality of computers (e.g. MPU (Micro Processing Unit), microcontroller). More specifically, the processor 10 displays a moving image on the display 12, accepts an operation for selecting a target image, and performs a search system 9 in order to search for attribute information of the subject displayed in the moving image. Send and receive data.
  • the processor 10 includes a designation control unit 14 and a search control unit 15. Details of the designation control unit 14 and the search control unit 15 will be described later.
  • the wireless communication unit 11 connects to a wireless communication network via a base station (or a wireless access point).
  • the wireless communication unit 11 performs transmission path coding, interleaving, modulation (transmission symbol mapping), frequency up-conversion, signal amplification, and the like on transmission data to generate a transmission signal.
  • the wireless communication unit 11 generates reception data by performing each process such as signal amplification, frequency down-conversion, demodulation, error correction decoding, and the like on the reception signal from the antenna.
  • the wireless communication unit 11 includes known cellular communication methods such as UTRA (UMTS Terrestrial Radio Access), E-UTRA (Evolved UTRA), GSM (Global System for Mobile Communications) (registered trademark), and wireless LAN (Local Area Network). ), A transceiver conforming to WiMAX (Worldwide Interoperability for Microwave Access) or the like may be used.
  • the display 12 displays an image including a moving image so that a viewer (user of the terminal 1) can visually recognize the display.
  • Specific examples of the display 12 are a liquid crystal display (LCD), an EL (electroluminescence) display, and a CRT (Cathode Ray Tube) display.
  • the moving image displayed on the display 12 may be an image taken by a camera (not shown) mounted on the communication terminal 1.
  • the moving image is encoded moving image data (eg MPEG-2 data) acquired from a memory (eg optical disc, hard disk, flash memory) built in the communication terminal 1 or an external device accessible via the wireless communication unit 11. , MPEG-4 data).
  • the input device 13 is a device that accepts user operations.
  • the input device 13 includes at least one of a pointing device operated by a viewer (user of the terminal 1), a microphone that collects the viewer's voice, and a pointing device operated by the viewer's line of sight. Including. Specific examples of the pointing device operated by the viewer's hand include a touch panel, a touch pad, and a mouse.
  • the designation control unit 14 accepts designation of the target image included in the moving image by operating the input device 13 by the viewer in order to search for the attribute information of the subject shown in the moving image displayed on the display 12.
  • the input device 13 is a touch panel
  • the viewer touches the touch panel so as to surround the display range of the subject with a finger (eg, draw an ellipse circle) at the timing when the desired subject is displayed on the display 12.
  • the subject may be specified.
  • the viewer may perform an operation of touching one point of the subject in order to specify the subject.
  • the designation control unit 14 may identify an image region including the subject by performing image recognition processing such as face recognition on the region including one point touched by the viewer.
  • the viewer may operate the pointing device so as to enclose the display range of the subject with a pointer (e.g., draw an ellipse).
  • the viewer may perform an operation of designating one point of the subject with a pointer in order to designate the subject.
  • the search control unit 15 may specify the target image based on the timing at which the viewer inputs a word or phrase, or other input devices (eg, touch panel, mouse, operation buttons, etc.) by the viewer.
  • the target image may be specified by an operation.
  • the target image may be a screen image of the entire screen, not a partial image.
  • the search control unit 15 may transmit the target image as the screen image and the word or phrase input by the viewer to the search system 9.
  • the search system 9 may recognize a subject corresponding to the word or phrase input by the viewer from the target image, and perform a similar image search using the recognized subject image.
  • the designation control unit 14 selects an image displayed on the display 12 before the reference time determined based on the operation by the viewer. Determine as the target image.
  • the reference time is before the time when the operation by the viewer is completed.
  • the designation control unit 14 determines an image displayed on the display 12 at least before the completion of the operation as a target image.
  • the reference time point may be, for example, the completion time of the target image designation operation, the start time of the designation operation, or the central time point of the period required for the designation operation.
  • the extent to which an image before the reference point of operation by the viewer (eg) is determined as the target image may be determined statically or may be changed according to the viewer or according to the video. Also good.
  • a period hereinafter referred to as a shift time
  • a shift time may be determined in consideration of the average reaction speed of the person.
  • the shift time is dynamically determined according to the viewer
  • calibration for measuring the reaction speed of the viewer may be performed. Specifically, a test video is displayed, and after a specific subject appears in the test video, the time required for the viewer to complete the operation to select the display range of the subject is measured and measured. What is necessary is just to determine shift time according to the length of performed time. Further, the viewer may be able to freely change the initial value of the shift time based on the average reaction speed of the person. For example, when the viewer operates the terminal 1, it is possible to correct that the shift time is too long (going back too far in the past) and the shift time is too short (poor going back). Also good. It is assumed that the time required for the viewer to complete the operation of specifying the target image is different depending on the influence of the viewer's age and the like. Therefore, the target image intended by the viewer can be more appropriately identified by changing the shift time according to the viewer.
  • the shift time when the shift time is dynamically determined according to the moving image, the shift time may be changed according to the speed of movement of the subject reflected in the moving image.
  • a specific example of changing the shift time according to the moving image will be described in detail in another embodiment (Embodiment 3).
  • the search control unit 15 transmits the target image specified by the designation control unit 14 or a substitute image thereof to the search system 9 via the wireless communication unit 11. Then, the search control unit 15 receives the attribute information searched based on the target image or its substitute image from the search system 9 via the wireless communication unit 11.
  • the substitute image is an image suitable for similar image search because it includes a subject that is substantially the same as the target image specified by the designation control unit 14 but has a higher image quality than the target image.
  • An example of using an alternative image will be described in detail in another embodiment (Embodiment 4).
  • FIG. 3 is a flowchart showing a specific example of the information search method by the communication terminal 1 according to the present embodiment.
  • the communication terminal 1 displays a moving image on the display 12.
  • the communication terminal 1 accepts an operation of the input device 13 by a viewer for designating a target image.
  • the communication terminal 1 selects an image displayed before the reference time point of the operation by the viewer, in other words, an image displayed on the display 12 at least before the time point when the operation by the viewer is completed. Determine as.
  • step S ⁇ b> 4 the communication terminal 1 transmits the determined target image or a substitute image thereof to the search system 9.
  • the communication terminal 1 receives attribute information from the search system 9.
  • FIGS. 4A to 4D and FIG. 4A to 4D and FIG. 5 show a case where the input device 13 is a touch panel as an example.
  • the viewer designates a target image as a partial image by touching the touch panel with a finger so as to surround the display range of the subject with the finger.
  • FIGS. 4A to 4C each show an image 40 of one screen of the display 12 on which the subject 401 is projected.
  • the subject 401 is a person.
  • the display on the display 12 is assumed to change in the order of FIGS. 4A, 4B, and 4C over time. In other words, the subject 401 moves to the right of the screen as indicated by the white arrow in FIGS. 4A to 4C.
  • the viewer decides to search the attribute information of the subject 401 at the timing of FIG. 4A, and starts an operation of selecting to surround the face portion of the subject 401 at the timing of FIG. 4B.
  • An operation locus 402 in FIG. 4B indicates a locus that the viewer touches with a finger.
  • trajectory 403 of FIG. 4C has shown the operation locus at the time of a viewer completing operation.
  • the display content of the moving image changes every moment. Therefore, at the time of FIG. 4C, the subject 401 has moved in the right direction on the screen, and the desired subject 401 image does not exist within the range surrounded by the operation locus 403 by the viewer.
  • the communication terminal 1 uses, for example, the completion time of the operation as a reference time, and a shift time that is set in advance from the reference time
  • the previous image (or the shift time determined according to the viewer or the moving image) is adopted as the target image.
  • the subject 401 can be correctly selected as in the image 43 shown in FIG. 4D by going back to the operation start time by the shift time.
  • the compensation of the delay time from when the viewer starts the operation until the operation is completed has been described.
  • a delay time from when the viewer decides to perform the search to when the operation for that purpose is started For example, it is assumed that the display content of the moving image changes during the delay time from when the viewer tries to search for the subject in the moving image until the input device 13 is actually operated, and the subject is not displayed.
  • the start time of the viewer's operation may be set as the reference time, and the display image that is the shift time before the start time of the viewer's operation may be selected as the target image. Thereby, a display image close to the time when the viewer decides to perform the search can be selected as the target image.
  • the target image designation procedure shown in FIGS. 4A to 4D is merely an example.
  • the communication terminal 1 (designation control unit 14) automatically recognizes the subject displayed in the moving image, and may accept the designation of the subject to be selected from the automatically recognized subjects by the operation of the viewer. Good.
  • the communication terminal 1 performs face recognition processing on the image 40 of one screen of the display 12 on which the subject 401 is projected, and includes a face including the detected person's face.
  • a frame 404 indicating a region may be displayed over the image 40.
  • the communication terminal 1 should just receive the operation of the viewer who designates the frame 404 using the input device 13, for example.
  • the input device 13 is a touch panel
  • an operation in which the viewer touches the frame 404 or an area inside the frame 404 may be used as the target image specifying operation.
  • the communication terminal 1 may display the detected plurality of subject regions (e.g. a face region) with a frame 404 or the like. That is, the operation of specifying the target image (subject) can be simplified because the communication terminal 1 can further simplify the operation of specifying the target image (subject) by using the subject automatic recognition function to present the target image candidate. Can be shortened.
  • a display image that is a shift time before the reference time point (e.g. start time, central time point, or completion time) of the viewer's operation may be selected as the target image.
  • the shift time here may be determined in consideration of the delay time from when the viewer decides to perform the search to when the operation is performed. Thereby, a display image close to the time when the viewer decides to perform the search can be selected as the target image.
  • the communication terminal 1 according to the present embodiment is effective regardless of the type of moving image displayed on the display 12.
  • the above-described problem relating to the designation of the subject in the moving image has a particularly large effect when the viewer is watching a television broadcast. This is because, in general, television broadcasting cannot be freely paused or rewound at the viewer's own will. Therefore, the communication terminal 1 according to the present embodiment is particularly effective when the moving image displayed on the display 12 is a television broadcast.
  • the processing performed by the designation control unit 14 and the search control unit 15 described in the present embodiment may be realized using a semiconductor processing apparatus including an ASIC (Application Specific Integrated Circuit).
  • ASIC Application Specific Integrated Circuit
  • These processes may be realized by causing a computer such as a microprocessor or a DSP (Digital Signal Processor) to execute a program.
  • One or a plurality of programs including a group of instructions for causing the computer to execute the algorithm described with reference to FIGS. 1 to 4 may be created and the programs may be supplied to the computer.
  • Non-transitory computer readable media include various types of tangible storage media (tangible storage medium). Examples of non-transitory computer-readable media include magnetic recording media (eg flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (eg magneto-optical discs), CD-ROMs (Read Only Memory), CD-Rs, CD-R / W, semiconductor memory (for example, mask ROM, PROM (Programmable ROM), EPROM (Erasable ROM), flash ROM, RAM (random access memory)) are included.
  • the program may also be supplied to the computer by various types of temporary computer-readable media. Examples of transitory computer readable media include electrical signals, optical signals, and electromagnetic waves.
  • the temporary computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire and an optical fiber, or a wireless communication path.
  • FIG. 6 shows a configuration example when the communication terminal 1 is manufactured using a computer system.
  • the LCD 121 is a specific example of the display 12.
  • the touch panel 131 and the microphone 132 are specific examples of the input device 13.
  • An operating system (OS) 103 and a search application program 104 stored in a nonvolatile storage unit 102 are loaded into a RAM (Random Access Memory) 101.
  • the MPU (Micro Processing Unit) 100 executes the OS 103 and the search application program 104 loaded in the RAM 101, thereby realizing the functions of the designation control unit 14 and the search control unit 15.
  • the communication terminal 1 determines the presence / absence of a scene change of a moving image within a first period before a reference time (eg operation start time or operation completion time) related to a viewer's operation. Then, when there is a scene change, the terminal 1 determines an image before the scene change as a target image. The presence or absence of a scene change may be determined by comparing an image related to the reference time point with an image before the reference time point.
  • a reference time eg operation start time or operation completion time
  • the magnitude of change in pixel value for each pixel or pixel block including a plurality of pixels is calculated, and the scene change occurs when the amount of change in pixel value exceeds a predetermined reference What is necessary is just to determine with existence.
  • the viewer can specify the subject relatively easily.
  • scenes sometimes switch discontinuously. Therefore, if a scene change occurs between the time when the viewer decides to search for a subject and the time when the operation is started, the viewer may lose the opportunity to search for the subject.
  • the presence / absence of a scene change is determined, and when there is a scene change, an image before the scene change is determined as a target image. For this reason, the viewer can appropriately select a target image for which a search is desired.
  • FIG. 7 is a flowchart showing a specific example of the information search method by the communication terminal 1 according to the present embodiment.
  • the processes in steps S1, S2, S4, and S5 in FIG. 7 are the same as the steps with the same reference numerals shown in FIG. Therefore, the redundant description regarding these steps is omitted.
  • Steps S31 to S34 in FIG. 7 show a modification of step S3 in FIG.
  • the communication terminal 1 determines whether or not there is a scene change within a predetermined period before the reference time point (e.g. start time of the e.g. operation) by the viewer.
  • the predetermined period of step S31 may be determined statically or may be dynamically determined according to the viewer or the moving image. Good.
  • the communication terminal 1 determines an image before the scene change as a target image (step S33).
  • the communication terminal 1 (designation control unit 14) may select the image at the reference time point of the operation by the viewer as the target image. The previous image may be selected (step S34).
  • the communication terminal 1 changes the shift time according to the speed of movement of the subject included in the moving image.
  • the communication terminal 1 according to the present embodiment changes the shift time according to the magnitude of the motion vector between a plurality of images included in the moving image.
  • a motion vector between a plurality of images included in a moving image may be calculated to determine the magnitude of the motion vector.
  • the shift time may be relatively longer as the movement of the subject shown in the moving image is faster, that is, as the motion vector is larger. It is assumed that the faster the movement of the subject, the more confusing the viewer's operation is, or the longer it takes to complete the operation to accurately select the subject. Therefore, the target image intended by the viewer can be more appropriately identified by increasing the shift time as the motion of the subject shown in the moving image is faster, that is, as the motion vector is larger.
  • FIG. 8 is a flowchart showing a specific example of the information search method by the communication terminal 1 according to the present embodiment.
  • the processes in steps S1, S2, S4, and S5 in FIG. 8 are the same as the steps with the same reference numerals shown in FIG. Therefore, the redundant description regarding these steps is omitted.
  • Steps S35 to S37 in FIG. 8 show a modification of step S3 in FIG.
  • the communication terminal 1 (designation control unit 14) calculates a motion vector between a plurality of images in the moving image.
  • step S36 the communication terminal 1 (designation control unit 14) determines the shift time according to the calculated magnitude of the motion vector.
  • the communication terminal 1 (designation control unit 14) determines, as a target image, a display image that is a shift time before the reference time point of the operation by the viewer.
  • FIG. 9 is a block diagram showing a configuration example of the communication terminal 4 according to the present embodiment.
  • the communication terminal 4 displays on the display 12 an image obtained by photographing a television broadcast screen displayed on an external television receiver or the like with the camera 161 mounted on the terminal 4.
  • the designation control unit 14 applies the target image from the moving image (that is, the video including the TV broadcast screen shot by the camera 161) displayed on the display 12 according to any of the methods described in the first to third embodiments. To decide.
  • the search control unit 45 acquires a substitute image corresponding to the target image determined by the designation control unit 14. More specifically, the search control unit 45 uses the TV tuner 162 mounted on the terminal 1 to display a TV broadcast image at substantially the same time as the target image displayed on the display 12 after being captured by the camera 161. It is acquired as a substitute image by using.
  • This embodiment assumes a case where the viewer searches for attribute information such as the name of the performer when the viewer is watching a television program using a general television broadcast receiver.
  • the viewer can also obtain a desired attribute by using any of the information retrieval methods described in Embodiments 1 to 3 described above for the moving image captured by the camera 161 and displayed on the display 12.
  • information can be acquired.
  • the television broadcast screen shot by the camera 161 may be inferior in image quality to the video obtained by the television tuner 162 mounted on the terminal 4.
  • image quality deteriorates when the screen of a television broadcast receiver is taken from a distance or when camera shake occurs during shooting.
  • the video obtained by the TV tuner 162 is used as a substitute image, there is an advantage that a substitute image having higher image quality than the target image can be used for similar image search.
  • the search control unit 45 compares the image of the camera 161 with the image of the TV tuner 162, or compares the sound acquired by the microphone (not shown) with the sound of the TV tuner 162, or Based on these combinations, it is only necessary to determine whether the TV broadcast screen shot by the camera 161 matches the channel of the TV tuner 162 video. Further, the search control unit 45 may detect characters in channel information included in the television broadcast screen shot by the camera 161. Furthermore, instead of these automatic channel identifications, the viewer may designate a channel by operating the terminal 4.
  • FIG. 10 is a flowchart showing a specific example of the information search method by the communication terminal 4 according to the present embodiment.
  • the processes in steps S2, S3, and S5 in FIG. 10 are the same as the steps with the same reference numerals shown in FIG. Therefore, the redundant description regarding these steps is omitted.
  • step S11 the communication terminal 4 displays on the display 12 a moving image obtained by photographing the TV broadcast screen with the camera 161.
  • the communication terminal 4 acquires a substitute image related to the target image determined in step S ⁇ b> 3 from the video obtained by the television tuner 162.
  • step S ⁇ b> 42 the communication terminal 4 transmits a substitute image to the search system 9.
  • the communication terminal 4 may detect a television broadcast screen reflected in an image captured by the camera 161. A specific example of detecting a television broadcast screen will be described with reference to FIG.
  • FIG. 11 is a flowchart showing a specific example of step S41 in FIG.
  • the search control unit 45 detects a television broadcast screen from images captured by the camera 161. More specifically, the search control unit 45 may detect a rectangular frame of the television broadcast receiver from the image captured by the camera 161.
  • step S412 the search control unit 45 identifies the partial image selected by the viewer (user of the terminal 4) based on the position and size of the television broadcast screen in the image captured by the camera 161.
  • step S413 the search control unit 45 determines a video by the TV tuner 162 corresponding to the partial image selected by the viewer as a substitute image.
  • FIG. 12 is a block diagram illustrating a configuration example of the communication terminal 5 according to the present embodiment.
  • the communication terminal 5 has a recording control unit 16.
  • the recording control unit 16 uses the television tuner 162 to automatically record the television broadcast program related to the attribute information acquired from the search system 9.
  • FIG. 13 is a flowchart showing a specific example of the information search method by the communication terminal 5 according to the present embodiment.
  • the processes in steps S1 to S5 in FIG. 13 are the same as the steps with the same reference numerals shown in FIG. Therefore, the redundant description regarding these steps is omitted.
  • the communication terminal 5 (recording control unit 16) automatically records a television broadcast program related to the attribute information acquired from the search system 9.
  • the recording control unit 16 may automatically record a television program in which a person corresponding to the subject name included in the attribute information appears.
  • the recording control unit 16 acquires a TV program guide or accesses a server (for example, a World Wide Web server) that holds TV program performer information, so that the person corresponding to the person name included in the attribute information
  • the communication terminal 5 will appear in the future in response to a viewer who is watching a television program specifying an image of a performer of the program and instructing an image search.
  • Other TV programs can be easily reserved automatically. That is, even when the viewer does not know the name of the performer, the viewer can make a recording reservation for another program in which the performer appears.
  • the example in which the display 12, the input device 13, the camera 161, and the television tuner 162 are mounted on the communication terminals 1, 4, and 5 together with the processor 10 has been described.
  • these devices only need to be combined with the processor 10 and need not be configured as an integrated communication terminal.
  • these devices and the processor 1 only need to be able to communicate using a wireless communication function such as a wireless LAN or Bluetooth (registered trademark) or a wired communication function.
  • Search control unit Recording control unit 40 to 43 Image 45 Search control unit 8 Network 9 Search system 100 MPU (Micro Processing Unit) 101 RAM (Random Access Memory) 102 Non-volatile storage unit 103 Operating system 104 Search application program 121 LCD (Liquid Crystal Display) 131 Touch Panel 132 Microphone 161 Camera 162 Television Tuner 401 Subject 402, 403 Operation Trajectory 404 Frame Showing Face Area

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Library & Information Science (AREA)
  • Human Computer Interaction (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • User Interface Of Digital Computer (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

In an embodiment, an information processing device (1) includes a specification control part (14) and a search control part (15). In order to search for attribute information for an object imaged in a video being displayed on a display (12), the specification control part (14) receives a specification for the target image included in the relevant video in accordance with a user operating an input device (13). The search control part (15) sends the target image or a substitute image thereof to a search system (9), and receives the attribute information retrieved on the basis of the target image or the substitute image from the search system (9). In addition, to compensate for the delay time required to execute the user operation, the specification control part (14) decides that the image being displayed on the display (12) is the target image even before the relevant operation has been completed.

Description

情報処理装置、通信端末、情報検索方法、及び非一時的なコンピュータ可読媒体Information processing apparatus, communication terminal, information search method, and non-transitory computer-readable medium
　本発明は、動画に表示されている被写体の関連情報を検索する技術に関する。 The present invention relates to a technique for searching for related information of a subject displayed in a moving image.
　ある画像をもとに類似画像を検索し、さらに検索された類似画像に関連付けられた属性情報を取得する技術が知られている（例えば、特許文献１及び２を参照）。特許文献１は、夜空の撮影画像をもとに類似画像検索を行うことによって星座を特定し、特定された星座に関する情報を提供するシステムを開示している。また、特許文献２は、医療分野における診断画像をもとに類似画像の検索を行い、検索された類似画像に関連付けられた症例データを提供するシステムを開示している。 A technique is known in which similar images are searched based on a certain image, and attribute information associated with the searched similar images is acquired (see, for example, Patent Documents 1 and 2). Patent Document 1 discloses a system that specifies a constellation by performing a similar image search based on a night sky photographed image and provides information on the specified constellation. Patent Document 2 discloses a system that searches for similar images based on diagnostic images in the medical field and provides case data associated with the searched similar images.
　一方、特許文献３は、テレビジョン放送等の動画に表示されている被写体（e.g. 俳優が身に付いけている衣装）の属性情報（e.g. 販売者、価格）を検索する技術を開示している。より具体的に述べると、特許文献３に開示された検索システムは、視聴者が選択した被写体を特定するために、動画の識別情報（e.g. チャンネル番号および放送日時）と、画面内での選択位置及び範囲を示す情報を視聴者側の端末からサーバに送信する。そして、サーバは、動画の識別情報と選択位置及び範囲を示す情報に関連付けられた属性情報を検索し、得られた属性情報を端末に送信する。 On the other hand, Patent Document 3 discloses a technique for retrieving attribute information (eg seller, price) of a subject (eg 衣装 costume worn by an actor) displayed in a video such as a television broadcast. . More specifically, the search system disclosed in Patent Document 3 uses moving picture identification information (eg channel number and broadcast date and time) and a selected position on the screen in order to identify the subject selected by the viewer. And information indicating the range is transmitted from the viewer side terminal to the server. And a server searches the attribute information linked | related with the information which shows the identification information, selection position, and range of a moving image, and transmits the obtained attribute information to a terminal.
　特許文献１及び２に開示されている一般的な類似画像検索は、画像照合技術を利用している。すなわち、２つの画像の特徴量を比較することで類似度を評価する。これに対して、特許文献３は、動画の識別情報（e.g. チャンネル番号および放送日時）と画面内での選択位置及び範囲を示す情報をキーとして検索を行うものであって、いわゆる類似画像検索を行うものではない。 The general similar image search disclosed in Patent Documents 1 and 2 uses an image matching technique. That is, the similarity is evaluated by comparing the feature amounts of two images. On the other hand, Patent Document 3 performs a search using the identification information (eg channel number and broadcast date and time) of the moving image and information indicating the selection position and range on the screen as a key. Not something to do.
特開２００５－１７４２４０号公報JP 2005-174240 A 特開２００４－００５３６４号公報JP 2004-005364 A 特開２００２－３３４０９２号公報JP 2002-334092 A
　視聴者が動画を閲覧している際に、その動画に表示されている被写体の属性情報を知りたいと思っても、それを即座に検索するためには問題が存在する。ここで、視聴者によって閲覧される動画は、例えば、テレビ放送、映画、又はスポーツ競技若しくはコンサート等を撮影したビデオなどである。また、この動画は、携帯通信端末（e.g. スマートフォン、タブレットコンピュータ、ノートＰＣ（Personal Computer））に搭載されたカメラによる撮影画像を当該端末のディスプレイに実質的にリアルタイムに表示したものであってもよい。また、この動画は、記録媒体（e.g. 光ディスク、ハードディスク、フラッシュメモリ）や通信媒体から取得された符号化動画データ（e.g. MPEG-2データ、MPEG-4データ）の再生画像であってもよい。また、被写体は、例えば、人、動物、植物、製品、アニメキャラクターなどである。そして、被写体の属性情報は、例えば、名前、国籍、出身地、又は生年月日等の人物プロフィール、キャラクターの名称、製品名、製造者、発売日、価格、ＵＲＬ（Uniform Resource Locator）などである。 When a viewer is viewing a video, there is a problem in that even if the viewer wants to know the attribute information of the subject displayed in the video, it can be searched immediately. Here, the moving image browsed by the viewer is, for example, a television broadcast, a movie, or a video of a sports competition or concert. In addition, the moving image may be an image obtained by displaying a photographed image by a camera mounted on a mobile communication terminal (eg smart phone, tablet computer, notebook PC (Personal Computer)) substantially in real time on the display of the terminal. . The moving image may be a reproduced image of encoded moving image data (e.g. MPEG-2 data, MPEG-4 data) acquired from a recording medium (e.g. optical disc, hard disk, flash memory) or a communication medium. The subject is, for example, a person, an animal, a plant, a product, an anime character, or the like. The subject attribute information includes, for example, a person profile such as name, nationality, birthplace, date of birth, character name, product name, manufacturer, release date, price, URL (Uniform Resource Locator), and the like. .
　より具体的に述べると、動画に表示されている被写体の属性情報の検索に際して、視聴者は、検索を希望する被写体が映った対象画像を指定する操作を行う必要がある。しかしながら、動画の表示画面は刻々と変化するため、この対象画像の選択操作を行うことが難しいという問題がある。特許文献１～３のいずれも、この問題の解決に寄与する技術について何ら開示していない。 More specifically, when searching for attribute information of a subject displayed in a moving image, the viewer needs to perform an operation of specifying a target image in which the subject desired to be searched is shown. However, since the moving image display screen changes every moment, there is a problem that it is difficult to perform the selection operation of the target image. None of Patent Documents 1 to 3 disclose a technique that contributes to the solution of this problem.
　本発明は、本件発明者による上述した知見および考察に基づいてなされたものであり、動画に表示されている被写体の属性情報の検索に際して、被写体が映った対象画像を動画中から指定する操作を容易化することが可能な情報処理装置、通信端末、情報検索方法、及びプログラムの提供を目的とする。 The present invention has been made on the basis of the above-mentioned knowledge and consideration by the present inventor, and when searching for attribute information of a subject displayed in a moving image, an operation for designating a target image showing the subject from the moving image. An object is to provide an information processing apparatus, a communication terminal, an information search method, and a program that can be facilitated.
　本発明の第１の態様は、情報処理装置を含む。当該情報処理装置は、指定制御部及び検索制御部を含む。前記指定制御部は、ディスプレイに表示されている動画に映った被写体の属性情報の検索のために、前記動画に含まれる対象画像の指定を利用者による入力デバイスの操作によって受け付ける。前記検索制御部は、前記対象画像又はその代替画像を検索システムに送信するとともに、前記対象画像又は前記代替画像に基づいて検索された属性情報を前記検索システムから受信する。さらに、前記指定制御部は、前記操作の実行に要する遅延時間を補償するために、前記操作の完了時点以前である基準時点よりも前に前記ディスプレイに表示されていた画像を前記対象画像として決定する。 The first aspect of the present invention includes an information processing apparatus. The information processing apparatus includes a designation control unit and a search control unit. The designation control unit accepts designation of a target image included in the moving image by an operation of an input device by a user in order to search for attribute information of a subject shown in the moving image displayed on the display. The search control unit transmits the target image or a substitute image thereof to a search system, and receives attribute information searched based on the target image or the substitute image from the search system. Further, the designation control unit determines, as the target image, an image that has been displayed on the display before a reference time point that is before the completion time of the operation in order to compensate for a delay time required for execution of the operation. To do.
　本発明の第２の態様は、通信端末を含む。当該通信端末は、上述した本発明の第１の態様に係る情報処理装置、ディスプレイ、入力デバイス、及び通信部を含む。当該通信部は、前記検索制御部による前記対象画像の送信、及び前記属性情報の受信のために使用される。 The second aspect of the present invention includes a communication terminal. The communication terminal includes the information processing apparatus, the display, the input device, and the communication unit according to the first aspect of the present invention described above. The communication unit is used for transmission of the target image and reception of the attribute information by the search control unit.
　本発明の第３の態様は、情報処理装置による情報検索方法を含む。当該情報検索方法は、以下のステップ（ａ）～（ｃ）を含む。
（ａ）ディスプレイに表示されている動画に映った被写体の属性情報の検索のために、前記動画に含まれる対象画像の指定を利用者による入力デバイスの操作によって受け付けること、
（ｂ）前記対象画像又はその代替画像を検索システムに送信すること、及び
（ｃ）前記対象画像又は前記代替画像に基づいて検索された属性情報を前記検索システムから受信すること。
　さらに、（ａ）における前記受け付けることは、前記操作の実行に要する遅延時間を補償するために、前記操作の完了時点以前である基準時点よりも前に前記ディスプレイに表示されていた画像を前記対象画像として決定することを含む。 A third aspect of the present invention includes an information search method by an information processing device. The information retrieval method includes the following steps (a) to (c).
(A) accepting designation of a target image included in the moving image by an operation of an input device by a user in order to search for attribute information of a subject reflected in the moving image displayed on the display;
(B) transmitting the target image or a substitute image thereof to a search system; and (c) receiving attribute information searched based on the target image or the substitute image from the search system.
Further, the accepting in (a) means that an image displayed on the display before a reference time point before the completion time of the operation is compensated for in order to compensate for a delay time required for execution of the operation. Including determining as an image.
　本発明の第３の態様は、上述した本発明の第３の態様に係る情報検索方法をコンピュータに行わせるためのプログラムを含む。 The third aspect of the present invention includes a program for causing a computer to perform the information search method according to the third aspect of the present invention described above.
　上述した本発明の各態様によれば、動画に表示されている被写体の属性情報の検索に際して、被写体が映った対象画像を動画中から指定する操作を容易化することが可能な情報処理装置、通信端末、情報検索方法、及びプログラムを提供できる。 According to each aspect of the present invention described above, an information processing apparatus capable of facilitating an operation of designating a target image in which a subject is displayed from the video when searching for attribute information of the subject displayed in the video, A communication terminal, an information search method, and a program can be provided.
本発明の実施の形態１に係る通信端末を含むネットワーク構成図である。It is a network block diagram including the communication terminal which concerns on Embodiment 1 of this invention. 本発明の実施の形態１に係る通信端末の構成例を示すブロック図である。It is a block diagram which shows the structural example of the communication terminal which concerns on Embodiment 1 of this invention. 本発明の実施の形態１に係る通信端末による情報検索方法の具体例を示すフローチャートである。It is a flowchart which shows the specific example of the information search method by the communication terminal which concerns on Embodiment 1 of this invention. 本発明の実施の形態１に係る通信端末における対象画像の指定操作を説明するための図である。It is a figure for demonstrating designation | designated operation of the target image in the communication terminal which concerns on Embodiment 1 of this invention. 本発明の実施の形態１に係る通信端末における対象画像の指定操作を説明するための図である。It is a figure for demonstrating designation | designated operation of the target image in the communication terminal which concerns on Embodiment 1 of this invention. 本発明の実施の形態１に係る通信端末における対象画像の指定操作を説明するための図である。It is a figure for demonstrating designation | designated operation of the target image in the communication terminal which concerns on Embodiment 1 of this invention. 本発明の実施の形態１に係る通信端末における対象画像の指定操作を説明するための図である。It is a figure for demonstrating designation | designated operation of the target image in the communication terminal which concerns on Embodiment 1 of this invention. 本発明の実施の形態１に係る通信端末における対象画像の指定操作を説明するための図である。It is a figure for demonstrating designation | designated operation of the target image in the communication terminal which concerns on Embodiment 1 of this invention. 本発明の実施の形態１に係る通信端末の他の構成例を示すブロック図である。It is a block diagram which shows the other structural example of the communication terminal which concerns on Embodiment 1 of this invention. 本発明の実施の形態２に係る通信端末による情報検索方法の具体例を示すフローチャートである。It is a flowchart which shows the specific example of the information search method by the communication terminal which concerns on Embodiment 2 of this invention. 本発明の実施の形態３に係る通信端末による情報検索方法の具体例を示すフローチャートである。It is a flowchart which shows the specific example of the information search method by the communication terminal which concerns on Embodiment 3 of this invention. 本発明の実施の形態４に係る通信端末の構成例を示すブロック図である。It is a block diagram which shows the structural example of the communication terminal which concerns on Embodiment 4 of this invention. 本発明の実施の形態４に係る通信端末による情報検索方法の具体例を示すフローチャートである。It is a flowchart which shows the specific example of the information search method by the communication terminal which concerns on Embodiment 4 of this invention. 発明の実施の形態４に係る通信端末による代替画像の指定方法の具体例を示すフローチャートである。It is a flowchart which shows the specific example of the designation | designated method of the alternative image by the communication terminal which concerns on Embodiment 4 of invention. 本発明の実施の形態５に係る通信端末の構成例を示すブロック図である。It is a block diagram which shows the structural example of the communication terminal which concerns on Embodiment 5 of this invention. 本発明の実施の形態５に係る通信端末による情報検索方法の具体例を示すフローチャートである。It is a flowchart which shows the specific example of the information search method by the communication terminal which concerns on Embodiment 5 of this invention.
　以下では、本発明を適用した具体的な実施の形態について、図面を参照しながら詳細に説明する。各図面において、同一又は対応する要素には同一の符号が付されており、説明の明確化のため、必要に応じて重複説明は省略される。 Hereinafter, specific embodiments to which the present invention is applied will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are denoted by the same reference numerals, and redundant description is omitted as necessary for clarification of the description.
＜発明の実施の形態１＞
　図１は、本実施の形態に係る通信端末を含むネットワーク構成を示す図である。通信端末１は、無線又は有線のトランシーバを有し、ネットワーク８を介して検索システム９と通信可能である。通信端末１の具体例は、携帯電話端末、スマートフォン、タブレットコンピュータ、ノートＰＣ、デスクトップＰＣ、通信機能を有するテレビ放送受信機を含む。ネットワーク８は、ＩＰ（Internet Protocol）ネットワーク等のデータ転送網である。ネットワーク８は、有線ネットワークでもよいし、無線ネットワークでもよいし、これらの組み合わせであってもよい。ネットワーク８は、例えば、通信事業者の無線アクセスネットワーク及びパケットコアネットワーク、ＩＰ専用線、公衆インターネットを含む。 <Embodiment 1 of the Invention>
FIG. 1 is a diagram showing a network configuration including communication terminals according to the present embodiment. The communication terminal 1 has a wireless or wired transceiver and can communicate with the search system 9 via the network 8. Specific examples of the communication terminal 1 include a mobile phone terminal, a smartphone, a tablet computer, a notebook PC, a desktop PC, and a television broadcast receiver having a communication function. The network 8 is a data transfer network such as an IP (Internet Protocol) network. The network 8 may be a wired network, a wireless network, or a combination thereof. The network 8 includes, for example, a radio access network and packet core network of a communication carrier, an IP leased line, and the public Internet.
　通信端末１は、ディスプレイに表示されている動画に映った被写体の属性情報の検索のために、視聴者によって動画中から選択された対象画像を検索システム９に送信する。対象画像は、１画面（１フレーム）分の画像でもよいし、画面内の一部に相当する部分画像であってもよい。通信端末１における対象画像の選択方法の詳細については後述する。 The communication terminal 1 transmits the target image selected from the video by the viewer to the search system 9 in order to search for the attribute information of the subject shown in the video displayed on the display. The target image may be an image for one screen (one frame) or a partial image corresponding to a part of the screen. Details of the method of selecting a target image in the communication terminal 1 will be described later.
　検索システム９は、通信端末１から受信した対象画像を用いて類似画像検索を行うことにより、対象画像に映されている被写体を特定する。さらに、検索システム９は、特定された被写体に関連する属性情報を通信端末１に送信する。被写体は、上述したように、例えば、人、動物、植物、製品、アニメキャラクターなどである。また、被写体の属性情報は、例えば、名前、国籍、出身地、又は生年月日等の人物プロフィール、キャラクターの名称、製品名、製造者、発売日、価格、ＵＲＬ（Uniform Resource Locator）などである。 The search system 9 specifies a subject shown in the target image by performing a similar image search using the target image received from the communication terminal 1. Further, the search system 9 transmits attribute information related to the identified subject to the communication terminal 1. As described above, the subject is, for example, a person, an animal, a plant, a product, an animation character, or the like. The subject attribute information includes, for example, a person profile such as name, nationality, birthplace, or date of birth, character name, product name, manufacturer, release date, price, URL (Uniform Resource Locator), and the like. .
　検索システム９は、インターネット上に配置された汎用的な画像検索サーバであってもよい。また、検索システム９は、特定の動画に特化した検索システムであってもよい。例えば、検索システム９は、テレビ放送に表示される人物（出演者）の検索に特化したシステムであってもよい。この場合、検索システム９は、検索が行われる時間帯に放送されているテレビ番組の情報を利用することで、対象画像と優先的に照合すべき画像を決定してもよい。具体的には、検索システム９は、検索が行われる時間帯に放送されているテレビ番組の出演者の画像を優先的に対象画像と照合すればよい。 The search system 9 may be a general-purpose image search server arranged on the Internet. The search system 9 may be a search system specialized for a specific moving image. For example, the search system 9 may be a system specialized in searching for a person (performer) displayed on a television broadcast. In this case, the search system 9 may determine an image to be preferentially collated with the target image by using information on a television program broadcast in the time zone in which the search is performed. Specifically, the search system 9 may preferentially collate the image of the performer of the television program broadcasted during the search time zone with the target image.
　上述したように、通信端末１は、視聴者（i.e. 端末１の利用者）によって動画中から選択された対象画像を検索システム９に送る。そのために、視聴者は、検索を希望する被写体が映った対象画像を指定する操作を通信端末１に対して行う必要がある。しかしながら、視聴者が被写体の検索を決意してから視聴者がこの操作をし終えるまでに動画の表示内容が変化し、視聴者が検索しようとした被写体が画面内の別の位置に移動したり、画面内に存在しなくなったりすることが想定される。つまり、動画の表示画面は刻々と変化するため、この対象画像の選択操作を行うことが難しいという問題がある。 As described above, the communication terminal 1 sends the target image selected from the moving image by the viewer (user of the i.e. terminal 1) to the search system 9. Therefore, the viewer needs to perform an operation on the communication terminal 1 to specify a target image in which a subject desired to be searched is shown. However, the display content of the video changes from when the viewer decides to search for the subject until the viewer finishes this operation, and the subject that the viewer wants to search moves to a different position on the screen. It is assumed that it no longer exists in the screen. That is, since the moving image display screen changes every moment, there is a problem that it is difficult to perform the selection operation of the target image.
　例えば、動画中の被写体を指定する方法の１つとして、通信端末のディスプレイに所望の被写体が表示されたタイミングにおいて、入力デバイス１３としてのタッチパネルを視聴者が操作することが考えられる。例えば、その下のディスプレイに所望の被写体が表示されているタッチパネル上の範囲を視聴者が指で囲むように（e.g. 円を描くように）触ることによって、被写体の指定を行えばよい。しかしながら、動画の表示内容は時間の経過に従って変化する。このため、視聴者が被写体の検索を決意してからタッチパネルの操作を終えるまでに動画の表示内容が変化し、視聴者が検索しようとした被写体が画面内の別の位置に移動したり、画面内に存在しなくなったりすることが想定される。 For example, as one method for designating a subject in a moving image, it is conceivable that a viewer operates a touch panel as the input device 13 at a timing when a desired subject is displayed on the display of the communication terminal. For example, the subject may be specified by touching the viewer so that the viewer surrounds the range on the touch panel where the desired subject is displayed on the display below (with an e.g. blue circle). However, the display content of the moving image changes with time. For this reason, the display content of the video changes from when the viewer decides to search for the subject to when the operation on the touch panel is finished, and the subject that the viewer tries to search moves to a different position on the screen, It is assumed that they no longer exist within.
　この問題に対処するため、通信端末１は、この対象画像を指定する操作を支援する機能を有する。すなわち、通信端末１は、視聴者による操作の実行に要する遅延時間を補償するために、視聴者の操作に基づいて定まる基準時点より前にディスプレイに表示されていた画像を対象画像として決定するよう構成されている。ここで、基準時点は、視聴者の操作の完了時点以前である。以下では、通信端末１の構成例及び動作について詳細に説明する。 In order to deal with this problem, the communication terminal 1 has a function of supporting an operation for designating the target image. That is, the communication terminal 1 determines, as a target image, an image displayed on the display before a reference time determined based on the viewer's operation in order to compensate for the delay time required for the viewer to perform the operation. It is configured. Here, the reference time point is before the time point when the viewer's operation is completed. Below, the structural example and operation | movement of the communication terminal 1 are demonstrated in detail.
　図２は、通信端末１の構成例を示すブロック図である。図２に示された通信端末１は、無線通信機能を有する。プロセッサ１０は、端末１の制御、監視、情報処理を行う。プロセッサ１０は、複数のコンピュータ（e.g. ＭＰＵ（Micro Processing Unit）、マイクロコントローラ）の組み合わせであってもよい。より具体的に述べると、プロセッサ１０は、動画に表示されている被写体の属性情報の検索を実現するために、ディスプレイ１２への動画表示、対象画像を選択する操作の受付、検索システム９とのデータ送受信を行う。プロセッサ１０は、指定制御部１４及び検索制御部１５を含む。指定制御部１４及び検索制御部１５の詳細については後述する。 FIG. 2 is a block diagram illustrating a configuration example of the communication terminal 1. The communication terminal 1 shown in FIG. 2 has a wireless communication function. The processor 10 performs control, monitoring, and information processing of the terminal 1. The processor 10 may be a combination of a plurality of computers (e.g. MPU (Micro Processing Unit), microcontroller). More specifically, the processor 10 displays a moving image on the display 12, accepts an operation for selecting a target image, and performs a search system 9 in order to search for attribute information of the subject displayed in the moving image. Send and receive data. The processor 10 includes a designation control unit 14 and a search control unit 15. Details of the designation control unit 14 and the search control unit 15 will be described later.
　無線通信部１１は、基地局（又は無線アクセスポイント）を介して無線通信ネットワークに接続する。無線通信部１１は、送信データに対する伝送路符号化、インタリービング、変調（送信シンボルマッピング）、周波数アップコンバージョン、信号増幅等の各処理を行って送信信号を生成する。また、無線通信部１１は、アンテナによる受信信号に対して信号増幅、周波数ダウンコンバージョン、復調、誤り訂正復号等の各処理を行って受信データを生成する。無線通信部１１には、公知のセルラ通信方式、例えば、ＵＴＲＡ（UMTS Terrestrial Radio Access）、Ｅ－ＵＴＲＡ（Evolved UTRA）、ＧＳＭ（Global System for Mobile Communications）（登録商標）、無線ＬＡＮ（Local Area Network）、ＷｉＭＡＸ（Worldwide Interoperability for Microwave Access）等に準拠した送受信機を用いればよい。 The wireless communication unit 11 connects to a wireless communication network via a base station (or a wireless access point). The wireless communication unit 11 performs transmission path coding, interleaving, modulation (transmission symbol mapping), frequency up-conversion, signal amplification, and the like on transmission data to generate a transmission signal. In addition, the wireless communication unit 11 generates reception data by performing each process such as signal amplification, frequency down-conversion, demodulation, error correction decoding, and the like on the reception signal from the antenna. The wireless communication unit 11 includes known cellular communication methods such as UTRA (UMTS Terrestrial Radio Access), E-UTRA (Evolved UTRA), GSM (Global System for Mobile Communications) (registered trademark), and wireless LAN (Local Area Network). ), A transceiver conforming to WiMAX (Worldwide Interoperability for Microwave Access) or the like may be used.
　ディスプレイ１２は、視聴者（端末１の利用者）が視認できるように動画を含む画像の表示を行う。ディスプレイ１２の具体例は、液晶ディスプレイ（ＬＣＤ：Liquid Cristal Display）、ＥＬ（electroluminescence）ディスプレイ、ＣＲＴ（Cathode Ray Tube）ディスプレイである。 The display 12 displays an image including a moving image so that a viewer (user of the terminal 1) can visually recognize the display. Specific examples of the display 12 are a liquid crystal display (LCD), an EL (electroluminescence) display, and a CRT (Cathode Ray Tube) display.
　ディスプレイ１２に表示される動画は、通信端末１に搭載されたカメラ（不図示）によって撮影された映像であってもよい。また、この動画は、通信端末１に内蔵されたメモリ（e.g. 光ディスク、ハードディスク、フラッシュメモリ）又は無線通信部１１を介してアクセス可能な外部装置から取得された符号化動画データ（e.g. MPEG-2データ、MPEG-4データ）の再生画像であってもよい。 The moving image displayed on the display 12 may be an image taken by a camera (not shown) mounted on the communication terminal 1. The moving image is encoded moving image data (eg MPEG-2 data) acquired from a memory (eg optical disc, hard disk, flash memory) built in the communication terminal 1 or an external device accessible via the wireless communication unit 11. , MPEG-4 data).
　入力デバイス１３は、ユーザによる操作を受け付けるデバイスである。入力デバイス１３は、視聴者（端末１の利用者）の手によって操作されるポインティングデバイス、視聴者の音声を集音するマイクロフォン、及び視聴者の視線によって操作されるポインティングデバイスのうち少なくとも１つを含む。また、視聴者の手によって操作されるポインティングデバイスの具体例は、タッチパネル、タッチパッド、及びマウスを含む。 The input device 13 is a device that accepts user operations. The input device 13 includes at least one of a pointing device operated by a viewer (user of the terminal 1), a microphone that collects the viewer's voice, and a pointing device operated by the viewer's line of sight. Including. Specific examples of the pointing device operated by the viewer's hand include a touch panel, a touch pad, and a mouse.
　指定制御部１４は、ディスプレイ１２に表示されている動画に映った被写体の属性情報の検索のために、この動画に含まれる対象画像の指定を視聴者による入力デバイス１３の操作によって受け付ける。例えば、入力デバイス１３がタッチパネルである場合、視聴者は、ディスプレイ１２に所望の被写体が表示されているタイミングにおいて、被写体の表示範囲を指で囲むように（e.g. 円を描くように）タッチパネルを触ることによって、被写体の指定を行えばよい。また、視聴者は、被写体の指定のために、被写体の一点をタッチする操作を行なってもよい。この場合、指定制御部１４は、視聴者によってタッチされた一点を含む領域に対して顔認識などの画像認識処理を行うことによって、被写体が含まれる画像領域を特定すればよい。 The designation control unit 14 accepts designation of the target image included in the moving image by operating the input device 13 by the viewer in order to search for the attribute information of the subject shown in the moving image displayed on the display 12. For example, when the input device 13 is a touch panel, the viewer touches the touch panel so as to surround the display range of the subject with a finger (eg, draw an ellipse circle) at the timing when the desired subject is displayed on the display 12. Thus, the subject may be specified. In addition, the viewer may perform an operation of touching one point of the subject in order to specify the subject. In this case, the designation control unit 14 may identify an image region including the subject by performing image recognition processing such as face recognition on the region including one point touched by the viewer.
　入力デバイス１３がその他のポインティングデバイスである場合、視聴者は、被写体の表示範囲をポインタで囲むように（e.g. 円を描くように）、ポインティングデバイスを操作すればよい。また、視聴者は、被写体の指定のために、被写体の一点をポインタで指定する操作を行なってもよい。 When the input device 13 is another pointing device, the viewer may operate the pointing device so as to enclose the display range of the subject with a pointer (e.g., draw an ellipse). In addition, the viewer may perform an operation of designating one point of the subject with a pointer in order to designate the subject.
　また、入力デバイス１３がマイクロフォンを含む場合、視聴者は被写体を特定するための単語又はフレーズ（e.g. 男性、女性、犬、猫、花、自動車など）をマイクロフォンに入力してもよい。この場合、後述する検索制御部１５は、視聴者が単語又はフレーズを入力したタイミングによって対象画像を特定してもよいし、視聴者による他の入力デバイス（e.g. タッチパネル、マウス、操作ボタン等）の操作によって対象画像を特定してもよい。このとき、対象画像は、部分画像ではなく一画面全体の画面画像であってもよい。検索制御部１５は、画面画像としての対象画像と、視聴者が入力した単語又はフレーズとを検索システム９に送信すればよい。検索システム９は、視聴者が入力した単語又はフレーズに対応する被写体を対象画像の中から認識し、この認識した被写体像を用いて類似画像検索を行えばよい。 In addition, when the input device 13 includes a microphone, the viewer may input a word or a phrase (e.g. male, female, dog, cat, flower, car, etc.) for specifying the subject to the microphone. In this case, the search control unit 15 to be described later may specify the target image based on the timing at which the viewer inputs a word or phrase, or other input devices (eg, touch panel, mouse, operation buttons, etc.) by the viewer. The target image may be specified by an operation. At this time, the target image may be a screen image of the entire screen, not a partial image. The search control unit 15 may transmit the target image as the screen image and the word or phrase input by the viewer to the search system 9. The search system 9 may recognize a subject corresponding to the word or phrase input by the viewer from the target image, and perform a similar image search using the recognized subject image.
　さらに、指定制御部１４は、視聴者による対象画像の指定操作の実行に要する遅延時間を補償するために、視聴者による操作に基づいて定まる基準時点より前にディスプレイ１２に表示されていた画像を対象画像として決定する。ここで、基準時点は、視聴者による操作の完了時点以前である。言い換えると、指定制御部１４は、少なくとも操作の完了時点より前にディスプレイ１２に表示されていた画像を対象画像として決定する。基準時点は、例えば、対象画像の指定操作の完了時点、当該指定操作の開始時点、又は当該指定操作に要した期間の中央の時点とすればよい。 Furthermore, in order to compensate for the delay time required for the viewer to perform the target image designation operation, the designation control unit 14 selects an image displayed on the display 12 before the reference time determined based on the operation by the viewer. Determine as the target image. Here, the reference time is before the time when the operation by the viewer is completed. In other words, the designation control unit 14 determines an image displayed on the display 12 at least before the completion of the operation as a target image. The reference time point may be, for example, the completion time of the target image designation operation, the start time of the designation operation, or the central time point of the period required for the designation operation.
　視聴者による操作の基準時点（e.g. 操作の完了時点）よりどの程度前の画像を対象画像とするかは、静的に定めてもよいし、視聴者に応じて又は動画に応じて変更してもよい。静的に定める場合には、人の平均的な反応速度を考慮して、基準時点から遡る期間（以下、シフト時間と呼ぶ）を決定すればよい。 The extent to which an image before the reference point of operation by the viewer (eg) is determined as the target image may be determined statically or may be changed according to the viewer or according to the video. Also good. When statically determined, a period (hereinafter referred to as a shift time) that goes back from the reference time point may be determined in consideration of the average reaction speed of the person.
　一方、視聴者に応じてシフト時間を動的に定める場合には、視聴者の反応速度を測定するキャリブレーションを行なってもよい。具体的には、テスト用の動画を表示し、テスト用の動画に特定の被写体が映し出された後に視聴者がその被写体の表示範囲を選択する操作を完了するまでに要する時間を計測し、計測された時間の長さに応じてシフト時間を決定すればよい。また、人の平均的な反応速度に基づいたシフト時間の初期値を視聴者が自由に変更できるようにしてもよい。例えば、視聴者が端末１を操作することによって、シフト時間が長すぎること（過去に遡り過ぎであること）及びシフト時間が短すぎること（遡りが不十分であること）を補正できるようにしてもよい。視聴者が対象画像を指定する操作の完了までに要する時間は、視聴者の年齢などの影響によって個々に異なると想定される。したがって、シフト時間を視聴者に応じて変更することで、視聴者が意図した対象画像をより適切に特定することができる。 On the other hand, when the shift time is dynamically determined according to the viewer, calibration for measuring the reaction speed of the viewer may be performed. Specifically, a test video is displayed, and after a specific subject appears in the test video, the time required for the viewer to complete the operation to select the display range of the subject is measured and measured. What is necessary is just to determine shift time according to the length of performed time. Further, the viewer may be able to freely change the initial value of the shift time based on the average reaction speed of the person. For example, when the viewer operates the terminal 1, it is possible to correct that the shift time is too long (going back too far in the past) and the shift time is too short (poor going back). Also good. It is assumed that the time required for the viewer to complete the operation of specifying the target image is different depending on the influence of the viewer's age and the like. Therefore, the target image intended by the viewer can be more appropriately identified by changing the shift time according to the viewer.
　また、動画に応じてシフト時間を動的に定める場合には、動画に映った被写体の動きの速さに応じてシフト時間を変更するとよい。動画に応じてシフト時間を変更する具体例については、他の実施の形態（実施の形態３）において詳しく説明する。 Also, when the shift time is dynamically determined according to the moving image, the shift time may be changed according to the speed of movement of the subject reflected in the moving image. A specific example of changing the shift time according to the moving image will be described in detail in another embodiment (Embodiment 3).
　検索制御部１５は、指定制御部１４により特定された対象画像又はその代替画像を、無線通信部１１を介して検索システム９に送信する。そして、検索制御部１５は、対象画像又はその代替画像に基づいて検索された属性情報を、無線通信部１１を介して検索システム９から受信する。ここで代替画像は、指定制御部１４により特定された対象画像と実質的に同一の被写体を含むが、対象画像よりも画質が優れる等の理由によって類似画像検索に適した画像である。代替画像を利用する例については、他の実施の形態（実施の形態４）において詳細に説明する。 The search control unit 15 transmits the target image specified by the designation control unit 14 or a substitute image thereof to the search system 9 via the wireless communication unit 11. Then, the search control unit 15 receives the attribute information searched based on the target image or its substitute image from the search system 9 via the wireless communication unit 11. Here, the substitute image is an image suitable for similar image search because it includes a subject that is substantially the same as the target image specified by the designation control unit 14 but has a higher image quality than the target image. An example of using an alternative image will be described in detail in another embodiment (Embodiment 4).
　図３は、本実施の形態に係る通信端末１による情報検索方法の具体例を示すフローチャートである。ステップＳ１では、通信端末１は、ディスプレイ１２に動画を表示する。ステップＳ２では、通信端末１は、対象画像を指定するための視聴者による入力デバイス１３の操作を受け付ける。ステップＳ３では、通信端末１は、視聴者による操作の基準時点より前に表示された画像、言い換えると、少なくとも視聴者による操作の完了時点より前にディスプレイ１２に表示されていた画像、を対象画像として決定する。ステップＳ４では、通信端末１は、決定された対象画像又はその代替画像を検索システム９に送信する。最後に、ステップＳ５では、通信端末１は、検索システム９から属性情報を受信する。 FIG. 3 is a flowchart showing a specific example of the information search method by the communication terminal 1 according to the present embodiment. In step S <b> 1, the communication terminal 1 displays a moving image on the display 12. In step S <b> 2, the communication terminal 1 accepts an operation of the input device 13 by a viewer for designating a target image. In step S3, the communication terminal 1 selects an image displayed before the reference time point of the operation by the viewer, in other words, an image displayed on the display 12 at least before the time point when the operation by the viewer is completed. Determine as. In step S <b> 4, the communication terminal 1 transmits the determined target image or a substitute image thereof to the search system 9. Finally, in step S5, the communication terminal 1 receives attribute information from the search system 9.
　続いて以下では、シフト時間を用いて対象画像の特定を行うことの利点について、図４Ａ～４Ｄ並びに図５を用いて説明する。なお、図４Ａ～４Ｄ並びに図５は、一例として、入力デバイス１３がタッチパネルである場合について示している。視聴者は、被写体の表示範囲を指で囲むようにタッチパネルを指で触ることによって、部分画像としての対象画像を指定する。 Subsequently, in the following, the advantage of specifying the target image using the shift time will be described with reference to FIGS. 4A to 4D and FIG. 4A to 4D and FIG. 5 show a case where the input device 13 is a touch panel as an example. The viewer designates a target image as a partial image by touching the touch panel with a finger so as to surround the display range of the subject with the finger.
　図４Ａ～４Ｃの各々は、被写体４０１が映されたディスプレイ１２の一画面の画像４０を示している。ここでは、被写体４０１は、人物である。ディスプレイ１２の表示は、時間の経過にそって、図４Ａ、図４Ｂ、図４Ｃの順序で変化するものとする。つまり、図４Ａ～４Ｃにて白抜き矢印で示されているように、被写体４０１は、画面の右方向に移動する。 4A to 4C each show an image 40 of one screen of the display 12 on which the subject 401 is projected. Here, the subject 401 is a person. The display on the display 12 is assumed to change in the order of FIGS. 4A, 4B, and 4C over time. In other words, the subject 401 moves to the right of the screen as indicated by the white arrow in FIGS. 4A to 4C.
　視聴者は、図４Ａのタイミングにおいて被写体４０１の属性情報の検索を決意し、図４Ｂのタイミングにおいて被写体４０１の顔部分を囲むように選択する操作を開始する。図４Ｂの操作軌跡４０２は、視聴者が指で触った軌跡を示している。そして、図４Ｃの操作軌跡４０３は、視聴者が操作を完了した時点の操作軌跡を示している。しかしながら、動画の表示内容は刻々と変化する。したがって、図４Ｃの時点では、被写体４０１は、画面右方向に移動しており、視聴者による操作軌跡４０３によって囲まれた範囲内に所望の被写体４０１の像は存在していない。 The viewer decides to search the attribute information of the subject 401 at the timing of FIG. 4A, and starts an operation of selecting to surround the face portion of the subject 401 at the timing of FIG. 4B. An operation locus 402 in FIG. 4B indicates a locus that the viewer touches with a finger. And the operation locus | trajectory 403 of FIG. 4C has shown the operation locus at the time of a viewer completing operation. However, the display content of the moving image changes every moment. Therefore, at the time of FIG. 4C, the subject 401 has moved in the right direction on the screen, and the desired subject 401 image does not exist within the range surrounded by the operation locus 403 by the viewer.
　このような視聴者の操作の実行に要する遅延時間を補償するため、本実施の形態に係る通信端末１は、例えば、操作の完了時点を基準時点とし、基準時点よりも予め定められたシフト時間（又は、視聴者若しくは動画に応じて決定されたシフト時間）だけ前の画像を対象画像として採用する。例えば、シフト時間によって操作の開始時点まで遡ることによって、図４Ｄに示す画像４３のように、被写体４０１を正しく選択することが可能となる。 In order to compensate for the delay time required for execution of the viewer's operation, the communication terminal 1 according to the present embodiment uses, for example, the completion time of the operation as a reference time, and a shift time that is set in advance from the reference time The previous image (or the shift time determined according to the viewer or the moving image) is adopted as the target image. For example, the subject 401 can be correctly selected as in the image 43 shown in FIG. 4D by going back to the operation start time by the shift time.
　なお、図４Ａ～４Ｄの例は、視聴者が操作を開始してから操作を完了するまでの遅延時間の補償について説明した。しかしながら、視聴者が検索の実施を決意してから、そのための操作を開始するまでにも遅延時間が存在する。例えば、視聴者が動画中の被写体を検索しようとしてから実際に入力デバイス１３を操作するまでの遅延時間の間に動画の表示内容が変化し、被写体が表示されなくなることが想定される。この問題に対処するためには、例えば、視聴者の操作の開始時点を基準時点とし、視聴者の操作の開始時点からシフト時間だけ前の表示画像を対象画像として選択すればよい。これにより、視聴者が検索の実施を決意した時点に近い表示画像を対象画像として選択することができる。 In the examples of FIGS. 4A to 4D, the compensation of the delay time from when the viewer starts the operation until the operation is completed has been described. However, there is a delay time from when the viewer decides to perform the search to when the operation for that purpose is started. For example, it is assumed that the display content of the moving image changes during the delay time from when the viewer tries to search for the subject in the moving image until the input device 13 is actually operated, and the subject is not displayed. In order to cope with this problem, for example, the start time of the viewer's operation may be set as the reference time, and the display image that is the shift time before the start time of the viewer's operation may be selected as the target image. Thereby, a display image close to the time when the viewer decides to perform the search can be selected as the target image.
　図４Ａ～４Ｄに示した対象画像の指定手順は一例に過ぎない。対象画像（被写体）の指定を容易にするため、又は対象画像（被写体）を指定する操作に要する遅延時間を短くするためには、対象画像（被写体）を指定する操作をより簡易化できるとよい。例えば、通信端末１（指定制御部１４）は、動画中に表示された被写体を自動認識し、自動認識された被写体の中から選択対象とすべき被写体の指定を視聴者の操作によって受け付けてもよい。人物を特定する場合、図５に示すように、通信端末１は、被写体４０１が映されたディスプレイ１２の一画面の画像４０に対して顔認識処理を行い、検出された人物の顔を含む顔領域を示す枠４０４を画像４０に重ねて表示してもよい。そして、通信端末１は、例えば、入力デバイス１３を用いて、枠４０４を指定する視聴者の操作を受け付ければよい。例えば、入力デバイス１３がタッチパネルである場合、視聴者が枠４０４又はその内側の領域をタッチする操作を、対象画像の指定操作として用いればよい。なお、画像４０に複数の被写体（e.g. 複数の人物）が表示されている場合、通信端末１は、検出された複数の被写体領域（e.g. 顔領域）を枠４０４等によって表示すればよい。つまり、通信端末１が、被写体の自動認識機能を用いて対象画像の候補を提示することによって、対象画像（被写体）を指定する操作をより簡易化できるため、対象画像（被写体）を指定する操作に要する遅延時間を短くできる。 The target image designation procedure shown in FIGS. 4A to 4D is merely an example. In order to facilitate the designation of the target image (subject) or to shorten the delay time required for the operation of designating the target image (subject), it is preferable to simplify the operation of specifying the target image (subject). . For example, the communication terminal 1 (designation control unit 14) automatically recognizes the subject displayed in the moving image, and may accept the designation of the subject to be selected from the automatically recognized subjects by the operation of the viewer. Good. When specifying a person, as shown in FIG. 5, the communication terminal 1 performs face recognition processing on the image 40 of one screen of the display 12 on which the subject 401 is projected, and includes a face including the detected person's face. A frame 404 indicating a region may be displayed over the image 40. And the communication terminal 1 should just receive the operation of the viewer who designates the frame 404 using the input device 13, for example. For example, when the input device 13 is a touch panel, an operation in which the viewer touches the frame 404 or an area inside the frame 404 may be used as the target image specifying operation. When a plurality of subjects (e.g.ga plurality of persons) are displayed in the image 40, the communication terminal 1 may display the detected plurality of subject regions (e.g. a face region) with a frame 404 or the like. That is, the operation of specifying the target image (subject) can be simplified because the communication terminal 1 can further simplify the operation of specifying the target image (subject) by using the subject automatic recognition function to present the target image candidate. Can be shortened.
　しかしながら、図５に示したような被写体の自動認識機能を用いることで、視聴者の操作に要する時間が短縮された場合であっても、視聴者が検索の実施を決意してから、そのための操作を行うまでの遅延時間は少なくとも存在する。したがって、視聴者の操作に関する基準時点（e.g. 操作の開始時点、中央時点、又は完了時点）からシフト時間だけ前の表示画像を対象画像として選択すればよい。ここでのシフト時間は、視聴者が検索の実施を決意してから、そのための操作を行うまでの遅延時間を考慮して決定すればよい。これにより、視聴者が検索の実施を決意した時点に近い表示画像を対象画像として選択することができる。 However, even if the time required for the viewer's operation is shortened by using the subject automatic recognition function as shown in FIG. There is at least a delay time until the operation is performed. Therefore, a display image that is a shift time before the reference time point (e.g. start time, central time point, or completion time) of the viewer's operation may be selected as the target image. The shift time here may be determined in consideration of the delay time from when the viewer decides to perform the search to when the operation is performed. Thereby, a display image close to the time when the viewer decides to perform the search can be selected as the target image.
　動画の表示画面が刻々と変化するために、動画中の被写体（対象画像）の選択操作を行うことが難しいという問題は、あらゆる動画に関して発生し得ることは勿論である。したがって、本実施の形態に係る通信端末１は、ディスプレイ１２に表示される動画の種別に依らず有効である。しかしながら、動画中の被写体の指定に関する上述の問題は、視聴者がテレビ放送を見ている場合に特に影響が大きい。なぜなら、テレビ放送は、一般的に、視聴者自身の意思で自由に再生を一時停止したり、巻き戻したりすることができないためである。したがって、本実施の形態に係る通信端末１は、ディスプレイ１２に表示されている動画がテレビ放送である場合に特に有効である。 Of course, the problem that it is difficult to select an object (target image) in a moving picture because the display screen of the moving picture changes every moment can occur in any moving picture. Therefore, the communication terminal 1 according to the present embodiment is effective regardless of the type of moving image displayed on the display 12. However, the above-described problem relating to the designation of the subject in the moving image has a particularly large effect when the viewer is watching a television broadcast. This is because, in general, television broadcasting cannot be freely paused or rewound at the viewer's own will. Therefore, the communication terminal 1 according to the present embodiment is particularly effective when the moving image displayed on the display 12 is a television broadcast.
　ところで、本実施の形態で説明した指定制御部１４及び検索制御部１５により行われる処理は、ＡＳＩＣ（Application Specific Integrated Circuit）を含む半導体処理装置を用いて実現してもよい。また、これらの処理は、マイクロプロセッサ、ＤＳＰ（Digital Signal Processor）等のコンピュータにプログラムを実行させることによって実現してもよい。図１～４を用いて説明したアルゴリズムをコンピュータに行わせるための命令群を含む１又は複数のプログラムを作成し、当該プログラムをコンピュータに供給すればよい。 By the way, the processing performed by the designation control unit 14 and the search control unit 15 described in the present embodiment may be realized using a semiconductor processing apparatus including an ASIC (Application Specific Integrated Circuit). These processes may be realized by causing a computer such as a microprocessor or a DSP (Digital Signal Processor) to execute a program. One or a plurality of programs including a group of instructions for causing the computer to execute the algorithm described with reference to FIGS. 1 to 4 may be created and the programs may be supplied to the computer.
　このプログラムは、様々なタイプの非一時的なコンピュータ可読媒体（non-transitory computer readable medium）を用いて格納され、コンピュータに供給することができる。非一時的なコンピュータ可読媒体は、様々なタイプの実体のある記録媒体（tangible storage medium）を含む。非一時的なコンピュータ可読媒体の例は、磁気記録媒体（例えばフレキシブルディスク、磁気テープ、ハードディスクドライブ）、光磁気記録媒体（例えば光磁気ディスク）、ＣＤ－ＲＯＭ（Read Only Memory）、ＣＤ－Ｒ、ＣＤ－Ｒ／Ｗ、半導体メモリ（例えば、マスクＲＯＭ、ＰＲＯＭ（Programmable ROM）、ＥＰＲＯＭ（Erasable PROM）、フラッシュＲＯＭ、ＲＡＭ（random access memory））を含む。また、プログラムは、様々なタイプの一時的なコンピュータ可読媒体（transitory computer readable medium）によってコンピュータに供給されてもよい。一時的なコンピュータ可読媒体の例は、電気信号、光信号、及び電磁波を含む。一時的なコンピュータ可読媒体は、電線及び光ファイバ等の有線通信路、又は無線通信路を介して、プログラムをコンピュータに供給できる。 This program can be stored using various types of non-transitory computer readable media and supplied to a computer. Non-transitory computer readable media include various types of tangible storage media (tangible storage medium). Examples of non-transitory computer-readable media include magnetic recording media (eg flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (eg magneto-optical discs), CD-ROMs (Read Only Memory), CD-Rs, CD-R / W, semiconductor memory (for example, mask ROM, PROM (Programmable ROM), EPROM (Erasable ROM), flash ROM, RAM (random access memory)) are included. The program may also be supplied to the computer by various types of temporary computer-readable media. Examples of transitory computer readable media include electrical signals, optical signals, and electromagnetic waves. The temporary computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire and an optical fiber, or a wireless communication path.
　通信端末１がコンピュータシステムを用いて製造される場合の構成例を図６に示す。ＬＣＤ１２１は、ディスプレイ１２の具体例である。タッチパネル１３１及びマイク１３２は、入力デバイス１３の具体例である。不揮発性記憶部１０２（例えばフラッシュメモリ、又はハードディスクドライブ）に格納されたオペレーティングシステム（ＯＳ：Operating System）１０３及び検索アプリケーションプログラム１０４は、ＲＡＭ（Random Access Memory）１０１にロードされる。ＭＰＵ（Micro Processing Unit）１００がＲＡＭ１０１にロードされたＯＳ１０３及び検索アプリケーションプログラム１０４を実行することにより、指定制御部１４及び検索制御部１５の機能が実現される。 FIG. 6 shows a configuration example when the communication terminal 1 is manufactured using a computer system. The LCD 121 is a specific example of the display 12. The touch panel 131 and the microphone 132 are specific examples of the input device 13. An operating system (OS) 103 and a search application program 104 stored in a nonvolatile storage unit 102 (for example, a flash memory or a hard disk drive) are loaded into a RAM (Random Access Memory) 101. The MPU (Micro Processing Unit) 100 executes the OS 103 and the search application program 104 loaded in the RAM 101, thereby realizing the functions of the designation control unit 14 and the search control unit 15.
＜発明の実施の形態２＞
　本実施の形態では上述した発明の実施の形態１にかかる通信端末１の第１の変形例について説明する。本実施の形態では、通信端末１は、視聴者の操作に関する基準時点（e.g. 操作開始時点、又は操作完了時点）より前の第１の期間内における動画のシーン変化の有無を判定する。そして、端末１は、シーン変化があった場合に、シーン変化より前の画像を対象画像として決定する。なお、シーン変化の有無は、基準時点に係る画像と基準時点より前の画像を比較することによって判定すればよい。具体的に述べると、２つの画像間において、画素毎又は複数の画素を含む画素ブロック毎の画素値の変化の大きさを計算し、画素値の変化量が所定の基準を超える場合にシーン変化有りと判定すればよい。 <Embodiment 2 of the Invention>
In the present embodiment, a first modification of the communication terminal 1 according to the first embodiment of the invention described above will be described. In the present embodiment, the communication terminal 1 determines the presence / absence of a scene change of a moving image within a first period before a reference time (eg operation start time or operation completion time) related to a viewer's operation. Then, when there is a scene change, the terminal 1 determines an image before the scene change as a target image. The presence or absence of a scene change may be determined by comparing an image related to the reference time point with an image before the reference time point. Specifically, between two images, the magnitude of change in pixel value for each pixel or pixel block including a plurality of pixels is calculated, and the scene change occurs when the amount of change in pixel value exceeds a predetermined reference What is necessary is just to determine with existence.
　動画内での被写体の動きが小さい場合、視聴者は比較的容易に被写体を指定することができる。しかしながら、テレビ放送や映画においては、シーンが不連続に切り替わることがある。したがって、視聴者が被写体の検索を決意してからそのための操作を開始するまでの間にちょうどシーン変化が発生すると、視聴者は、被写体を検索する機会を失ってしまうおそれがある。しかしながら、本実施の形態では、シーン変化の有無を判定し、シーン変化があった場合に、シーン変化より前の画像を対象画像として決定する。このため、視聴者が検索を求める対象画像を適切に選択することができる。 When the movement of the subject in the video is small, the viewer can specify the subject relatively easily. However, in television broadcasting and movies, scenes sometimes switch discontinuously. Therefore, if a scene change occurs between the time when the viewer decides to search for a subject and the time when the operation is started, the viewer may lose the opportunity to search for the subject. However, in the present embodiment, the presence / absence of a scene change is determined, and when there is a scene change, an image before the scene change is determined as a target image. For this reason, the viewer can appropriately select a target image for which a search is desired.
　図７は、本実施の形態に係る通信端末１による情報検索方法の具体例を示すフローチャートである。図７中のステップＳ１、Ｓ２、Ｓ４、及びＳ５における処理は、図３に示された同一符号のステップとそれぞれ同様である。したがって、これらのステップに関する重複説明は省略する。 FIG. 7 is a flowchart showing a specific example of the information search method by the communication terminal 1 according to the present embodiment. The processes in steps S1, S2, S4, and S5 in FIG. 7 are the same as the steps with the same reference numerals shown in FIG. Therefore, the redundant description regarding these steps is omitted.
　図７のステップＳ３１～Ｓ３４は、図３のステップＳ３の変形例を示している。ステップＳ３１では、通信端末１（指定制御部１４）は、視聴者による操作の基準時点（e.g. 操作の開始時点）より前の所定期間内におけるシーン変化の有無を判定する。なお、発明の実施の形態１で述べたことから理解されるように、ステップＳ３１の所定期間は、静的に決定されてもよいし、視聴者又は動画に応じて動的に決定されてもよい。シーン変化があった場合（ステップＳ３２でＹＥＳ）、通信端末１（指定制御部１４）は、シーン変化の前の画像を対象画像として決定する（ステップＳ３３）。一方、シーン変化が検出されなかった場合（ステップＳ３２でＮＯ）、通信端末１（指定制御部１４）は、視聴者による操作の基準時点の画像を対象画像として選択してもよいし、それより前の画像を選択してもよい（ステップＳ３４）。 Steps S31 to S34 in FIG. 7 show a modification of step S3 in FIG. In step S31, the communication terminal 1 (designation control unit 14) determines whether or not there is a scene change within a predetermined period before the reference time point (e.g. start time of the e.g. operation) by the viewer. As can be understood from the description in the first embodiment of the present invention, the predetermined period of step S31 may be determined statically or may be dynamically determined according to the viewer or the moving image. Good. When there is a scene change (YES in step S32), the communication terminal 1 (designation control unit 14) determines an image before the scene change as a target image (step S33). On the other hand, when the scene change is not detected (NO in step S32), the communication terminal 1 (designation control unit 14) may select the image at the reference time point of the operation by the viewer as the target image. The previous image may be selected (step S34).
＜発明の実施の形態３＞
　本実施の形態では上述した発明の実施の形態１にかかる通信端末１の第２の変形例について説明する。本実施の形態では、通信端末１は、動画に含まれる被写体の動きの速さに応じて、シフト時間を変更する。言い換えると、本実施の形態に係る通信端末１は、動画に含まれる複数の画像の間における動きベクトルの大きさに応じて、シフト時間を変更する。 <Third Embodiment of the Invention>
In the present embodiment, a second modification of the communication terminal 1 according to the first embodiment of the invention described above will be described. In the present embodiment, the communication terminal 1 changes the shift time according to the speed of movement of the subject included in the moving image. In other words, the communication terminal 1 according to the present embodiment changes the shift time according to the magnitude of the motion vector between a plurality of images included in the moving image.
　被写体の動きの速さを判定するためには、動画に含まれる複数の画像の間の動きベクトルを計算し、動きベクトルの大きさを求めればよい。例えば、動画に映った被写体の動きが早いほど、つまり動きベクトルが大きいほど、シフト時間が相対的に長くなるようにしてもよい。被写体の動きが早いほど、視聴者の操作に戸惑いが生じたり、正確に被写体を選択しようとするあまりに操作の完了まで時間を要したりすることが想定される。したがって、動画に映った被写体の動きが早いほど、つまり動きベクトルが大きいほど、シフト時間を大きくすることで、視聴者が意図した対象画像をより適切に特定することができる。 In order to determine the speed of movement of a subject, a motion vector between a plurality of images included in a moving image may be calculated to determine the magnitude of the motion vector. For example, the shift time may be relatively longer as the movement of the subject shown in the moving image is faster, that is, as the motion vector is larger. It is assumed that the faster the movement of the subject, the more confusing the viewer's operation is, or the longer it takes to complete the operation to accurately select the subject. Therefore, the target image intended by the viewer can be more appropriately identified by increasing the shift time as the motion of the subject shown in the moving image is faster, that is, as the motion vector is larger.
　図８は、本実施の形態に係る通信端末１による情報検索方法の具体例を示すフローチャートである。図８中のステップＳ１、Ｓ２、Ｓ４、及びＳ５における処理は、図３に示された同一符号のステップとそれぞれ同様である。したがって、これらのステップに関する重複説明は省略する。 FIG. 8 is a flowchart showing a specific example of the information search method by the communication terminal 1 according to the present embodiment. The processes in steps S1, S2, S4, and S5 in FIG. 8 are the same as the steps with the same reference numerals shown in FIG. Therefore, the redundant description regarding these steps is omitted.
　図８のステップＳ３５～Ｓ３７は、図３のステップＳ３の変形例を示している。ステップＳ３５では、通信端末１（指定制御部１４）は、動画中の複数の画像間における動きベクトルを計算する。ステップＳ３６では、通信端末１（指定制御部１４）は、計算された動きベクトルの大きさに応じてシフト時間を決定する。ステップＳ３７では、通信端末１（指定制御部１４）は、視聴者による操作の基準時点よりシフト時間だけ前の表示画像を対象画像として決定する。 Steps S35 to S37 in FIG. 8 show a modification of step S3 in FIG. In step S35, the communication terminal 1 (designation control unit 14) calculates a motion vector between a plurality of images in the moving image. In step S36, the communication terminal 1 (designation control unit 14) determines the shift time according to the calculated magnitude of the motion vector. In step S37, the communication terminal 1 (designation control unit 14) determines, as a target image, a display image that is a shift time before the reference time point of the operation by the viewer.
＜発明の実施の形態４＞
　本実施の形態では上述した発明の実施の形態１にかかる通信端末１の第３の変形例について説明する。具体的には、本実施の形態は、指定制御部１４により決定された対象画像に対応する代替画像を検索システム９に送信する例を示す。 <Embodiment 4 of the Invention>
In the present embodiment, a third modification of the communication terminal 1 according to the first embodiment of the invention described above will be described. Specifically, the present embodiment shows an example in which a substitute image corresponding to the target image determined by the designation control unit 14 is transmitted to the search system 9.
　図９は、本実施の形態に係る通信端末４の構成例を示すブロック図である。通信端末４は、外部のテレビ受信機等に表示されているテレビ放送の画面を、端末４に搭載されたカメラ１６１によって撮影することにより得られる映像をディスプレイ１２に表示する。 FIG. 9 is a block diagram showing a configuration example of the communication terminal 4 according to the present embodiment. The communication terminal 4 displays on the display 12 an image obtained by photographing a television broadcast screen displayed on an external television receiver or the like with the camera 161 mounted on the terminal 4.
　指定制御部１４は、上述した実施の形態１～３に記載されたいずれかの方法に従って、ディスプレイ１２に表示された動画（つまり、カメラ１６１によって撮影されたテレビ放送画面を含む映像）から対象画像を決定する。 The designation control unit 14 applies the target image from the moving image (that is, the video including the TV broadcast screen shot by the camera 161) displayed on the display 12 according to any of the methods described in the first to third embodiments. To decide.
　検索制御部４５は、指定制御部１４により決定された対象画像に対応する代替画像を取得する。具体的に述べると、検索制御部４５は、カメラ１６１で撮影された後にディスプレイ１２に表示された対象画像と実質的に同一時刻におけるテレビ放送の画像を、端末１に搭載されたテレビチューナー１６２を用いることによって代替画像として取得する。 The search control unit 45 acquires a substitute image corresponding to the target image determined by the designation control unit 14. More specifically, the search control unit 45 uses the TV tuner 162 mounted on the terminal 1 to display a TV broadcast image at substantially the same time as the target image displayed on the display 12 after being captured by the camera 161. It is acquired as a substitute image by using.
　本実施の形態は、視聴者が一般的なテレビ放送受信機を用いてテレビ番組を視聴しているときに、その出演者の氏名などの属性情報を検索するケースを想定している。なお、カメラ１６１で撮影された後にディスプレイ１２に表示された動画に対して、上述した実施の形態１～３で述べたいずれかの情報検索方法を用いることによっても、視聴者は、所望の属性情報を取得できることは勿論である。しかしながら、カメラ１６１で撮影されたテレビ放送画面は、端末４に搭載されたテレビチューナー１６２によって得られる映像よりも画質が劣る場合がある。例えば、テレビ放送受信機の画面を遠方から撮影している場合、又は撮影時に手ぶれが生じた場合に画質が劣化する。本実施の形態では、テレビチューナー１６２によって得られた映像を代替画像として用いるため、対象画像よりも画質が優れている代替画像を類似画像検索のために使用できる利点がある。 This embodiment assumes a case where the viewer searches for attribute information such as the name of the performer when the viewer is watching a television program using a general television broadcast receiver. Note that the viewer can also obtain a desired attribute by using any of the information retrieval methods described in Embodiments 1 to 3 described above for the moving image captured by the camera 161 and displayed on the display 12. Of course, information can be acquired. However, the television broadcast screen shot by the camera 161 may be inferior in image quality to the video obtained by the television tuner 162 mounted on the terminal 4. For example, image quality deteriorates when the screen of a television broadcast receiver is taken from a distance or when camera shake occurs during shooting. In the present embodiment, since the video obtained by the TV tuner 162 is used as a substitute image, there is an advantage that a substitute image having higher image quality than the target image can be used for similar image search.
　なお、代替画像を決定するためには、カメラ１６１により撮影されたテレビ放送画面のチャンネルを識別する必要がある。例えば、検索制御部４５は、カメラ１６１の画像とテレビチューナー１６２の画像とを比較することによって、若しくはマイクロフォン（不図示）によって取得された音声とテレビチューナー１６２の音声とを比較することによって、又はこれらの組み合わせによって、カメラ１６１により撮影されているテレビ放送画面とテレビチューナー１６２の映像とのチャンネルの一致を判定すればよい。また、検索制御部４５は、カメラ１６１によって撮影されたテレビ放送画面に含まれているチャンネル情報を文字検出してもよい。さらにまた、これらの自動的なチャンネルの識別に代えて、視聴者が端末４を操作することによってチャンネルを指定してもよい。 In order to determine a substitute image, it is necessary to identify a channel of a television broadcast screen shot by the camera 161. For example, the search control unit 45 compares the image of the camera 161 with the image of the TV tuner 162, or compares the sound acquired by the microphone (not shown) with the sound of the TV tuner 162, or Based on these combinations, it is only necessary to determine whether the TV broadcast screen shot by the camera 161 matches the channel of the TV tuner 162 video. Further, the search control unit 45 may detect characters in channel information included in the television broadcast screen shot by the camera 161. Furthermore, instead of these automatic channel identifications, the viewer may designate a channel by operating the terminal 4.
　図１０は、本実施の形態に係る通信端末４による情報検索方法の具体例を示すフローチャートである。図１０中のステップＳ２、Ｓ３、及びＳ５における処理は、図３に示された同一符号のステップとそれぞれ同様である。したがって、これらのステップに関する重複説明は省略する。 FIG. 10 is a flowchart showing a specific example of the information search method by the communication terminal 4 according to the present embodiment. The processes in steps S2, S3, and S5 in FIG. 10 are the same as the steps with the same reference numerals shown in FIG. Therefore, the redundant description regarding these steps is omitted.
　ステップＳ１１では、通信端末４は、テレビ放送画面をカメラ１６１で撮影して得られる動画をディスプレイ１２に表示する。ステップＳ４１では、通信端末４は、ステップＳ３で決定された対象画像に関する代替画像を、テレビチューナー１６２によって得られる映像から取得する。ステップＳ４２では、通信端末４は、代替画像を検索システム９に送信する。 In step S11, the communication terminal 4 displays on the display 12 a moving image obtained by photographing the TV broadcast screen with the camera 161. In step S <b> 41, the communication terminal 4 acquires a substitute image related to the target image determined in step S <b> 3 from the video obtained by the television tuner 162. In step S <b> 42, the communication terminal 4 transmits a substitute image to the search system 9.
　さらに、視聴者がカメラ１６１を用いてテレビ放送画面を撮影する場合、テレビ放送画面をカメラ１６１の画枠にぴったりと合わせて撮影できるとよい。しかしながら、被写体検索のためにテレビ放送画面を急いで撮影する場合を考慮すると、テレビ放送画面をカメラ１６１の画枠にぴったりと合わせて撮影することは容易ではない。また、視聴者の利便性を考えると、テレビ放送画面の撮影をもっと容易に行えることが望ましい。したがって、通信端末４は、カメラ１６１による撮影画像に映ったテレビ放送画面を検出してもよい。テレビ放送画面の検出を行う具体例について図１１を用いて説明する。 Furthermore, when a viewer uses the camera 161 to shoot a television broadcast screen, it is preferable that the viewer can shoot the television broadcast screen so that it fits exactly on the image frame of the camera 161. However, in consideration of the case where the TV broadcast screen is shot in a hurry for the subject search, it is not easy to shoot the TV broadcast screen so as to fit the image frame of the camera 161. Also, considering the convenience of the viewer, it is desirable that the TV broadcast screen can be captured more easily. Therefore, the communication terminal 4 may detect a television broadcast screen reflected in an image captured by the camera 161. A specific example of detecting a television broadcast screen will be described with reference to FIG.
　図１１は、図１０のステップＳ４１の具体例を示すフローチャートである。ステップＳ４１１では、検索制御部４５は、カメラ１６１による撮影画像の中からテレビ放送画面を検出する。具体的に述べると、検索制御部４５は、カメラ１６１による撮影画像の中からテレビ放送受信機の四角形のフレームを検知すればよい。 FIG. 11 is a flowchart showing a specific example of step S41 in FIG. In step S411, the search control unit 45 detects a television broadcast screen from images captured by the camera 161. More specifically, the search control unit 45 may detect a rectangular frame of the television broadcast receiver from the image captured by the camera 161.
　ステップＳ４１２では、検索制御部４５は、カメラ１６１による撮影画像内におけるテレビ放送画面の位置及び大きさに基づいて、視聴者（端末４の利用者）により選択された部分画像を特定する。 In step S412, the search control unit 45 identifies the partial image selected by the viewer (user of the terminal 4) based on the position and size of the television broadcast screen in the image captured by the camera 161.
　ステップＳ４１３では、検索制御部４５は、視聴者により選択された部分画像に対応するテレビチューナー１６２による映像を代替画像として決定する。 In step S413, the search control unit 45 determines a video by the TV tuner 162 corresponding to the partial image selected by the viewer as a substitute image.
＜発明の実施の形態５＞
　本実施の形態では上述した発明の実施の形態１にかかる通信端末１の第４の変形例について説明する。図１２は、本実施の形態に係る通信端末５の構成例を示すブロック図である。通信端末５は、録画制御部１６を有する。録画制御部１６は、テレビチューナー１６２を用いることにより、検索システム９から取得された属性情報に関連するテレビ放送番組を自動的に録画する。 <Embodiment 5 of the Invention>
In the present embodiment, a fourth modification of the communication terminal 1 according to the first embodiment of the invention described above will be described. FIG. 12 is a block diagram illustrating a configuration example of the communication terminal 5 according to the present embodiment. The communication terminal 5 has a recording control unit 16. The recording control unit 16 uses the television tuner 162 to automatically record the television broadcast program related to the attribute information acquired from the search system 9.
　図１３は、本実施の形態に係る通信端末５による情報検索方法の具体例を示すフローチャートである。図１３中のステップＳ１～Ｓ５における処理は、図３に示された同一符号のステップとそれぞれ同様である。したがって、これらのステップに関する重複説明は省略する。 FIG. 13 is a flowchart showing a specific example of the information search method by the communication terminal 5 according to the present embodiment. The processes in steps S1 to S5 in FIG. 13 are the same as the steps with the same reference numerals shown in FIG. Therefore, the redundant description regarding these steps is omitted.
　図１３のステップＳ６では、通信端末５（録画制御部１６）は、検索システム９から取得された属性情報に関連するテレビ放送番組を自動的に録画する。例えば、録画制御部１６は、属性情報に含まれる被写体の人名に対応する人物が出演するテレビ番組を自動的に録画すれよい。録画制御部１６は、テレビ番組表を取得したり、テレビ番組の出演者情報を保持するサーバ（例えば、World Wide Webサーバ）にアクセスしたりすることによって、属性情報に含まれる人名に対応する人物が出演するテレビ番組を決定すればよい。 In step S6 of FIG. 13, the communication terminal 5 (recording control unit 16) automatically records a television broadcast program related to the attribute information acquired from the search system 9. For example, the recording control unit 16 may automatically record a television program in which a person corresponding to the subject name included in the attribute information appears. The recording control unit 16 acquires a TV program guide or accesses a server (for example, a World Wide Web server) that holds TV program performer information, so that the person corresponding to the person name included in the attribute information The TV program that appears in
　本実施の形態によれば、通信端末５は、テレビ番組を視聴中の視聴者がその番組の出演者の画像を指定して画像検索を指示したことに応じて、その出演者が今後出演する他のテレビ番組を簡単に自動的に予約することができる。つまり、視聴者は、出演者の名前を知らない場合であっても、その出演者が出演する他の番組の録画予約を行うことができる。
＜その他の実施の形態＞ According to the present embodiment, the communication terminal 5 will appear in the future in response to a viewer who is watching a television program specifying an image of a performer of the program and instructing an image search. Other TV programs can be easily reserved automatically. That is, even when the viewer does not know the name of the performer, the viewer can make a recording reservation for another program in which the performer appears.
<Other embodiments>
　上述した発明の実施の形態１～５では、ディスプレイ１２、入力デバイス１３、カメラ１６１、及びテレビチューナー１６２が、プロセッサ１０と共に、通信端末１、４、及び５に搭載されている例について説明した。しかしながら、これらのデバイスは、プロセッサ１０と結合されていればよく、一体の通信端末として構成される必要はない。例えば、これらのデバイス及びプロセッサ１は、無線ＬＡＮ又はＢｌｕｅｔｏｏｔｈ（登録商標）などの無線通信機能、若しくは有線通信機能を用いて通信可能であればよい。 In the above-described first to fifth embodiments of the invention, the example in which the display 12, the input device 13, the camera 161, and the television tuner 162 are mounted on the communication terminals 1, 4, and 5 together with the processor 10 has been described. However, these devices only need to be combined with the processor 10 and need not be configured as an integrated communication terminal. For example, these devices and the processor 1 only need to be able to communicate using a wireless communication function such as a wireless LAN or Bluetooth (registered trademark) or a wired communication function.
　さらに、本発明は上述した実施の形態のみに限定されるものではなく、既に述べた本発明の要旨を逸脱しない範囲において種々の変更が可能であることは勿論である。 Furthermore, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present invention already described.
　この出願は、２０１１年１２月８日に出願された日本出願特願２０１１－２６８９９４を基礎とする優先権を主張し、その開示の全てをここに取り込む。 This application claims priority based on Japanese Patent Application No. 2011-268994 filed on Dec. 8, 2011, the entire disclosure of which is incorporated herein.
１、４、５　通信端末
１０　プロセッサ
１１　無線通信部
１２　ディスプレイ
１３　入力デバイス
１４　指定制御部
１５　検索制御部
１６　録画制御部
４０～４３　画像
４５　検索制御部
８　ネットワーク
９　検索システム
１００　ＭＰＵ（Micro Processing Unit）
１０１　ＲＡＭ（Random Access Memory）
１０２　不揮発性記憶部
１０３　オペレーティングシステム
１０４　検索アプリケーションプログラム
１２１　ＬＣＤ（Liquid Crystal Display）
１３１　タッチパネル
１３２　マイク
１６１　カメラ
１６２　テレビチューナー
４０１　被写体
４０２、４０３　操作軌跡
４０４　顔領域を示す枠 1, 4, 5 Communication terminal 10 Processor 11 Wireless communication unit 12 Display 13 Input device 14 Designation control unit 15 Search control unit 16 Recording control unit 40 to 43 Image 45 Search control unit 8 Network 9 Search system 100 MPU (Micro Processing Unit)
101 RAM (Random Access Memory)
102 Non-volatile storage unit 103 Operating system 104 Search application program 121 LCD (Liquid Crystal Display)
131 Touch Panel 132 Microphone 161 Camera 162 Television Tuner 401 Subject 402, 403 Operation Trajectory 404 Frame Showing Face Area

Claims (38)

  1. 　ディスプレイに表示されている動画に映った被写体の属性情報の検索のために、前記動画に含まれる対象画像の指定を利用者による入力デバイスの操作によって受け付ける指定制御手段と、
    　前記対象画像又はその代替画像を検索システムに送信するとともに、前記対象画像又は前記代替画像に基づいて検索された属性情報を前記検索システムから受信する検索制御手段と、
    を備え、
    　前記指定制御手段は、前記操作の実行に要する遅延時間を補償するために、前記操作の完了時点以前である前記操作の基準時点よりも前に前記ディスプレイに表示されていた画像を前記対象画像として決定する、
    情報処理装置。     A designation control means for accepting designation of a target image included in the video by an operation of an input device by a user for searching for attribute information of a subject reflected in the video displayed on the display;
    Search control means for transmitting the target image or a substitute image thereof to a search system and receiving attribute information searched based on the target image or the substitute image from the search system;
    With
    The designation control means uses, as the target image, an image displayed on the display before a reference time point of the operation, which is before the completion time of the operation, in order to compensate for a delay time required for execution of the operation. decide,
    Information processing device.
  2. 　前記指定制御手段は、前記基準時点より前の第１の期間内における前記動画のシーン変化の有無を判定するとともに、前記シーン変化があった場合に前記シーン変化より前の画像を前記対象画像として決定する、請求項１に記載の情報処理装置。 The designation control means determines whether or not there is a scene change of the moving image within a first period before the reference time point, and when there is the scene change, an image before the scene change is set as the target image. The information processing device according to claim 1, wherein the information processing device is determined.
  3. 　前記指定制御手段は、前記基準時点に係る画像と前記基準時点より前の画像を比較することによって、前記シーン変化の有無を判定する、請求項２に記載の情報処理装置。 3. The information processing apparatus according to claim 2, wherein the designation control unit determines the presence or absence of the scene change by comparing an image related to the reference time point with an image before the reference time point.
  4. 　前記指定制御手段は、前記基準時点より第１の期間だけ前に表示されていた画像を前記対象画像として決定する、請求項１に記載の情報処理装置。 2. The information processing apparatus according to claim 1, wherein the designation control unit determines an image that has been displayed for a first period before the reference time as the target image.
  5. 　前記指定制御手段は、前記第１の期間を変更可能である、請求項２～４のいずれか１項に記載の情報処理装置。 5. The information processing apparatus according to claim 2, wherein the designation control unit can change the first period.
  6. 　前記指定制御手段は、前記利用者の反応時間を測定するためのキャリブレーションを実施し、前記反応時間に応じて前記第１の期間を決定する、請求項５に記載の情報処理装置。 6. The information processing apparatus according to claim 5, wherein the designation control unit performs calibration for measuring the reaction time of the user and determines the first period according to the reaction time.
  7. 　前記指定制御手段は、前記動画に含まれる複数の画像の間における動きベクトルの大きさに応じて、前記第１の期間を決定する、請求項５に記載の情報処理装置。 The information processing apparatus according to claim 5, wherein the designation control unit determines the first period according to a magnitude of a motion vector between a plurality of images included in the moving image.
  8. 　前記指定制御手段は、前記動きベクトルの大きさが大きい程、前記第１の期間が相対的に長くなるように決定する、請求項７に記載の情報処理装置。 The information processing apparatus according to claim 7, wherein the designation control unit determines that the first period is relatively longer as the magnitude of the motion vector is larger.
  9. 　前記基準時点は、前記操作の完了時点、前記操作の開始時点、又は前記操作に要した期間の中央の時点である、請求項１～８のいずれか１項に記載の情報処理装置。 The information processing apparatus according to any one of claims 1 to 8, wherein the reference time point is a time point when the operation is completed, a time point when the operation is started, or a center time point of the period required for the operation.
  10. 　前記動画は、前記情報処理装置に結合されたカメラによって撮影される映像である、
    請求項１～９のいずれか１項に記載の情報処理装置。     The moving image is a video shot by a camera coupled to the information processing apparatus.
    The information processing apparatus according to any one of claims 1 to 9.
  11. 　前記動画は、テレビ放送の表示画面を前記カメラによって撮影することにより得られる映像である、請求項１０に記載の情報処理装置。 The information processing apparatus according to claim 10, wherein the moving image is a video obtained by shooting a display screen of a television broadcast with the camera.
  12. 　前記検索制御手段は、
    　前記カメラで撮影された後に前記ディスプレイに表示された前記対象画像と実質的に同一時刻におけるテレビ放送の画像を、前記情報処理装置に結合されたテレビチューナーを用いることにより前記代替画像として取得し、
    　前記代替画像を前記検索システムに送信する、
    請求項１１に記載の情報処理装置。     The search control means includes
    An image of a television broadcast at substantially the same time as the target image displayed on the display after being captured by the camera is acquired as the substitute image by using a television tuner coupled to the information processing device,
    Sending the substitute image to the search system;
    The information processing apparatus according to claim 11.
  13. 　前記検索制御手段は、
    　前記カメラによる撮影画像から前記テレビ放送の表示画面を検出し、
    　前記撮影画像内における前記表示画面の大きさ及び位置に基づいて、前記テレビチューナーによる映像の中から前記代替画像を取得する、
    請求項１２に記載の情報処理装置。     The search control means includes
    Detecting the TV broadcast display screen from the image taken by the camera,
    Acquiring the substitute image from the video by the television tuner based on the size and position of the display screen in the captured image;
    The information processing apparatus according to claim 12.
  14. 　前記対象画像は、前記動画に含まれる一画面の画像の部分画像である、請求項１～１３のいずれか１項に記載の情報処理装置。 14. The information processing apparatus according to claim 1, wherein the target image is a partial image of a one-screen image included in the moving image.
  15. 　前記指定制御手段は、前記動画に含まれる画像から被写体を自動認識するとともに、認識された被写体を示す表示を前記ディスプレイに出力し、
    　前記操作は、前記被写体を示す表示を指定することを含む、請求項１４に記載の情報処理装置。     The designation control means automatically recognizes a subject from an image included in the video, and outputs a display indicating the recognized subject to the display.
    The information processing apparatus according to claim 14, wherein the operation includes designating a display indicating the subject.
  16. 　前記入力デバイスは、前記ディスプレイの前面に配置されたタッチパネルを含み、
    　前記操作は、前記ディスプレイに所望の被写体が表示されている前記タッチパネル上の範囲を前記利用者が指で囲むように触ることを含む、
    請求項１４に記載の情報処理装置。     The input device includes a touch panel disposed in front of the display,
    The operation includes touching the user so as to surround the range on the touch panel where a desired subject is displayed on the display with a finger.
    The information processing apparatus according to claim 14.
  17. 　前記入力デバイスは、前記利用者の音声を集音するマイクロフォンを含み、
    　前記検索制御手段は、前記部分画像としての前記対象画像を含む前記一画面の画像と、前記マクロフォンによって入力された前記被写体を指定するための音声情報を前記検索システムに送信する、
    請求項１４に記載の情報処理装置。     The input device includes a microphone that collects the voice of the user;
    The search control means transmits the one-screen image including the target image as the partial image and audio information for designating the subject input by the macrophone to the search system.
    The information processing apparatus according to claim 14.
  18. 　前記情報処理装置に結合されたテレビチューナーを用いることにより、前記検索システムから取得した前記属性情報に関連するテレビ放送番組を自動的に録画する録画制御手段をさらに備える、請求項１～１７のいずれか１項に記載の情報処理装置。 The recording control means for automatically recording a television broadcast program related to the attribute information acquired from the search system by using a television tuner coupled to the information processing apparatus. The information processing apparatus according to claim 1.
  19. 　請求項１～１７のいずれか１項に記載の情報処理装置と、
    　前記ディスプレイと、
    　前記入力デバイスと、
    　前記検索制御手段による前記対象画像の送信、及び前記属性情報の受信のために使用される通信手段と、
    を備える通信端末。     An information processing apparatus according to any one of claims 1 to 17,
    The display;
    The input device;
    Communication means used for transmission of the target image by the search control means and reception of the attribute information;
    A communication terminal comprising:
  20. 　前記入力デバイスは、前記利用者の手によって操作されるポインティングデバイス、前記利用者の音声を集音するマイクロフォン、及び前記利用者の視線によって操作されるポインティングデバイスのうち少なくとも１つを含む、請求項１９に記載の通信端末。 The input device includes at least one of a pointing device operated by the user's hand, a microphone that collects the user's voice, and a pointing device operated by the user's line of sight. 19. The communication terminal according to 19.
  21. 　情報処理装置による情報検索方法であって、
    　ディスプレイに表示されている動画に映った被写体の属性情報の検索のために、前記動画に含まれる対象画像の指定を利用者による入力デバイスの操作によって受け付けること、
    　前記対象画像又はその代替画像を検索システムに送信すること、及び
    　前記対象画像又は前記代替画像に基づいて検索された属性情報を前記検索システムから受信すること、
    を備え、
    　前記受け付けることは、前記操作の実行に要する遅延時間を補償するために、前記操作の完了時点以前である前記操作の基準時点よりも前に前記ディスプレイに表示されていた画像を前記対象画像として決定することを含む、
    情報検索方法。     An information search method by an information processing device,
    Accepting designation of a target image included in the video by an operation of an input device by a user for searching for attribute information of a subject reflected in the video displayed on the display;
    Transmitting the target image or a substitute image thereof to a search system; and receiving attribute information searched based on the target image or the substitute image from the search system;
    With
    The accepting determines, as the target image, an image that has been displayed on the display before a reference time point of the operation before the completion time of the operation in order to compensate for a delay time required for execution of the operation. Including
    Information retrieval method.
  22. 　前記対象画像として決定することは、前記基準時点より前の第１の期間内における前記動画のシーン変化の有無を判定するとともに、前記シーン変化があった場合に前記シーン変化より前の画像を前記対象画像として決定することを含む、請求項２１に記載の情報検索方法。 Determining as the target image determines whether or not there is a scene change in the moving image within a first period before the reference time, and if there is a scene change, an image before the scene change is determined as the target image. The information search method according to claim 21, comprising determining as a target image.
  23. 　前記対象画像として決定することは、前記基準時点に係る画像と前記基準時点より前の画像を比較することによって、前記シーン変化の有無を判定することを含む、請求項２２に記載の情報検索方法。 The information search method according to claim 22, wherein determining the target image includes determining whether or not the scene has changed by comparing an image related to the reference time point with an image before the reference time point. .
  24. 　前記対象画像として決定することは、前記基準時点より第１の期間だけ前に表示されていた画像を前記対象画像として決定することを含む、請求項２１に記載の情報検索方法。 The information search method according to claim 21, wherein the determination as the target image includes determining an image that has been displayed only a first period before the reference time point as the target image.
  25. 　前記第１の期間を変更することをさらに備える、請求項２２～２４のいずれか１項に記載の情報検索方法。 The information search method according to any one of claims 22 to 24, further comprising changing the first period.
  26. 　前記第１の期間を変更することは、前記利用者の反応時間を測定するためのキャリブレーションを実施し、前記反応時間に応じて前記第１の期間を決定することを含む、請求項２５に記載の情報検索方法。 26. The method according to claim 25, wherein changing the first period includes performing calibration for measuring a reaction time of the user and determining the first period according to the reaction time. Information retrieval method described.
  27. 　前記第１の期間を変更することは、前記動画に含まれる複数の画像の間における動きベクトルの大きさに応じて、前記第１の期間を決定することを含む、請求項２５に記載の情報検索方法。 The information according to claim 25, wherein changing the first period includes determining the first period according to a magnitude of a motion vector between a plurality of images included in the moving image. retrieval method.
  28. 　前記第１の期間は、前記動きベクトルの大きさが大きい程、相対的に長くなるように決定される、請求項２７に記載の情報検索方法。 28. The information search method according to claim 27, wherein the first period is determined to be relatively longer as the magnitude of the motion vector is larger.
  29. 　前記基準時点は、前記操作の完了時点、前記操作の開始時点、又は前記操作に要した期間の中央の時点である、請求項２１～２８のいずれか１項に記載の情報検索方法。 The information search method according to any one of claims 21 to 28, wherein the reference time point is a time point when the operation is completed, a time point when the operation is started, or a center time point of the period required for the operation.
  30. 　前記動画は、前記情報処理装置に結合されたカメラによって撮影される映像である、
    請求項２１～２９のいずれか１項に記載の情報検索方法。     The moving image is a video shot by a camera coupled to the information processing apparatus.
    The information search method according to any one of claims 21 to 29.
  31. 　前記動画は、テレビ放送の表示画面を前記カメラによって撮影することにより得られる映像である、請求項３０に記載の情報検索方法。 The information retrieval method according to claim 30, wherein the moving image is a video obtained by shooting a display screen of a television broadcast with the camera.
  32. 　前記送信することは、
    　前記カメラで撮影された後に前記ディスプレイに表示された前記対象画像と実質的に同一時刻におけるテレビ放送の画像を、前記情報処理装置に結合されたテレビチューナーを用いることにより前記代替画像として取得すること、及び
    　前記代替画像を前記検索システムに送信すること、
    を含む、請求項３１に記載の情報検索方法。     The sending is
    Acquiring a television broadcast image at substantially the same time as the target image displayed on the display after being captured by the camera as the substitute image by using a television tuner coupled to the information processing apparatus. Sending the substitute image to the search system;
    The information search method according to claim 31, comprising:
  33. 　前記取得することは、
    　前記カメラによる撮影画像から前記テレビ放送の表示画面を検出すること、及び
    　前記撮影画像内における前記表示画面の大きさ及び位置に基づいて、前記テレビチューナーによる映像の中から前記代替画像を取得すること、
    を含む、請求項３２に記載の情報検索方法。     The obtaining is
    Detecting a display screen of the television broadcast from a photographed image by the camera, and acquiring the substitute image from a video by the television tuner based on a size and a position of the display screen in the photographed image. ,
    The information search method according to claim 32, comprising:
  34. 　前記対象画像は、前記動画に含まれる一画面の画像の部分画像である、請求項２１～３３のいずれか１項に記載の情報検索方法。 The information search method according to any one of claims 21 to 33, wherein the target image is a partial image of a one-screen image included in the moving image.
  35. 　前記動画に含まれる画像から被写体を自動認識するとともに、認識された被写体を示す表示を前記ディスプレイに出力することをさらに備え、
    　前記操作は、前記被写体を示す表示を指定することを含む、請求項３４に記載の情報検索方法。     Automatically recognizing a subject from an image included in the video, and further outputting a display indicating the recognized subject to the display,
    The information search method according to claim 34, wherein the operation includes designating a display indicating the subject.
  36. 　前記入力デバイスは、前記ディスプレイの前面に配置されたタッチパネルを含み、
    　前記操作は、前記ディスプレイに所望の被写体が表示されている前記タッチパネル上の範囲を前記利用者が指で囲むように触ることを含む、
    請求項３４に記載の情報検索方法。     The input device includes a touch panel disposed in front of the display,
    The operation includes touching the user so as to surround the range on the touch panel where a desired subject is displayed on the display with a finger.
    The information search method according to claim 34.
  37. 　前記情報処理装置に結合されたテレビチューナーを用いることにより、前記検索システムから取得した前記属性情報に関連するテレビ放送番組を自動的に録画することをさらに備える、請求項２１～３６のいずれか１項に記載の情報検索方法。 The television broadcast program related to the attribute information acquired from the search system is further recorded automatically by using a television tuner coupled to the information processing device, further comprising any one of claims 21 to 36. The information search method described in the section.
  38. 　情報検索方法をコンピュータに行わせるためのプログラムを格納した非一時的なコンピュータ可読媒体であって、
    　前記方法は、
    　ディスプレイに表示されている動画に映った被写体の属性情報の検索のために、前記動画に含まれる対象画像の指定を利用者による入力デバイスの操作によって受け付けること、
    　前記対象画像又はその代替画像を検索システムに送信すること、及び
    　前記対象画像又は前記代替画像に基づいて検索された属性情報を前記検索システムから受信すること、
    を含み、
    　前記受け付けることは、前記操作の実行に要する遅延時間を補償するために、前記操作の完了時点以前である前記操作の基準時点よりも前に前記ディスプレイに表示されていた画像を前記対象画像として決定することを含む、
    非一時的なコンピュータ可読媒体。     A non-transitory computer-readable medium storing a program for causing a computer to perform an information retrieval method,
    The method
    Accepting designation of a target image included in the video by an operation of an input device by a user for searching for attribute information of a subject reflected in the video displayed on the display;
    Transmitting the target image or a substitute image thereof to a search system; and receiving attribute information searched based on the target image or the substitute image from the search system;
    Including
    The accepting determines, as the target image, an image that has been displayed on the display before a reference time point of the operation before the completion time of the operation in order to compensate for a delay time required for execution of the operation. Including
    A non-transitory computer readable medium.
PCT/JP2012/007342 2011-12-08 2012-11-15 Information processing device, communication terminal, information search method, and non-temporary computer-readable medium WO2013084422A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2011-268994 2011-12-08
JP2011268994 2011-12-08

Publications (1)

Publication Number Publication Date
WO2013084422A1 true WO2013084422A1 (en) 2013-06-13

Family

ID=48573815

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2012/007342 WO2013084422A1 (en) 2011-12-08 2012-11-15 Information processing device, communication terminal, information search method, and non-temporary computer-readable medium

Country Status (2)

Country Link
JP (1) JPWO2013084422A1 (en)
WO (1) WO2013084422A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110944118A (en) * 2018-09-25 2020-03-31 富士施乐株式会社 Storage medium, image processing apparatus, and image processing method

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP6403368B2 (en) * 2013-09-13 2018-10-10 京セラ株式会社 Mobile terminal, image search program, and image search method

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH0855131A (en) * 1994-08-12 1996-02-27 Nippon Telegr & Teleph Corp <Ntt> Method and device for identifying object in dynamic image
JPH10187759A (en) * 1996-01-31 1998-07-21 Mitsubishi Electric Corp Moving picture anchor displaying, selecting, and stetting device, moving picture hypermedia device using moving picture anchor, and moving picture provision system
JP2000113208A (en) * 1998-09-30 2000-04-21 Toshiba Corp Information presenting method, information presenting device and recording medium
JP2000132563A (en) * 1998-10-26 2000-05-12 Omron Corp Image retrieval method, image processing method, information retrieval method, recording medium recording execution programs of the methods, image processor, and information retrieval system
JP2004054435A (en) * 2002-07-17 2004-02-19 Toshiba Corp Hypermedia information presentation method, hypermedia information presentation program and hypermedia information presentation device
JP2008276340A (en) * 2007-04-26 2008-11-13 Hitachi Ltd Retrieving device

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH0855131A (en) * 1994-08-12 1996-02-27 Nippon Telegr & Teleph Corp <Ntt> Method and device for identifying object in dynamic image
JPH10187759A (en) * 1996-01-31 1998-07-21 Mitsubishi Electric Corp Moving picture anchor displaying, selecting, and stetting device, moving picture hypermedia device using moving picture anchor, and moving picture provision system
JP2000113208A (en) * 1998-09-30 2000-04-21 Toshiba Corp Information presenting method, information presenting device and recording medium
JP2000132563A (en) * 1998-10-26 2000-05-12 Omron Corp Image retrieval method, image processing method, information retrieval method, recording medium recording execution programs of the methods, image processor, and information retrieval system
JP2004054435A (en) * 2002-07-17 2004-02-19 Toshiba Corp Hypermedia information presentation method, hypermedia information presentation program and hypermedia information presentation device
JP2008276340A (en) * 2007-04-26 2008-11-13 Hitachi Ltd Retrieving device

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110944118A (en) * 2018-09-25 2020-03-31 富士施乐株式会社 Storage medium, image processing apparatus, and image processing method
CN110944118B (en) * 2018-09-25 2024-01-26 富士胶片商业创新有限公司 Computer readable storage medium, image processing apparatus, and image processing method

Also Published As

Publication number Publication date
JPWO2013084422A1 (en) 2015-04-27

Similar Documents

Publication Publication Date Title
KR101680714B1 (en) Method for providing real-time video and device thereof as well as server, terminal device, program, and recording medium
KR102164481B1 (en) Appratus and method for tracking user viewing behavior using pattern matching and character recognition, system
JP2016538657A (en) Browse videos by searching for multiple user comments and overlaying content
CN106559712B (en) Video playing processing method and device and terminal equipment
KR101181588B1 (en) Image processing apparatus, image processing method, image processing system and recording medium
US20150341698A1 (en) Method and device for providing selection of video
TW202122989A (en) Information processing method and apparatus, electronic device and computer readable storage medium
US11545188B2 (en) Video processing method, video playing method, devices and storage medium
JP2016535351A (en) Video information sharing method, apparatus, program, and recording medium
JP6385429B2 (en) Method and apparatus for reproducing stream media data
US10728583B2 (en) Multimedia information playing method and system, standardized server and live broadcast terminal
US20070092220A1 (en) System for reproducing video
JP6999516B2 (en) Information processing equipment
US20140157294A1 (en) Content providing apparatus, content providing method, image displaying apparatus, and computer-readable recording medium
CN110719530A (en) Video playing method and device, electronic equipment and storage medium
CN113542610A (en) Shooting method, mobile terminal and storage medium
US20230401030A1 (en) Selecting options by uttered speech
JP2017501598A5 (en)
CN106254939B (en) Information prompting method and device
US20140036149A1 (en) Information processor and information processing method
KR20180043712A (en) Method for displaying an image and an electronic device thereof
WO2013084422A1 (en) Information processing device, communication terminal, information search method, and non-temporary computer-readable medium
US20140003656A1 (en) System of a data transmission and electrical apparatus
US20160360293A1 (en) Method and apparatus for playing 3d film sources in smart tv
CN108933881B (en) Video processing method and device

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 12855918

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2013548069

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 12855918

Country of ref document: EP

Kind code of ref document: A1