WO2024116400A1 - 探索支援装置、探索支援システム、探索支援方法、およびプログラム - Google Patents

探索支援装置、探索支援システム、探索支援方法、およびプログラム Download PDF

Info

Publication number
WO2024116400A1
WO2024116400A1 PCT/JP2022/044520 JP2022044520W WO2024116400A1 WO 2024116400 A1 WO2024116400 A1 WO 2024116400A1 JP 2022044520 W JP2022044520 W JP 2022044520W WO 2024116400 A1 WO2024116400 A1 WO 2024116400A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
search
search target
unit
captured
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2022/044520
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
範行 福井
明▲徳▼ 平
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Mitsubishi Electric Corp
Original Assignee
Mitsubishi Electric Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Mitsubishi Electric Corp filed Critical Mitsubishi Electric Corp
Priority to EP22967230.8A priority Critical patent/EP4529155B1/en
Priority to PCT/JP2022/044520 priority patent/WO2024116400A1/ja
Priority to KR1020247042908A priority patent/KR102901590B1/ko
Priority to JP2024557081A priority patent/JP7675945B2/ja
Priority to CN202280097856.8A priority patent/CN120226342A/zh
Publication of WO2024116400A1 publication Critical patent/WO2024116400A1/ja
Priority to US19/021,643 priority patent/US20250157177A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • H04N7/181—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—Two-dimensional [2D] image generation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/07—Target detection
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/10—Recognition assisted with metadata

Definitions

  • This disclosure relates to a search support device, a search support system, a search support method, and a program.
  • Patent Document 1 describes a method of photographing a subject using a multi-lens camera and applying a vector median filter to remove unwanted objects from an image in which text is partially obstructed.
  • Patent Document 2 describes a method of extracting a target person from an image and using images from multiple points to identify the person even if part of the person's image is missing due to an obstacle.
  • Non-Patent Document 1 describes a method of integrating images from a visible light camera and an invisible light camera to increase the visibility of the target object.
  • the above-mentioned technology uses images captured by a multi-lens camera, images taken at multiple locations, or multiple types of images to search for objects in remote locations, but there are cases where only images captured from a specific direction can be obtained due to limitations on the number of cameras, etc. in a remote location. If the object to be searched for is hidden by surrounding objects in an image captured from a specific direction, it may not be possible to detect the object to be searched for.
  • the present disclosure has been made to solve the problems described above, and aims to provide a search support device, a search support system, a search support method, and a program that can detect a search target object with a high probability even if the search target object is hidden by surrounding objects.
  • the search support device includes an image capture unit that captures captured video, an object information capture unit that captures object information indicating an object included in the captured video, a setting unit that sets a search target image including a search target object, a determination unit that uses the search target image or the object information to determine whether a part of a search target candidate is hidden by a surrounding object, and an image creation unit that creates a comparison image to be compared to search for the search target object using an image of the captured video that indicates the surrounding object when the determination unit determines that a part of the search target candidate is hidden by the surrounding object.
  • the search support method includes the steps of setting a search target image including a search target object, acquiring a captured image and object information indicating an object included in the captured image, determining whether or not a part of the search target candidate is hidden by a surrounding object using the search target image or the object information, and, if it is determined that a part of the search target candidate is hidden by a surrounding object, creating a comparison image to be compared in order to search for the search target object using an image of the captured image indicating the surrounding object.
  • the program according to the third aspect causes a computer mounted on the search support device to execute the steps of: setting a search target image including a search target object; acquiring a captured image and object information indicating an object included in the captured image; determining whether or not a part of the search target candidate is hidden by a surrounding object using the search target image or the object information; and, if it is determined that a part of the search target candidate is hidden by a surrounding object, creating a comparison image to be compared in order to search for the search target object using an image of the captured image indicating the surrounding object.
  • the search support system includes a search support device including an imaging device that generates an imaged video, an object information generating device that generates object information indicating an object included in the imaged video, a setting unit that sets a search target image including a search target object, a determination unit that uses the search target image or the object information to determine whether a part of the search target candidate is hidden by a surrounding object, and an image creation unit that creates a comparison image to be compared to search for the search target object using an image of the imaged video that indicates the surrounding object when the determination unit determines that a part of the search target candidate is hidden by the surrounding object.
  • FIG. 1 is a block diagram illustrating an example of a search support system according to a first embodiment.
  • 1 is a block diagram showing an example of a functional configuration of a search support device according to a first embodiment.
  • 4 is a diagram for explaining processing of a search support device in the first embodiment;
  • FIG. 1 is a flowchart illustrating an example of a processing procedure in a search support device according to a first embodiment.
  • FIG. 13 is a block diagram showing an example of a functional configuration of a search support device according to a second embodiment. 13 is a diagram for explaining processing of a search support device in a second embodiment.
  • FIG. 13 is a flowchart illustrating an example of a processing procedure in a search support device according to the second embodiment.
  • FIG. 13 is a block diagram showing an example of a functional configuration of a search support device according to a third embodiment.
  • FIG. 13 is a diagram for explaining processing by a search support device in a third embodiment.
  • 13 is a flowchart illustrating an example of a processing procedure in a search support device according to a third embodiment.
  • FIG. 13 is a diagram for explaining a modified example of the search support system.
  • FIG. 13 is a block diagram showing an example of a functional configuration of a search support device according to a fourth embodiment.
  • FIG. 13 is a diagram for explaining processing by a search support device in a fourth embodiment.
  • 23 is a flowchart illustrating an example of a processing procedure in a search support device according to the fourth embodiment.
  • FIG. 13 is a diagram for explaining a modified example of a search support device in the fourth embodiment.
  • FIG. 1 is a block diagram showing an example of a search support system according to a first embodiment.
  • the search support system includes, for example, a search support device 100, a camera system 200, and a searcher's terminal device 300.
  • the search support device 100 and the searcher's terminal device 300 are installed in, for example, a management room that manages search targets such as lost items in a search area.
  • the camera system 200 is placed in a search area in a remote location away from the search support device 100 and the searcher's terminal device 300.
  • the search support device 100 and the camera system 200 are connected via a network NW such as the Internet.
  • NW such as the Internet.
  • the search support device 100, the camera system 200, and the searcher's terminal device 300 are shown as separate devices, but this is not limiting.
  • a single information processing device may have the functions of the search support device 100 and the searcher's terminal device 300, and the camera system 200 may have the functions of the search support device 100.
  • the search support device 100 performs processing to support the search for a search target object.
  • the search support device 100 may be configured, for example, to be connected to multiple camera systems 200 and multiple searcher terminal devices 300, and to provide a cloud service that supports the search for a search target object.
  • the search target object may be, for example, a lost item in a store, but is not limited to this, and may also be a person such as a lost child.
  • the explorer's terminal device 300 is an information processing device such as a personal computer or smartphone that is operated by the explorer who is searching for the object to be searched.
  • the explorer's terminal device 300 is equipped with an operation interface that accepts operations by the explorer, a display interface that presents various information to the explorer, a storage device, and a control unit such as a CPU (Central Processing Unit) that performs various processes.
  • a CPU Central Processing Unit
  • the camera system 200 is installed in the search area.
  • the search area may be, for example, a shopping mall, a restaurant, or other store, but is not limited thereto.
  • the camera system 200 includes a plurality of camera devices 210, an object information generating device 216, and a control unit 220.
  • the object information generating device 216 may be, for example, an infrared camera device 212 and a radar device 214.
  • the camera device 210 generates an image of the search area.
  • the camera device 210 generates an image by irradiating visible light.
  • one or more camera devices 210 are installed in one search area. Each of the camera devices 210 generates an image of the search area from its respective installation location.
  • the camera device 210 generates a time-continuous image, but is not limited thereto, and may generate images that are not time-continuous.
  • the object information generating device 216 includes, for example, an infrared camera device 212 and a radar device 214, but is not limited thereto, and may have another configuration as long as it can generate object information as described below.
  • the object information generating device 216 acquires object information indicating objects contained in the captured image.
  • the infrared camera device 212 generates infrared images as object information by emitting infrared light.
  • the radar device 214 irradiates a search area with pulses of laser light and measures the reflected light to measure distance information to the object as object information.
  • the object information is information indicating the two-dimensional or three-dimensional shape of the search target object.
  • the object information is information indicating the two-dimensional or three-dimensional shape of the search target object or a group of surrounding objects consisting of one surrounding object or multiple surrounding objects that hide a part of the search target object.
  • the object information generating device 216 is an infrared camera device 212
  • the object information is information indicating a two-dimensional shape.
  • the object information is information indicating a three-dimensional shape obtained, for example, by shifting the transmission direction of a radar transmission signal vertically and horizontally.
  • the object information acquisition unit 108 of the search support device 100 described later acquires object information of the search target by separating the search target from the perspective image and the surrounding objects that hide the search target.
  • the object information acquiring unit 108 of the search support device 100 which will be described later, acquires object information about the search object by distinguishing between the search object and surrounding objects that hide the search object based on differences in distance information.
  • the control unit 220 controls the camera device 210, the infrared camera device 212, and the radar device 214.
  • the control unit 220 operates the camera device 210, the infrared camera device 212, and the radar device 214 in time synchronization. In this way, the control unit 220 acquires captured images and object information that are synchronized in time.
  • the control unit 220 transmits the captured images and object information that are synchronized in time to the search support device 100.
  • the control unit 220 may, for example, perform camera control such as changing the imaging direction and zooming the imaging range for the camera device 210, the infrared camera device 212, or the radar device 214 based on the operation of the searcher.
  • FIG. 2 is a block diagram showing an example of the functional configuration of the search support device 100 in the first embodiment.
  • FIG. 3 is a diagram for explaining the processing of the search support device 100 in the first embodiment.
  • the search support device 100 includes, for example, a search area side communication unit 102, a terminal side communication unit 104, a captured image acquisition unit 106, an object information acquisition unit 108, a setting unit 110, a conversion unit 112, a determination unit 114, an image creation unit 116, and a search unit 118.
  • the search area side communication unit 102 is a communication interface such as a NIC (Network Interface Card) or a wireless communication module for connecting to the network NW.
  • the terminal side communication unit 104 is a communication interface that communicates, for example, via a LAN line.
  • Each unit such as the captured image acquisition unit 106, the object information acquisition unit 108, the setting unit 110, the conversion unit 112, the determination unit 114, the image creation unit 116, and the search unit 118 is realized by, for example, a computer such as a CPU mounted on the search support device 100 executing a program stored in a program memory.
  • the captured image acquisition unit 106 acquires the captured image P10.
  • the captured image P10 is supplied via the search area side communication unit 102.
  • the captured image acquisition unit 106 stores, for example, search area information indicating the search area, camera information that uniquely identifies the camera device 210, and the captured image in a storage device (not shown) in association with each other.
  • the object information acquisition unit 108 acquires object information indicating objects included in the captured video P10.
  • the object information is supplied via the search area side communication unit 102.
  • the object information includes object information of the search target and object information of surrounding objects.
  • the object information acquisition unit 108 associates, for example, search area information indicating the search area, device information that uniquely identifies the infrared camera device 212 or the radar device 214, and object information, and stores them in a storage device (not shown).
  • the object information acquisition unit 108 may associate the acquired object information with the search area information or device information and store them.
  • the setting unit 110 sets a search target image P12 including a search target object input via the terminal side communication unit 104.
  • the search target image P12 is a still image viewed from a specific viewpoint provided by a person (search requester) requesting a search for a search target object, such as an image of a lost item or an image of the face of a lost child.
  • the setting unit 110 may obtain, as the search target image P12, an image uploaded to the search support device 100 from another device, such as the search requester's smartphone.
  • the conversion unit 112 converts the search target image P12 set by the setting unit 110 into a viewpoint conversion image P14 viewed from an arbitrary viewpoint.
  • the conversion unit 112 may generate one viewpoint conversion image P14 from the search target image P12, or may generate multiple viewpoint conversion images P14.
  • the conversion unit 112 may, for example, create a viewpoint conversion image P14 in which the viewpoint of the search target is converted by cutting out an area including the search target included in the search target image P12 and performing image processing such as angle correction, rotation, and size change on the area including the search target.
  • the conversion unit 112 may, for example, create a viewpoint conversion image P14 in which the viewpoint of the search target is converted by predicting a portion of the search target image P12 in which the search target does not appear and combining the predicted portion with an area including the search target included in the search target image P12.
  • the conversion unit 112 may, for example, predict the shape, etc. of the search target by searching a search site or the like for an object similar to the search target based on the product name information of the search target.
  • the determination unit 114 uses the search target image P12 or the object information to determine whether a part of the search target candidate is hidden by a surrounding object.
  • the search target candidate may be an object included in the captured video P10.
  • the determination unit 114 may compare the viewpoint converted image P14 obtained by converting the search target image P12 with the object included in the captured video P10 to detect an object included in the captured video P10 that matches one of the viewpoint converted images P14 as a search target candidate, and determine whether a part of the detected search target candidate is hidden by a surrounding object.
  • the determination unit 114 may determine whether a part of the object included in the captured video P10 is hidden by a surrounding object by determining whether a part of the object information is missing.
  • the search target candidate is, for example, an object included in the captured video P10 that matches at least one of the search target image P12 and one or more viewpoint converted images P14.
  • the image creation unit 116 creates a comparison image to be compared to search for the search target object using an image showing the surrounding object from the captured image P10.
  • the image creation unit 116 creates, for example, an overall image P16 that shows the entire search target candidate as a comparison image.
  • the image creation unit 116 interpolates the parts hidden by the surrounding objects by using, for example, an infrared image acquired by the infrared camera device 212 to see through the surrounding objects.
  • the image creation unit 116 identifies the locations where the surrounding objects are located using a distance image acquired by the radar device 214, and interpolates the locations where the surrounding objects are located. In this way, the image creation unit 116 creates the overall image P16.
  • the search unit 118 searches for the object to be searched for by comparing the overall image P16 created by the image creation unit 116 with the viewpoint conversion image P14 converted by the conversion unit 112.
  • the search unit 118 transmits the search results to the searcher's terminal device 300 via the terminal side communication unit 104. For example, when a search target candidate is found, the search unit 118 may transmit the captured image P10 including the search target candidate and position information of the search target candidate as the search results. This allows the searcher to visually recognize the search target by displaying the captured image P10 and to search for the search target by checking the image of the camera device 210 that captures the position of the search target candidate.
  • FIG. 4 is a flowchart showing an example of a processing procedure in the search support device 100 in the first embodiment.
  • the terminal communication unit 104 acquires the search target image P12 from the searcher's terminal device 300, and the setting unit 110 sets the search target image P12 (step S100).
  • the conversion unit 112 creates a viewpoint conversion image P14 from the search target image P12 (step S102).
  • the captured image acquisition unit 106 acquires the captured image P10 of the search area from the camera system 200, and the object information acquisition unit 108 acquires the object information (step S104).
  • the determination unit 114 determines whether or not an object (search target candidate) that matches the viewpoint conversion image P14 among the objects included in the captured image P10 has been detected (step S106).
  • the determination unit 114 may determine that a search target candidate that matches the viewpoint conversion image P14 has been detected when the search target candidate matches the viewpoint conversion image P14 with a degree of match equal to or greater than a predetermined reference value. That is, the determination unit 114 is not limited to determining whether the entire images match, but may determine whether parts of the images match.
  • the reference value is preferably a value that can determine that an object included in the captured video P10 and the viewpoint converted image P14 are the same object, but may be a value that is appropriately changed by the searcher. If the determination unit 114 does not detect an object that matches the viewpoint converted image P14 (step S106: NO), the process returns to step S104 and the search continues.
  • step S106 When the determination unit 114 detects a search target candidate that matches the viewpoint conversion image P14 (step S106: YES), it determines whether or not a part of the search target candidate is hidden by a peripheral object (step S108). When the determination unit 114 determines that a part of the search target candidate is not hidden by a peripheral object (step S108: NO), it notifies the search result (step S116).
  • the video creation unit 116 cuts out and extracts an image including an image showing a peripheral object (peripheral object image) (step S110), and creates an overall video P16 in which the entire search target candidate can be confirmed using the peripheral object image (step S112).
  • the video creation unit 116 recognizes the part of the image showing the search target candidate that is hidden by the peripheral object based on the peripheral object image, and uses the object information to interpolate the part of the image showing the search target candidate that is hidden by the peripheral object.
  • the search unit 118 determines whether the viewpoint converted image P14 matches the overall video P16 (step S114). If the search unit 118 does not detect a viewpoint converted image P14 that matches the overall video P16 (step S114: NO), the process returns to step S104. If the search unit 118 detects a viewpoint converted image P14 that matches the overall video P16 (step S114: YES), the search unit 118 notifies the search result (step S116). In step S116, the search unit 118 transmits, for example, an image showing the search target candidate that was the basis of the overall video P16, position information of the search target candidate, and the like, as the search result to the searcher's terminal device 300.
  • the captured image P10 and object information are acquired, and it is determined whether or not a part of the search target candidate is hidden by a surrounding object. If it is determined that a part of the search target candidate is hidden by a surrounding object, it is possible to create an overall image P16 representing the entire search target candidate using the object information. If the search target candidate is hidden by a surrounding object, there is a high possibility that the true search target cannot be searched for. In addition, if the number of camera devices 210 is limited, there is a high possibility that the true search target cannot be searched for.
  • the search support device 100 if the search target candidate detected based on the comparison between the captured image P10 and the viewpoint conversion image P14 is hidden by a surrounding object, it is possible to create the overall image P16 using the object information.
  • the search support device 100 can, for example, transmit the overall image P16 to the searcher's terminal device 300 and show the overall image P16 to the searcher to allow the searcher to search for whether the search target candidate is the true search target.
  • the search support device 100 of the first embodiment can detect the search target object with a high probability even if the search target object is hidden by surrounding objects.
  • the search support system includes an infrared camera device 212 or a radar device 214 to acquire object information, and can create an overall image P16 using the object information acquired by the infrared camera device 212 or the radar device 214.
  • the search support system and search support device 100 can create an overall image P16 by performing processing such as interpolating parts of the search target candidates that are hidden by surrounding objects.
  • the search support device 100 may convert the search target image P12 into a viewpoint converted image P14 seen from an arbitrary viewpoint, and compare the viewpoint converted image P14 with objects included in the captured video P10 to detect an object included in the captured video P10 that matches one of the viewpoint converted images P14 as a search target candidate.
  • the first embodiment of the search support device 100 includes a search unit 118 that searches for the search target object by comparing the entire video P16 with the viewpoint converted image P14, so that the search target object can be searched for even if it is hidden by surrounding objects.
  • FIG. 5 is a block diagram showing an example of the functional configuration of a search support device 100B in the second embodiment.
  • FIG. 6 is a diagram for explaining the processing of the search support device 100B in the second embodiment.
  • the search support device 100B in the second embodiment differs from the first embodiment in that it creates a composite image P20 by combining an image of a surrounding object that is hiding a part of a search target candidate with a viewpoint conversion image P14, and searches for a search target object by comparing the created composite image P20 with an object included in the captured video P10.
  • the following description will focus on the differences.
  • the image creation unit 116 includes a synthesis unit 120 that creates a synthetic image as a comparison image by synthesizing an image showing a surrounding object that is hiding part of a search target candidate with a converted image.
  • the synthetic image in the second embodiment corresponds to the comparison image that is compared to search for the search target object in the first embodiment.
  • the search support device 100B of the second embodiment includes a search unit 122 that searches for target candidates by comparing objects included in the captured image with the synthetic image created by the synthesis unit 120.
  • the synthesis unit 120 creates multiple viewpoint conversion images P14 from the search target image P12, and synthesizes the surrounding object images with each of the created viewpoint conversion images P14 to create multiple synthetic images P20, which are used as comparison images.
  • the synthetic image P20 may be an image or a video, for example, a still image, or a video including multiple images that are consecutive in time.
  • the synthesis unit 120 cuts out and extracts an image including a surrounding object image from the captured video P10 using object information of the surrounding objects, and creates the synthetic image P20 by superimposing the extracted surrounding object image on the viewpoint conversion image P14.
  • the search unit 118 compares the synthetic image P20 with the captured video P10. As a result, the search unit 122 can detect a search target candidate if at least one of the synthetic images P20 matches a search target candidate included in the captured video P10.
  • FIG. 7 is a flowchart showing an example of a processing procedure in the search support device 100B in the second embodiment.
  • the terminal side communication unit 104 and the setting unit 110 acquire the search target image P12 from the searcher's terminal device 300 (step S200).
  • the conversion unit 112 creates a viewpoint conversion image P14 from the search target image P12 (step S202).
  • the search area side communication unit 102 and the object information acquisition unit 108 acquire the captured image P10 of the search area and object information from the camera system 200 (step S204).
  • the determination unit 114 determines whether or not an object that matches the viewpoint conversion image P14 has been detected among the objects included in the captured image P10 (step S206). If the determination unit 114 does not detect an object that matches the viewpoint conversion image P14 (step S206: NO), the process returns to step S204 and the search continues.
  • the determination unit 114 detects an object (search target candidate) that matches the viewpoint conversion image P14 (step S206: YES), it determines whether the search target candidate is hidden by a surrounding object (step S208). When the search target candidate is not hidden by a surrounding object (step S208: NO), the determination unit 114 notifies the search result (step S216). When the search target candidate is hidden by a surrounding object (step S208: YES), the synthesis unit 120 cuts out and extracts an image including an image showing a surrounding object (surrounding object image) (step S210), and creates a synthetic video P20 by synthesizing the surrounding object image with the viewpoint conversion image P14 (step S212).
  • the search unit 122 determines whether the composite image P20 matches the captured video P10 (step S214). If the composite image P20 does not match the captured video P10 (step S214: NO), the search unit 122 returns the process to step S204. If the composite image P20 matches the captured video P10 (step S214: YES), the search unit 122 notifies the search result (step S216).
  • the search unit 122 transmits, for example, an image showing the search target candidate that matches the composite image P20, location information of the search target candidate, and the like, as the search result to the searcher's terminal device 300.
  • the second embodiment of the search support device 100B includes a synthesis unit 120 that creates a synthetic image P20 by synthesizing an image of a surrounding object that is hiding a part of a search target candidate with a viewpoint conversion image P14, and a search unit 122 that searches for the search target object by comparing an object included in the captured video P10 with the synthetic image P20 created by the synthesis unit 120.
  • the search support device 100B can detect the search target object with a high probability even if the search target object is hidden by surrounding objects.
  • FIG. 8 is a block diagram showing an example of the functional configuration of a search support device 100C in the third embodiment.
  • FIG. 9 is a diagram for explaining the processing of the search support device 100C in the third embodiment.
  • the search support device 100C in the third embodiment performs both the processing of comparing the entire video P16 with the viewpoint conversion image P14 as in the first embodiment, and the processing of comparing the composite image P20 with the captured video P10 as in the second embodiment. This allows the search support device 100C in the third embodiment to detect the search target object with an even higher probability.
  • the image creation unit 130 creates an overall image P16 in which the entire search target candidate can be confirmed from the captured image P10 and the object information, and further creates a composite image P20a by synthesizing the image of the surrounding object included in the captured image P10 with the viewpoint converted image P14.
  • the search unit 132 compares the overall image P16 with the viewpoint converted image P14 as in the first embodiment, and compares the composite image P20a with the captured image P10 as in the second embodiment.
  • the search unit 132 searches for the search target object based on the comparison result between the overall image P20a and the viewpoint converted image P14 and the comparison result between the composite image P20a and the captured image P10.
  • the search unit 132 may determine that the search target object has been found, for example, when both the similarity between the overall image P20 and the viewpoint converted image P14 and the similarity between the composite image P20a and the captured image P10 are equal to or higher than a reference.
  • the search unit 132 may notify that the search target object may have been found if, for example, either the similarity between the entire video P20 and the viewpoint conversion image P14 or the similarity between the composite image P20a and the captured video P10 is equal to or greater than a criterion. This allows the search support device 100C to search for the search target object with higher accuracy.
  • FIG. 10 is a flowchart showing an example of a processing procedure in the search support device 100C in the third embodiment.
  • the search support device 100C performs steps S200 to S206 as described above.
  • the determination unit 114 detects an object (search target candidate) that matches the viewpoint conversion image P14 (step S206: YES), it determines whether the search target candidate is hidden by a peripheral object (step S208). When the search target candidate is not hidden by a peripheral object (step S208: NO), the determination unit 114 notifies the search result (step S216).
  • the image creation unit 130 cuts out and extracts an image including an image showing a peripheral object (peripheral object image) (step S210), creates a composite image P20 by combining the peripheral object image with the viewpoint conversion image P14, and creates an overall image P16 using the peripheral object image (step S300).
  • the search unit 132 determines whether the composite image P20 matches the captured image P10, and whether the viewpoint converted image P14 matches the overall video P16 (step S302). If the composite image P20 does not match the captured image P10 and the viewpoint converted image P14 does not match the overall video P16 (step S302: NO), the search unit 132 returns the process to step S204. If the composite image P20 matches the captured image P10 or the viewpoint converted image P14 matches the overall video P16 (step S302: YES), the search unit 122 notifies the search result (step S304). If the composite image P20 matches the captured image P10, the search unit 132 transmits an image showing the search target candidate and position information of the search target candidate as the search result to the searcher's terminal device 300.
  • the search unit 132 transmits to the searcher's terminal device 300 an image showing the search target candidate that was the basis of the overall video P16, position information of the search target candidate, and the like, as search results.
  • FIG. 11 is a diagram for explaining a modified example of the search support system.
  • the camera system 200 of the search support system in the modified example may include a mobile robot 230 having a camera device.
  • the mobile robot 230 moves, for example, based on control information from the search support device 100 or the operation of the searcher's terminal device 300.
  • the mobile robot 230 moves in the search area, acquires captured images, and transmits them to the search support device 100.
  • the search support device 100 acquires the captured images received from the mobile robot 230 as object information.
  • the search support device 100 creates an overall image P16 of the search target candidate included in the captured images received from the camera device 210 using the object information received from the mobile robot 230.
  • the search support device 100 can create the overall image P16 using captured images captured from an imaging direction different from that of the camera device 210, for example.
  • the mobile robot 230 may move based on the operation of the explorer, and transmit the captured image P10 acquired while moving to the search support device 100. This allows the mobile robot 230 to provide the captured image P10 to the search support device 100 in place of the camera device 210. As a result, the mobile robot 230 can assist in searching for a search target even in a location where a camera device such as the camera device 210 or the infrared camera device 212 is not installed.
  • the exploration support device 100 may control the movement of the mobile robot 230 to acquire captured image P10, for example, to create a 360-degree free viewpoint video.
  • the exploration support device 100 detects a search target candidate using the fixed camera device 210, and when it is desired to check the captured image P10 of the search target candidate in more detail, it transmits to the mobile robot 230 control information for moving the mobile robot 230 to the location where the search target candidate was found.
  • the exploration support device 100 can acquire an image for creating a free viewpoint video by controlling the zoom and imaging direction of the mobile robot 230.
  • the search support device 100D in the fourth embodiment differs from the above-described embodiments in that the search support device 100D searches for a search target based on an object specified by the searcher in the captured image P10.
  • FIG. 12 is a block diagram showing an example of a search support device 100D according to the fourth embodiment.
  • FIG. 13 is a diagram for explaining the processing of the search support device 100D according to the fourth embodiment.
  • the setting unit 110# of the search support device 100D according to the fourth embodiment includes a target image acquisition unit 110a that acquires, as a search target image, an object image P30 specified based on an operation by the searcher from among object images included in the captured video P10.
  • the setting unit 110# sets the object image P30 acquired by the target image acquisition unit 110a as a search target image including the object to be searched.
  • FIG. 14 is a flowchart showing an example of a processing procedure in the search support device 100D in the fourth embodiment.
  • the search support device 100D transmits the captured image P10 received from the camera system 200 by the search area side communication unit 102 to the searcher's terminal device 300 via the terminal side communication unit 104# (step S400).
  • the searcher's terminal device 300 then presents the captured image P10 to the searcher.
  • the searcher's terminal device 300 transmits designation information indicating that an object included in the captured image P10 is designated by the searcher's operation to the search support device 100D.
  • the terminal side communication unit 104# of the search support device 100D receives the designation information (step S402).
  • the target image acquisition unit 110a acquires the object image P30 designated by the searcher based on the designation information as a search target image (step S404).
  • the conversion unit 112# creates a viewpoint-converted image P32 by converting the viewpoint of the object image P30 acquired by the target image acquisition unit 110a as a search target image (step S406).
  • the captured image acquisition unit 106 acquires the captured image P10, and the object information acquisition unit 108 acquires the object information (step S408).
  • the determination unit 114# detects an object included in the captured image P10 that matches one of the viewpoint-converted images P32 as a search target candidate by comparing the viewpoint-converted image P32 with the captured image P10 (step S412: YES), and if the determination unit 114# detects a search target candidate by comparing the viewpoint-converted image P32 with the captured image P10 (step S412: YES), it determines whether the search target candidate is hidden by a surrounding object (step S414). If the determination unit 114# does not detect an object that matches the viewpoint-converted image P32 (step S412: NO), it returns the process to step S408.
  • the overall image creation unit 116# cuts out and extracts an image including the surrounding object image (step S418), and creates an overall image P34 in which the entire search target candidate can be confirmed from the viewpoint conversion image P32 using the surrounding object image (step S420).
  • the terminal side communication unit 104# transmits the overall image P34 to the searcher's terminal device 300 (step S422). If a part of the search target candidate is not hidden by a surrounding object (step S414: NO), the search support device 100D transmits the viewpoint conversion image P32 that matches the object image P30 to the searcher's terminal device 300 via the terminal side communication unit 104# (step S316). This allows the search support device 100D to search for the search target based on the object specified by the searcher in the captured image P10.
  • FIG. 15 is a diagram for explaining a modified example of the fourth embodiment.
  • the camera system 200 of the search support system in the modified example includes a mobile robot 240 having a camera device.
  • the mobile robot 240 moves, for example, based on control information from the search support device 100D or the operation of the searcher's terminal device 300.
  • the mobile robot 240 moves through a search area, for example, based on the operation of the searcher, acquires a captured image P10, and transmits it to the search support device 100D.
  • the search support device 100D acquires the captured image P10 received from the mobile robot 240. This allows the searcher to search for the search target by moving the mobile robot 240.
  • the mobile robot 240 can assist in searching for the search target even in places where a camera device such as the camera device 210 or the infrared camera device 212 is not installed.
  • the search support device 100D may acquire captured image P10 from the camera device 210, and may acquire captured image acquired from the mobile robot 240 as object information.
  • the search support device 100D creates an overall image of the search target candidates included in the captured image P10 received from the camera device 210 using the captured image received from the mobile robot 240. This allows the search support device 100D to create an overall image using captured image captured from a different imaging direction from that of the camera device 210, for example.
  • the exploration support device 100D may control the movement of the mobile robot 240 to acquire captured images, for example, to create a 360-degree free viewpoint video.
  • the exploration support device 100D detects a search target candidate using the fixed camera device 210, and when it is desired to check the captured image of the search target candidate in more detail, it transmits to the mobile robot 240 control information for moving the mobile robot 240 to the location where the search target candidate was found.
  • the exploration support device 100D can acquire images for creating a free viewpoint video by controlling the zoom and image capture direction of the mobile robot 240.
  • the explorer's terminal device 300 has a goggle-type display device (not shown) that is worn on the explorer's head, and may display the captured image P10 on the goggle-type display device.
  • the captured image P10 displayed on the goggle-type display device is preferably a 360-degree free viewpoint image so that the explorer can freely change the viewpoint, but is not limited to this.
  • the captured image P10 may be captured by controlling the imaging direction and imaging range of the camera system 200 to match the explorer's viewpoint, may be captured by controlling the movement of the mobile robot 240 to match the explorer's viewpoint, or may be a virtual reality image that virtually creates an image of the search range.
  • the searcher's terminal device 300 is equipped with a detection device that detects the searcher's gestures, detects finger movements as the searcher's gestures, generates specified information, and transmits it to the search support device 100D.
  • the terminal side communication unit 104# of the search support device 100D receives the specified information, and the target image acquisition unit 110a acquires the object image P30 specified by the searcher based on the specified information as a search target image.
  • the conversion unit 112# creates a viewpoint conversion image P32 by performing viewpoint conversion on the object image P30 specified by the searcher.
  • the determination unit 114# detects search target candidates by comparing the viewpoint conversion image P32 with the captured image P10, and determines whether the search target candidates are hidden by surrounding objects.
  • the overall image creation unit 116# creates an overall image P34 from the viewpoint conversion image P32 using object information of the search target object and object information of the surrounding objects.
  • Overall video creation unit 116# transmits overall video P34 to searcher's terminal device 300 via terminal side communication unit 104#.
  • search support device 100D transmits viewpoint converted image P32 that matches object image P30 to searcher's terminal device 300 via terminal side communication unit 104#. This allows search support device 100D to search for the search target using overall video P34 or viewpoint converted image P32.
  • search support device 100D can have the searcher wear a goggle-type display device to give the searcher the feeling of going to the site and searching.
  • 100, 100B, 100C, 100D...search support device 102...search area side communication unit, 104...terminal side communication unit, 106...captured image acquisition unit, 108...object information acquisition unit, 110...setting unit, 110a...target image acquisition unit, 112...conversion unit, 114...determination unit, 116...image creation unit, 116#...overall image creation unit, 118...search unit, 120...synthesis unit, 122...search unit, 130...image creation unit, 132...search unit, 200...camera system, 210...camera device, 212...infrared camera device, 214...radar device, 216...object information generation device, 220...control unit, 230, 240...mobile robot, 300...searcher terminal device

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Signal Processing (AREA)
  • Computing Systems (AREA)
  • Artificial Intelligence (AREA)
  • Databases & Information Systems (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Closed-Circuit Television Systems (AREA)
  • Alarm Systems (AREA)
  • Processing Or Creating Images (AREA)
PCT/JP2022/044520 2022-12-02 2022-12-02 探索支援装置、探索支援システム、探索支援方法、およびプログラム Ceased WO2024116400A1 (ja)

Priority Applications (6)

Application Number Priority Date Filing Date Title
EP22967230.8A EP4529155B1 (en) 2022-12-02 2022-12-02 Search assistance device, search assistance system, search assistance method, and program
PCT/JP2022/044520 WO2024116400A1 (ja) 2022-12-02 2022-12-02 探索支援装置、探索支援システム、探索支援方法、およびプログラム
KR1020247042908A KR102901590B1 (ko) 2022-12-02 2022-12-02 탐색 지원 장치, 탐색 지원 시스템, 탐색 지원 방법, 및 기록 매체
JP2024557081A JP7675945B2 (ja) 2022-12-02 2022-12-02 探索支援装置、探索支援システム、探索支援方法、およびプログラム
CN202280097856.8A CN120226342A (zh) 2022-12-02 2022-12-02 搜索辅助装置、搜索辅助系统、搜索辅助方法和程序
US19/021,643 US20250157177A1 (en) 2022-12-02 2025-01-15 Search assistance device, search assistance system, search assistance method, and recording medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2022/044520 WO2024116400A1 (ja) 2022-12-02 2022-12-02 探索支援装置、探索支援システム、探索支援方法、およびプログラム

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US19/021,643 Continuation US20250157177A1 (en) 2022-12-02 2025-01-15 Search assistance device, search assistance system, search assistance method, and recording medium

Publications (1)

Publication Number Publication Date
WO2024116400A1 true WO2024116400A1 (ja) 2024-06-06

Family

ID=91323168

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2022/044520 Ceased WO2024116400A1 (ja) 2022-12-02 2022-12-02 探索支援装置、探索支援システム、探索支援方法、およびプログラム

Country Status (6)

Country Link
US (1) US20250157177A1 (https=)
EP (1) EP4529155B1 (https=)
JP (1) JP7675945B2 (https=)
KR (1) KR102901590B1 (https=)
CN (1) CN120226342A (https=)
WO (1) WO2024116400A1 (https=)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113014799B (zh) * 2021-01-28 2023-01-31 维沃移动通信有限公司 图像显示方法、装置和电子设备

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2007272436A (ja) * 2006-03-30 2007-10-18 Univ Of Electro-Communications 物体追跡装置、異常状態検知装置及び物体追跡方法
JP2010237804A (ja) * 2009-03-30 2010-10-21 Casio Computer Co Ltd 画像検索システム及び画像検索方法
JP2011204111A (ja) * 2010-03-26 2011-10-13 Honda Motor Co Ltd 車両周辺監視装置
JP2012080221A (ja) * 2010-09-30 2012-04-19 Jvc Kenwood Corp 目標追跡装置、目標追跡方法
JP2018093423A (ja) 2016-12-06 2018-06-14 富士通株式会社 画像処理プログラム、画像処理装置および画像処理方法
JP2019024258A (ja) 2013-07-09 2019-02-14 キヤノン株式会社 情報処理装置、情報処理方法およびプログラム

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9805274B2 (en) 2016-02-03 2017-10-31 Honda Motor Co., Ltd. Partially occluded object detection using context and depth ordering
JP6528723B2 (ja) * 2016-05-25 2019-06-12 トヨタ自動車株式会社 物体認識装置、物体認識方法及びプログラム
JP6558387B2 (ja) 2017-03-14 2019-08-14 オムロン株式会社 画像解析装置、画像解析方法、及び画像解析プログラム
EP3608873B1 (en) * 2017-12-14 2021-06-09 Canon Kabushiki Kaisha Generation device, generation method and program for three-dimensional model
JP7723638B2 (ja) * 2022-05-09 2025-08-14 Ａｓｔｅｍｏ株式会社 異常診断装置

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2007272436A (ja) * 2006-03-30 2007-10-18 Univ Of Electro-Communications 物体追跡装置、異常状態検知装置及び物体追跡方法
JP2010237804A (ja) * 2009-03-30 2010-10-21 Casio Computer Co Ltd 画像検索システム及び画像検索方法
JP2011204111A (ja) * 2010-03-26 2011-10-13 Honda Motor Co Ltd 車両周辺監視装置
JP2012080221A (ja) * 2010-09-30 2012-04-19 Jvc Kenwood Corp 目標追跡装置、目標追跡方法
JP2019024258A (ja) 2013-07-09 2019-02-14 キヤノン株式会社 情報処理装置、情報処理方法およびプログラム
JP2018093423A (ja) 2016-12-06 2018-06-14 富士通株式会社 画像処理プログラム、画像処理装置および画像処理方法

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
NEC CORPORATIONTOKYO INSTITUTE OF TECHNOLOGY, NEC AND TOKYO INSTITUTE OF TECHNOLOGY JOINTLY DEVELOP ''MULTI-MODAL IMAGE FUSION TECHNOLOGY'' THAT USES AL TO DRAMATICALLY IMPROVE VISIBILITY UNDER POOR CONDITIONS - AUTOMATIC COMBINING OF VISIBLE AND INVISIBLE LIGHT IMAGES, 5 June 2017 (2017-06-05), Retrieved from the Internet <URL:https://www.titech.ac.jp/news/pdf/webtokyotechpr20170605_okutomi.pdf>
See also references of EP4529155A4

Also Published As

Publication number Publication date
EP4529155B1 (en) 2026-04-22
US20250157177A1 (en) 2025-05-15
CN120226342A (zh) 2025-06-27
EP4529155A4 (en) 2025-08-13
JP7675945B2 (ja) 2025-05-13
EP4529155A1 (en) 2025-03-26
KR102901590B1 (ko) 2025-12-17
KR20250004946A (ko) 2025-01-08
JPWO2024116400A1 (https=) 2024-06-06

Similar Documents

Publication Publication Date Title
EP3340177B1 (en) Method and system for creating virtual message onto a moving object and searching the same
JP6647433B1 (ja) 点群データ通信システム、点群データ送信装置および点群データ送信方法
KR101303948B1 (ko) 비가시 객체에 대한 증강 현실 정보 제공 장치 및 방법
KR101295714B1 (ko) 3차원 증강 현실 제공 장치 및 방법
US9898829B2 (en) Monitoring apparatus and system using 3D information of images and monitoring method using the same
US10970932B2 (en) Provision of virtual reality content
US8154616B2 (en) Data processing apparatus and method, and recording medium
JP5147761B2 (ja) 画像監視装置
US20150213649A1 (en) Three-dimensional environment sharing system and three-dimensional environment sharing method
JP2015061239A (ja) 撮像管理システム、撮像管理装置及びそれらの制御方法、プログラム
KR101791553B1 (ko) Cctv 영상에 대한 실시간 얼굴 인식 시스템 및 방법
US11758081B2 (en) Server and method for displaying 3D tour comparison
CN106341665A (zh) 一种跟踪监控方法及装置
JP2012142925A (ja) 映像処理装置および映像処理方法
JP7675945B2 (ja) 探索支援装置、探索支援システム、探索支援方法、およびプログラム
JP2019174164A (ja) 物体認識情報及び受信電磁波情報に係るモデルを用いて端末位置を推定する装置、プログラム及び方法
KR101611427B1 (ko) 영상 처리 방법 및 이를 수행하는 영상 처리 장치
US11076097B2 (en) Method for street view service and apparatus for performing same method
CN120856961A (zh) 基于红外热成像的火灾人员智能搜救方法及装置
US12095966B2 (en) Systems and methods for predictive streaming of image data for spatial computing
JP2005184624A (ja) 商品販売・管理方法、商品販売・管理システムおよびサーバ
US20250014312A1 (en) Image processing system, image processing method, and non-transitory computer-readable medium
KR102046043B1 (ko) 영상의 3차원 정보를 이용한 감시 장치 및 시스템, 그리고 이를 이용한 방법
JP2023026894A (ja) 情報処理装置、情報処理システム、及び情報処理プログラム
KR101749679B1 (ko) 촬영 영상을 이용한 추적대상의 삼차원 형상 생성시스템 및 그 방법

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22967230

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2024557081

Country of ref document: JP

WWE Wipo information: entry into national phase

Ref document number: 2022967230

Country of ref document: EP

ENP Entry into the national phase

Ref document number: 2022967230

Country of ref document: EP

Effective date: 20241217

ENP Entry into the national phase

Ref document number: 20247042908

Country of ref document: KR

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 1020247042908

Country of ref document: KR

WWE Wipo information: entry into national phase

Ref document number: 202280097856.8

Country of ref document: CN

WWP Wipo information: published in national office

Ref document number: 202280097856.8

Country of ref document: CN

NENP Non-entry into the national phase

Ref country code: DE

WWG Wipo information: grant in national office

Ref document number: 2022967230

Country of ref document: EP