JP2022134143A - Detection apparatus, detection method, detection program, and detection system - Google Patents

Abstract

To allow reduction of a load required for searching for image data including a search target such as a person.SOLUTION: A query extracting unit 115 extracts, as a query image relating to an object, image data which are those including the object among a plurality of image data captured by a camera and satisfy a condition. A retrieval unit 116, based on a selected images selected from the query images, retrieves a gallery image including the object included in the selected image from gallery images that are at least a part of image data other than the query image among the plurality of images.SELECTED DRAWING: Figure 2

Description

The present disclosure relates to technology for searching for a designated object from a plurality of image data.

There are cases where a target person such as a lost child or a suspicious person is searched for in facilities such as commercial facilities or transportation facilities. At present, the method of searching for a target person is by calling out to a person in the facility, or by visually searching for the target person from the image data obtained by imaging with a surveillance camera. As the facility becomes larger and the number of people in the facility increases, the burden of searching for the target person becomes heavier.

Japanese Patent Application Laid-Open No. 2004-100000 discloses a method for accumulating human feature amounts and human image data detected from frame images of a plurality of cameras, and accumulating persons similar to persons included in image data designated by a user. It describes searching from image data.

Japanese Patent Application Laid-Open No. 2019-160001

By using the technique described in Patent Literature 1, it becomes possible to mechanically search for a target person from image data obtained by imaging with a monitoring camera or the like. However, with the technique described in Patent Document 1, the user needs to designate a target person for searching from a large amount of accumulated image data. Therefore, the burden of searching for image data containing the target person is heavy, and the target person is found late.
An object of the present disclosure is to reduce the burden of searching for image data that includes an object to be searched for, such as a person.

A detection device according to the present disclosure includes:
a query extraction unit for extracting image data including an object from among a plurality of image data captured by a camera and satisfying a condition as a query image for the object;
Based on the selected image selected from the query image extracted by the query extracting unit, a gallery image that is at least part of the image data other than the query image among the plurality of image data is included in the selected image. a search unit for searching gallery images containing the object.

The query extracting unit extracts, as the query image, image data that satisfies a moving direction condition of the moving trajectory of the object.

The movement direction condition is set by an angle formed between a movement vector indicating the movement locus and a horizontal straight line of the image data.

The query extraction unit extracts the query image from image data captured by a specific camera among a plurality of image data captured by a plurality of cameras.

The search unit searches for a gallery image including an object included in the selected image from among the gallery images whose image-captured time falls within a designated period.

The detection device further
a statistic calculation unit that calculates a statistic about the feature amount of the object from the feature amount of the object calculated from each of the selected image and other image data including the same object,
The search unit searches for gallery images including objects included in the selected image based on the statistics calculated by the statistics calculation unit.

The detection device further
combining the selected image and the one or more other image data based on the feature amount of the object calculated from each of the selected image and the one or more other image data including the same object; a statistic calculation unit that classifies the object into the above groups and calculates statistics for the feature amount of the object calculated from the image data included in the target group for each of the one or more groups ,
The search unit searches for gallery images including objects included in the selected image based on the statistics for each of the one or more groups calculated by the statistics calculation unit.

The detection device further
A query high-dimensional feature amount, which is a feature amount of the object included in the query image and is represented by a multi-dimensional vector, and a feature amount of the object included in the gallery image, wherein a plurality of a feature amount calculation unit that calculates a gallery high-dimensional feature amount that is a feature amount represented by a dimensional vector;
compressing the dimension of the query high-dimensional feature calculated by the feature calculating unit to generate a query low-dimensional feature, and compressing the dimension of the gallery high-dimensional feature calculated by the feature calculating unit; and a dimensional compression unit that generates a gallery low-dimensional feature amount by
The search unit narrows down the gallery images to be searched by comparing the query low-dimensional feature amount generated by the dimension compression unit and the gallery low-dimensional feature amount, and then narrows down the gallery images to be searched. A gallery image including an object included in the selected image is searched by comparing the feature amount with the gallery high-dimensional feature amount of the gallery image to be searched.

The detection device further
A query high-dimensional feature amount, which is a feature amount of the object included in the query image and is represented by a multi-dimensional vector, and a feature amount of the object included in the gallery image, wherein a plurality of a feature amount calculation unit that calculates a gallery high-dimensional feature amount that is a feature amount represented by a dimensional vector;
A query low-dimensional feature amount is generated by excluding elements that do not change even between different objects from the query high-dimensional feature amount calculated by the feature amount calculation unit, and the query low-dimensional feature amount calculated by the feature amount calculation unit A dimensional compression unit that generates a gallery low-dimensional feature amount by excluding elements that do not change even between different objects from the gallery high-dimensional feature amount,
The search unit searches for a gallery image including an object included in the selected image by comparing the query low-dimensional feature amount generated by the dimension compression unit and the gallery low-dimensional feature amount.

The search unit searches for a gallery image including an object included in the selected image from among the gallery images captured between a specified time and a reference time before the specified time. , outputting a gallery image including the same object as that contained in the searched gallery image and having the image taken at the designated time.

A detection method according to the present disclosure includes:
extracting, from among a plurality of image data captured by a camera, image data that includes an object and that satisfies a condition as a query image for the object;
Based on a selected image selected from the query image, a gallery image including an object included in the selected image is retrieved from gallery images that are at least part of image data other than the query image among the plurality of image data. do.

A detection program according to the present disclosure includes:
a query extraction process for extracting image data including an object from among a plurality of image data obtained by imaging with a camera, and image data satisfying a condition, as a query image for the object;
Based on the selected image selected from the query image extracted by the query extraction process, from a gallery image that is at least part of the image data other than the query image among the plurality of image data, included in the selected image The computer is made to function as a detection device that performs search processing for searching gallery images that include objects.

A detection system according to the present disclosure includes:
A detection system comprising one or more detection devices and a search instruction device,
each of the one or more sensing devices comprising:
a query extracting unit for extracting, from among a plurality of image data captured by a camera, image data that includes an object and that satisfies a condition as a query image for the object;
The search instruction device is
a query selection unit that selects a selection image from the query images extracted by the query extraction unit of each of the one or more sensing devices;
each of the one or more sensing devices comprising:
Based on the selected image selected by the query selection unit, a gallery image including an object included in the selected image is selected from a gallery image that is at least part of image data other than the query image among the plurality of image data. a search unit for searching for

In the present disclosure, among a plurality of pieces of image data, image data that includes an object and satisfies a condition is extracted as a query image for the object, and a selected image is selected from the query images. Therefore, the image data from which the selected image is selected is narrowed down, and the burden of searching for the image data including the object is light.

1 is a configuration diagram of a detection system 100 according to Embodiment 1. FIG. 1 is a configuration diagram of a detection device 10 according to Embodiment 1. FIG. 1 is a configuration diagram of a search instruction device 20 according to Embodiment 1. FIG. 4 is a flowchart showing an outline of processing of the detection system 100 according to Embodiment 1; 5 is a flowchart of query extraction processing (step S1 in FIG. 4) according to the first embodiment; FIG. 4 is an explanatory diagram of information stored in the data storage process according to the first embodiment; FIG. 6 is a flowchart of query determination processing (step S15 in FIG. 5) according to the first embodiment; FIG. 8 is an explanatory diagram of direction determination processing (step S155 in FIG. 7) according to the first embodiment; 5 is a flowchart of search processing (step S2 in FIG. 4) according to the first embodiment; FIG. 10 is an explanatory diagram of query condition acquisition processing (step S21 in FIG. 9) according to the first embodiment; FIG. The block diagram of the detection apparatus 10 which concerns on the modification 4. FIG. FIG. 11 is a configuration diagram of a search instruction device 20 according to Modification 4; FIG. 2 is a configuration diagram of a detection device 10 according to Embodiment 3; FIG. 4 is a configuration diagram of a detection device 10 according to Embodiment 4; 5 is a flowchart of search processing (step S2 in FIG. 4) according to Embodiment 4; FIG. 16 is an explanatory diagram of a similarity calculation process (step S25A in FIG. 15) according to the fourth embodiment; FIG. FIG. 10 is an explanatory diagram of a result transmission process (step S25 in FIG. 9) according to the fifth embodiment; FIG.

Embodiment 1.
*** Configuration description ***
A configuration of a detection system 100 according to Embodiment 1 will be described with reference to FIG.
The detection system 100 includes one or more detection devices 10 , a search instruction device 20 and a user terminal 30 . The detection device 10 is a computer that detects an object from image data. The search instruction device 20 is a computer that instructs the detection device 10 to search. The user terminal 30 is a computer such as a PC (Personal Computer) or a tablet terminal used by the user.
Each detection device 10 , search instruction device 20 and user terminal 30 are connected via a transmission line 90 . One or more cameras 40 are connected to each detection device 10 . A storage device 50 is connected to each detection device 10 .

A configuration of the detection device 10 according to the first embodiment will be described with reference to FIG.
The detection device 10 comprises hardware including a processor 11 , a memory 12 , a storage 13 and a communication interface 14 . The processor 11 is connected to other hardware via signal lines and controls these other hardware.

The detection device 10 includes an acquisition unit 111, a detection unit 112, a feature amount calculation unit 113, a tracking unit 114, a query extraction unit 115, a search unit 116, and an output unit 117 as functional components. The function of each functional component of the detection device 10 is realized by software.
The storage 13 stores a program that implements the function of each functional component of the detection device 10 . This program is read into the memory 12 by the processor 11 and executed by the processor 11 . Thereby, the function of each functional component of the detection device 10 is realized.

A configuration of the search instruction device 20 according to the first embodiment will be described with reference to FIG.
The search instruction device 20 includes hardware including a processor 21 , a memory 22 , a storage 23 and a communication interface 24 . The processor 21 is connected to other hardware via signal lines and controls these other hardware.

The search instruction device 20 includes a query selection unit 211 and an image selection unit 212 as functional components. The function of each functional component of the search instruction device 20 is implemented by software.
The storage 23 stores a program that implements the function of each functional component of the search instruction device 20 . This program is read into the memory 22 by the processor 21 and executed by the processor 21 . Thereby, the function of each functional component of the search instruction device 20 is realized.

The processors 11 and 21 are ICs (Integrated Circuits) that perform processing. As a specific example, the processors 11 and 21 are CPUs (Central
Processing Unit), DSP (Digital Signal Processor), and GPU (Graphics Processing Unit).

The memories 12 and 22 are storage devices that temporarily store data. Specific examples of the memories 12 and 22 are SRAM (Static Random Access Memory) and DRAM (Dynamic Random Access Memory).

The storages 13 and 23 are storage devices that store data. As a specific example, the storages 13 and 23 are HDDs (Hard Disk Drives). The storages 13 and 23 are SD (registered trademark, Secure Digital) memory cards, CF (Compact Flash, registered trademark), NAND flashes, flexible disks, optical disks, compact disks, Blu-ray (registered trademark) disks, DVDs (Digital Versatile Disks). ) may be a portable recording medium.

Communication interfaces 14 and 24 are interfaces for communicating with external devices. Specific examples of the communication interfaces 14 and 24 are Ethernet (registered trademark), USB (Universal Serial Bus), and HDMI (registered trademark, High-Definition Multimedia Interface) ports.

Only one processor 11 was shown in FIG. However, there may be a plurality of processors 11, and the plurality of processors 11 may cooperate to execute programs that implement each function. Similarly, only one processor 21 was shown in FIG. However, there may be a plurality of processors 21, and the plurality of processors 21 may cooperate to execute programs that implement each function.

***Description of operation***
The operation of the detection system 100 according to Embodiment 1 will be described with reference to FIGS. 4 to 10. FIG.
The operation procedure of the detection system 100 according to the first embodiment corresponds to the detection method according to the first embodiment. Also, a program that realizes the operation of the detection system 100 according to the first embodiment corresponds to the detection program according to the first embodiment.

An overview of the processing of the detection system 100 according to the first embodiment will be described with reference to FIG.
(Step S1: query extraction processing)
The acquisition unit 111 in each detection device 10 acquires a plurality of image data captured by the camera 40 connected to the detection device 10 . The query extracting unit 115 extracts, from among the plurality of image data, image data that includes an object and that satisfies a condition as a query image for the object.

(Step S2: search processing)
The query selection unit 211 in the search instruction device 20 selects a query image about an object to be searched for as a selection image from the query images extracted by the query extraction unit 115 of each detection device 10 by an operation from the user terminal 30. .
The search unit 116 in each detection device 10 selects at least a part of the image data other than the query image for the object from among the plurality of image data based on the selected image selected by the query selection unit 211 of the search instruction device 20. A gallery image is searched for a gallery image containing an object included in the selected image. The output unit 117 outputs the searched gallery image to the user terminal 30 . Also, the output unit 117 may add a plurality of frames before and after the searched gallery image and output it to the user terminal 30 as a moving image.

The query extraction process (step S1 in FIG. 4) according to the first embodiment will be described with reference to FIG.
Here, each camera 40 connected to each detection device 10 is assigned a unique camera ID (IDentifier), and is installed so as to capture a part of the target area for detecting the target object. The objects are people, animals, and cars. Embodiment 1 will be described by taking a person as an example of a target object. The target area is facilities such as commercial facilities and transportation facilities.
Each camera 40 acquires video data with a predetermined frame rate. Each camera 40 transmits the image data together with the camera ID and the imaging time of the image data to the connected detection device 10 when the image data of the new frame forming the video data is acquired.

(Step S11: Image Acquisition Processing)
The acquisition unit 111 acquires image data, camera ID, and imaging time transmitted from the camera 40 connected to the detection device 10 . The acquisition unit 111 associates the image data, the camera ID, and the imaging time and writes them in the storage device 50 .

(Step S12: detection processing)
The detection unit 112 detects an object from the image data acquired in step S11.
Specifically, the detection unit 112 receives image data as an input, and for each object included in the image data, determines the area where the object exists in the image data, the confidence that it is the object, and the attributes of the object. Use a detection model that outputs That is, the detection unit 112 supplies the image data acquired in step S11 as an input to the detection model, and acquires the region where the object exists, the certainty that it is the object, and the attributes of the object. Detect objects. In Embodiment 1, the object is a person. Therefore, the detection unit 112 obtains an area in which a person exists in the image data, a degree of certainty that the person is a person, and attributes of the person by giving the image data to the detection model as an input. Specific examples of human attributes include gender and age group.

The processing from step S13 to step S16 is executed for each object detected in step S12.

(Step S13: feature amount calculation processing)
The feature quantity calculation unit 113 calculates a feature quantity for the object to be processed. Here, the feature amount is numerical information represented by a multi-dimensional vector. Note that any method may be used to calculate the feature amount. Further, the feature amount may be numerical information expressed as a matrix of N rows and M columns based on arbitrary integers N and M.

(Step S14: Tracking process)
The tracking unit 114 tracks an object to be processed.
Specifically, the tracking unit 114 performs tracking by specifying the object to be processed from the objects detected from the image data acquired in step S11 during the past tracking period. In the first embodiment, the target object is a person. Therefore, the tracking unit 114 identifies the same person as the person to be processed from among the persons detected from the image data acquired in step S11 in the past tracking period. The same ID as the object ID assigned during the tracking period is assigned to the person to be processed, and is associated with the image data stored in step S11.
For example, the tracking unit 114 tracks each object based on the similarity of the feature amount calculated in step S13 and the positional relationship. Any specific tracking method may be used.
If an object ID is given to each object in the detection process of step S12, there is a high possibility that different object IDs will be given to the same person. By assigning an object ID after identifying the same person in the tracking process of step S14, the same object ID can be assigned to the same person.

(Step S15: query determination processing)
The query extraction unit 115 determines whether or not the image data acquired in step S11 is used as a query image for the object to be processed. Query determination processing will be described later.

(Step S16: data storage processing)
As shown in FIG. 6, the output unit 117 outputs the object ID of the object to be processed identified in step S14, the existing area detected in step S12, the degree of certainty, and the attribute of the object to be processed. Then, the feature amount calculated in step S13 and the determination result determined in step S15 are written in the storage device 50 in association with the image data stored in step S11. As a determination result, if the image is determined to be the query image, it is set as the query image, and if it is determined not to be the query image, the image is set as the gallery image.

The query determination process (step S15 in FIG. 5) according to the first embodiment will be described with reference to FIG.
(Step S151: camera determination processing)
The query extracting unit 115 obtains the image data stored in the storage device 50 in step S11 by using a specific camera 40 for obtaining a query image among the cameras 40 connected to the detection device 10. It is determined whether or not the image data is the
If the image data stored in the storage device 50 in step S11 is image data captured by a specific camera 40, the query extraction unit 115 advances the process to step S152. On the other hand, if the image data stored in the storage device 50 in step S11 is not image data captured by a specific camera 40, the query extraction unit 115 advances the process to step S158.

(Step S152: Object determination processing)
The query extraction unit 115 determines whether or not the query image for the object to be processed has not yet been extracted. That is, in Embodiment 1, the query extraction unit 115 determines whether or not the query image for the person to be processed has not yet been extracted. Specifically, the query extracting unit 115 performs the processing of steps S151 to S158 corresponding to step S15 to extract the object linked to the person to be processed among the objects stored in the storage device 50. If an object having the same ID as the ID and the determination result is the query image is not stored, it is determined that the object has not been extracted. A query image is image data set as a query image as a determination result.
The query extracting unit 115 advances the process to step S153 when the query is not extracted yet. On the other hand, the query extracting unit 115 advances the process to step S158 if the query is not extracted yet.

(Step S153: Past determination process)
The query extraction unit 115 determines whether or not the object to be processed has been detected from the image data acquired in the past tracking period by the camera 40 from which the image data acquired in step S11 was acquired. Specifically, the query extracting unit 115 refers to the result of tracking in step S14, and determines whether or not the object to be processed has been detected from the image data acquired during the past tracking period. .
If the object to be processed has been detected, the query extracting unit 115 advances the process to step S154. On the other hand, if the object to be processed has not been detected, the query extraction unit 115 advances the process to step S158.
This determination is made because the object to be processed must have been detected in the past in order to make a determination based on the movement trajectory of the object to be processed in the processes of steps S155 and S156, which will be described later.

(Step S154: position determination processing)
The query extracting unit 115 determines whether or not the object to be processed is within the detection area for acquiring the query image. Specifically, the query extracting unit 115 determines whether or not the reference position of the area in which the processing target object identified in step S12 exists is within the detection area. In Embodiment 1, the reference position is the position of the feet of the person who is the object. For example, if the area in which the object exists is a rectangular area surrounding the object, the query extraction unit 115 may set the position of the center point of the lower side of the rectangular area as the position of the foot. The detection area is an area that is set in advance and is an area within the imaging area for the specific camera 40 in step S151. As the detection area, for example, an area in which the object can be imaged in a sufficiently large size is set.
If the object to be processed is within the detection area, the query extracting unit 115 advances the process to step S155. On the other hand, if the object to be processed is not within the detection area, the query extraction unit 115 advances the process to step S158.

(Step S155: direction determination processing)
The query extraction unit 115 determines whether or not the object to be processed has moved from the back side to the front side of the camera 40 from which the image data was acquired. Thereby, the query extracting unit 115 determines whether or not the front side of the object is captured. Note that it is assumed that the object is moving forward.
Specifically, the query extraction unit 115 determines that the position of the target object to be processed is closer to the camera 40 than the position of the target object to be processed detected from the image data acquired in step S11 in the past. Determine whether or not The query extracting unit 115 determines that the object to be processed has moved from the back side to the front side if it is on the front side. Here, the query extracting unit 115 determines that the reference position of the object to be processed is closer to the camera 40 than the reference position of the object to be processed detected from the image data acquired in step S11 in the past. Determine whether or not
As shown in FIG. 8, it is assumed that the x-axis is taken in the width direction of the image data and the y-axis is taken in the depth direction. It is assumed that the y-axis coordinate value increases toward the near side (lower side of the image data). In this case, the query extraction unit 115 determines that the y-coordinate of the reference position of the object to be processed detected from the image data acquired in step S11 in the past is greater than the y-coordinate of the reference position of the object to be processed. is smaller, it is determined that the object to be processed has moved from the back side to the front side.
If the object to be processed has moved from the back side to the front side, the query extraction unit 115 advances the process to step S156. On the other hand, if the object to be processed has not moved from the back side to the front side, the query extraction unit 115 advances the process to step S158.

(Step S156: Angle determination processing)
As shown in FIG. 8, the query extracting unit 115 determines whether the absolute value of the cosine value of the angle between the movement vector of the target object to be processed and the straight line in the width direction of the image data in the detection region is equal to or less than a threshold. determine whether Accordingly, the query extracting unit 115 determines whether or not the target object to be processed is not moving obliquely toward the camera 40 but moving toward the camera 40 to some extent or more.
If the absolute value of the cosine value of the formed angle is equal to or less than the threshold, the query extraction unit 115 advances the process to step S157. On the other hand, if the absolute value of the cosine value of the formed angle is not equal to or less than the threshold, the query extraction unit 115 advances the process to step S158.

(Step S157: query identification process)
The query extraction unit 115 determines that the image data acquired in step S11 is the query image.

(Step S158: Gallery Specific Processing)
The query extraction unit 115 determines that the image data acquired in step S11 should not be used as a query image, but should be used as a gallery image.

That is, in steps S155 and S156, the query extracting unit 115 determines whether or not the moving trajectory of the object to be processed satisfies the moving direction condition. Then, if the moving trajectory satisfies the moving direction condition, the image data is determined to be image data obtained by imaging the object from the front, and the query extracting unit 115 sets the image data as a query image in step S157. . On the other hand, if the moving trajectory does not satisfy the moving direction condition, the image data is not image data obtained by imaging the object from the front, and the query extracting unit 115 treats the image data as a gallery image in step S157. .

The search process (step S2 in FIG. 4) according to the first embodiment will be described with reference to FIG.
(Step S21: Query condition acquisition process)
The query selection unit 211 of the search instruction device 20 acquires the narrowing condition 1 of the query image from the user terminal 30 . The narrowing-down condition 1 is specified by the camera ID, the imaging time range, the degree of certainty, the attribute, and the like.
Specifically, as shown in FIG. 10, the query selection unit 211 displays on the user terminal 30 a search screen 60 including an input field 61 for the narrowing condition 1 of the query image, and prompts the user to specify the narrowing condition 1. . The query selection unit 211 acquires the designated narrowing condition 1 .

(Step S22: Query Image Acquisition Processing)
The query selection unit 211 of the search instruction device 20 transmits the narrowing condition 1 acquired in step S21 to each detection device 10 . Then, the search unit 116 in each detection device 10 identifies a query image that satisfies the narrowing-down condition 1 . Specifically, the search unit 116 searches image data (see FIG. 6) stored in the storage device 50 connected to the detection device 10, and extracts image data corresponding to the camera ID and imaging time range indicated by the narrowing-down condition 1. identify. The search unit 116 searches the identified image data for objects that include objects that meet the certainty and attributes indicated by the narrowing-down condition 1, and searches for objects that meet the certainty and attributes as a query image as a determination result. Extract the set image data.
Then, the output unit 117 transmits the partial image data obtained by extracting only the region where the target object exists from the extracted image data and the feature amount of the target object to the search instruction device 20. . At this time, the output unit 117 transmits information such as the camera ID and imaging time of the original image data of the partial image data, and the certainty and attributes of the target indicated by the partial image data, along with the partial image data and the feature amount. You may

(Step S23: Query image selection process)
The query selection unit 211 of the search instruction device 20 selects a selection image, which is a query image for an object to be searched, based on the partial image data transmitted in step S22.
Specifically, as shown in FIG. 10, the query selection unit 211 displays the partial image data transmitted in step S22 in the query image field 62 of the search screen 60, and selects a query image about the object to be searched. Let the user choose. At this time, the query selection unit 211 may display information such as the camera ID, the imaging time, the degree of certainty, and the attributes on the search screen 60 together with the partial image data. Then, the query selection unit 211 sets the selected partial image data as the selected image.
The query selection unit 211 transmits the feature amount of the selected image to each detection device 10 . The feature amount of the selected image is the feature amount transmitted together with the selected image. At this time, the query selection unit 211 may transmit the narrowing condition 2 for narrowing down the gallery images together with the feature amount. When sending the narrowing condition 2, an input field for the narrowing condition 2 is provided on the search screen 60, and the user specifies the narrowing condition 2. FIG. The narrowing-down condition 2 is specified by a specified period, a certainty factor, a camera ID, and the like.

(Step S24: Similarity calculation process)
The search unit 116 in each detection device 10 searches the gallery images stored in the storage device 50 for gallery images including the object included in the selected image based on the feature amount of the selected image.
Specifically, the search unit 116 calculates the degree of similarity between the feature amount of the selected image transmitted in step S23 and the feature amount of each gallery image. A gallery image is partial image data of an area in which an object, which is set as a gallery image as a result of determination, exists among the image data stored in the storage device 50 connected to the detection device 10 (see FIG. 6). The feature amount of the gallery image is the feature amount of the object set as the gallery image as the determination result.
It should be noted that when the narrowing condition 2 is transmitted in step S23, the search unit 116 searches only the gallery images corresponding to the narrowing condition 2 among the image data stored in the storage device 50 connected to the detection device 10. As a target, similarity is calculated. When a specified period is specified as the narrowing-down condition 2, the search unit 116 targets only gallery images whose imaging times are included in the specified period. When certainty is specified as narrowing-down condition 2, search unit 116 targets only gallery images with certainty greater than or equal to the certainty specified as narrowing-down condition 2. FIG. When the camera ID is specified as the narrowing-down condition 2, the search unit 116 targets only the gallery images of the camera ID specified as the narrowing-down condition 2. FIG.
Note that the imaging time used in the present embodiment may be the acquisition time at which the detection device 10 acquires the image data from the camera 40 . By narrowing down the search using the shooting time or the acquisition time as a specified period, it is possible to search for the target object in a short processing time when the approximate time zone in which the target object exists can be grasped.

(Step S25: result transmission process)
The search unit 116 in each detection device 10 identifies the gallery image with the highest similarity calculated in step S24. The search unit 116 transmits the specified gallery image to the search instruction device 20, along with the camera ID and imaging time of the gallery image, and the degree of similarity calculated in step S24.
Note that the search unit 116 may specify the gallery image with the highest similarity for each camera 40 . Then, for each camera 40, the search unit 116 transmits the specified gallery image, the camera ID, the imaging time, the object ID, and the similarity calculated in step S24 to the search instruction device 20. You may send.

(Step S26: confirmation image selection process)
The image selection unit 212 of the search instruction device 20 selects a gallery image to be confirmed based on the information transmitted in step S25.
Specifically, as shown in FIG. 10, the gallery images and the like transmitted in step S25 are displayed in the gallery image field 63 of the search screen 60 to prompt the user to select the confirmed gallery image.

(Step S27: image output processing)
The image selection unit 212 of the search instruction device 20 transmits the camera ID, imaging time, and object ID of the gallery image selected in step S26 to the detection device 10 that transmitted the selected gallery image. Then, the output unit 117 in the detection device 10 that is the transmission source reads the image data specified by the transmitted camera ID and imaging time from the storage device 50 . Then, the output unit 117 transmits the image data to the user terminal 30 after, for example, enclosing the area in which the object indicated by the transmitted object ID exists with a frame.
After that, the output unit 117 transmits the image data before and after the time to the user terminal 30 according to the operation from the user terminal 30 . Also at this time, the output unit 117 transmits the image data after enclosing the area in which the object indicated by the object ID exists with a frame or the like.

*** Effect of Embodiment 1 ***
As described above, the detection apparatus 10 according to Embodiment 1 extracts, from among a plurality of pieces of image data, image data that includes an object and satisfies the conditions as a query image for the object. , a selection image is selected from the query image. Therefore, the image data from which the selected image is selected is narrowed down, and the burden of searching for the image data including the object is light.

The detection apparatus 10 according to Embodiment 1 extracts image data that satisfies the condition of the movement direction of the moving trajectory of the object as a query image. This makes it easier to extract image data obtained by capturing an object from the front as a query image. As a result, it becomes easier to appropriately extract the gallery image containing the object to be searched for.

The detection device 10 according to Embodiment 1 extracts image data in which an object is located in the detection area as a query image. This makes it difficult to extract image data in which the target object is cut off as a query image, and makes it easier to extract image data including the entire target object or the like as a query image. As a result, it becomes easier to appropriately extract the gallery image containing the object to be searched for.

The detection system 100 according to Embodiment 1 includes a plurality of detection devices 10, extracts a query image from image data captured by a camera 40 to which each detection device 10 is connected, and performs a search based on the selected image. conduct. As a result, even when the number of cameras 40 is large, it is possible to perform searches at high speed. Further, when the number of cameras 40 increases, it is possible to cope with this by increasing the number of detection devices 10 .

***Other Configurations***
<Modification 1>
The detection area used in step S154 of FIG. 7 and the threshold value used in step S156 may be set for each camera 40 . For example, the detection area and the threshold are set according to the installation position of the camera 40, the area through which the object passes, and the like.

<Modification 2>
The feature amount of the object calculated in step S13 of FIG. 5 may be calculated based on the image data of the entire object, or may be calculated based on the image data of a specific portion of the object. For example, when the object is a person, the feature amount may be calculated based on the image data of only the face or the image data of only the upper half of the body.

<Modification 3>
In Embodiment 1, the image data was stored in the storage device 50 connected to each sensing device 10 . However, the image data may also be stored in the storage 13 of each sensing device 10 .

<Modification 4>
In Embodiment 1, each functional component is realized by software. However, as a modification 4, each functional component may be realized by hardware. Regarding this modification 4, points different from the first embodiment will be described.

The configuration of the detection device 10 according to Modification 4 will be described with reference to FIG. 11 .
If each functional component is implemented in hardware, the sensing device 10 includes an electronic circuit 15 instead of the processor 11 , memory 12 and storage 13 . The electronic circuit 15 is a dedicated circuit that realizes the functions of each functional component, memory 12 and storage 13 .

The configuration of the search instruction device 20 according to Modification 4 will be described with reference to FIG.
When each functional component is implemented by hardware, the search instruction device 20 includes an electronic circuit 25 instead of the processor 21, memory 22 and storage 23. FIG. The electronic circuit 25 is a dedicated circuit that realizes the functions of each functional component, memory 22 and storage 23 .

The electronic circuits 15 and 25 include single circuits, composite circuits, programmed processors, parallel programmed processors, logic ICs, GAs (Gate Arrays), ASICs (Application Specific Integrated Circuits), and FPGAs (Field-Programmable Gate Arrays). ) is assumed.
Each functional component may be implemented by one electronic circuit 15, 25, or each functional component may be implemented by being distributed among a plurality of electronic circuits 15, 25. FIG.

<Modification 5>
As Modified Example 5, some functional components may be implemented by hardware, and other functional components may be implemented by software.

Processors 11 and 21, memories 12 and 22, storages 13 and 23, and electronic circuits 15 and 25 are called processing circuits. That is, the function of each functional component is realized by the processing circuit.

Embodiment 2.
Embodiment 2 differs from Embodiment 1 in that real-time search is performed. In the second embodiment, this different point will be explained, and the explanation of the same point will be omitted.

***Description of operation***
The operation of the detection system 100 according to Embodiment 2 will be described with reference to FIG.
The operation procedure of the detection system 100 according to the second embodiment corresponds to the detection method according to the second embodiment. Also, a program that realizes the operation of the detection system 100 according to the second embodiment corresponds to the detection program according to the second embodiment.

The search process (step S2 in FIG. 4) according to the second embodiment will be described with reference to FIG.
The processing from step S21 to step S23 in FIG. 9 and the processing from step S25 to step S26 are the same as those in the first embodiment.

(Step S24: Similarity calculation process)
The search unit 116 in each detection device 10 searches for a gallery image including the target object included in the selected image from the most recently acquired gallery images based on the feature amount of the selected image. That is, the search unit 116 calculates the degree of similarity between the feature amount of the selected image and the most recently acquired gallery image. Then, in step S25, the gallery image with the highest degree of similarity is identified.

That is, in the first embodiment, the gallery images that include the object included in the selected image are searched from the gallery images accumulated in the storage device 50 . On the other hand, in the second embodiment, a gallery image including the object included in the selected image is searched from the gallery images of the most recently acquired frame.
Note that the real-time search described in the present embodiment targets the most recently acquired gallery image. For example, if the latest is defined as five minutes before the current time, the specified period is from five minutes before the current time. A gallery image including an object included in a selected image, which is a query image for an object to be searched for, is searched from the gallery images included in the specified period in which the image was taken most recently.

*** Effect of Embodiment 2 ***
As described above, the detection system 100 according to Embodiment 2 realizes real-time search. For example, when looking for a lost child, it is necessary to identify the current location rather than the past location. Therefore, real-time search is useful.
It should be noted that if only real-time search is implemented and past image data is not searched, there is no need to store gallery images of past frames in the storage device 50 . Therefore, the gallery image may not be stored in the storage device 50, and only the gallery image of the most recent frame may be stored in the memory 12. FIG. This allows faster searches.

Embodiment 3.
Embodiment 3 differs from Embodiments 1 and 2 in that a plurality of query images are extracted for one object. In the third embodiment, this different point will be explained, and the explanation of the same point will be omitted.
Embodiment 3 describes a case where the functions of Embodiment 1 are changed. However, it is also possible to change the function of the second embodiment.

*** Configuration description ***
The configuration of the detection device 10 according to Embodiment 3 will be described with reference to FIG. 13 .
The detection device 10 differs from the detection device 10 shown in FIG. 2 in that it includes a statistic calculator 118 as a functional component. The functions of the statistic calculation unit 118 are realized by software or hardware, like other functional components.

***Description of operation***
The operation of the detection system 100 according to Embodiment 3 will be described with reference to FIG.
The operation procedure of the detection system 100 according to the third embodiment corresponds to the detection method according to the third embodiment. A program that realizes the operation of the detection system 100 according to the third embodiment corresponds to the detection program according to the third embodiment.

In Embodiment 3, the query extraction unit 115 extracts multiple query images for one object.
Specifically, when the query image is determined to be a query image in the query determination process (step S15 in FIG. 5), the query extracting unit 115 includes the imaging time in the reference time before and after the image data determined to be the query image. Image data that includes the same object is set as a related query image for that object.
Alternatively, if the query image is determined to be a query image in the query determination process (step S15 in FIG. 5), the query extracting unit 115 extracts image data captured by a specific camera 40 out of the other cameras 40 and extracts the same image data. set the image data containing the object to be the related query image for that object.

The statistic calculation unit 118 calculates the statistic of the feature amounts of the query image and related query images. A statistic is, for example, an average value. Since the feature amount is a multi-dimensional vector, when the statistic is the average value, the average value for the elements of each dimension is calculated. The statistic calculation unit 118 sets the statistic as the feature amount of the query image.

The search process (step S2 in FIG. 4) according to the third embodiment will be described with reference to FIG.
The processing from step S21 to step S23 in FIG. 9 and the processing from step S25 to step S26 are the same as those in the first embodiment.

(Step S24: Similarity calculation process)
The search unit 116 in each detection device 10 searches the gallery images stored in the storage device 50 for gallery images including the object included in the selected image based on the statistics of the selected image. In other words, the retrieval unit 116 differs from the first embodiment in that it uses a statistic instead of the feature of the selected image.

*** Effect of Embodiment 3 ***
As described above, the detection apparatus 10 according to Embodiment 3 searches for gallery images using statistics calculated from feature amounts of a plurality of query images (query images and related query images).
Objects included in the gallery image may be imaged from different angles or in different poses than the objects included in the query image. By using the statistics calculated from the feature amounts of a plurality of query images, it becomes possible to search for gallery images by taking into account the feature amounts of objects with various appearances. As a result, it is possible to appropriately search for gallery images in which objects are captured from different angles and gallery images including objects in different postures.

***Other Configurations***
<Modification 6>
The statistic calculation unit 118 classifies a plurality of query images (query images and related query images) into one or more groups, targets each of the one or more groups, and calculates query images (query images and related query images) may be calculated. At this time, the statistic calculation unit 118 may classify the plurality of query images into one or more groups based on the feature amounts of the plurality of query images. A plurality of query images are grouped into one or more groups based on at least one of the dimensional elements that change depending on at least one of the difference in the angle at which the object is captured and the difference in the orientation of the object, among the elements that make up the feature amount. can be classified.
In this case, in step S24 of FIG. 9, the search unit 116 searches for gallery images including the object included in the selected image based on the statistics for each of the one or more groups.
This makes it possible to more appropriately search for gallery images in which objects are captured from different angles and gallery images including objects in different postures.

Embodiment 4.
Embodiment 4 differs from Embodiments 1 to 3 in that the feature quantity, which is a multi-dimensional vector, is compressed and searched. In the fourth embodiment, this different point will be explained, and the explanation of the same point will be omitted.
Embodiment 4 describes a case where the functions of Embodiment 1 are changed. However, it is also possible to change the functions of the second and third embodiments.

*** Configuration description ***
The configuration of the detection device 10 according to Embodiment 4 will be described with reference to FIG. 14 .
The detection device 10 differs from the detection device 10 shown in FIG. 2 in that it includes a dimension compression unit 119 as a functional component. The functions of the dimensional compression unit 119 are realized by software or hardware, like other functional components.

The operation of the detection system 100 according to Embodiment 4 will be described with reference to FIGS. 15 and 16. FIG.
The operation procedure of the detection system 100 according to the fourth embodiment corresponds to the detection method according to the fourth embodiment. A program that realizes the operation of the detection system 100 according to the fourth embodiment corresponds to the detection program according to the fourth embodiment.

The search process (step S2 in FIG. 4) according to the fourth embodiment will be described with reference to FIG.
The processing from step S21A to step S23A is the same as the processing from step S21 to step S21 to step S23 in FIG. The processing from step S26A to step S28A is the same as the processing from step S25 to step S27 in FIG.

(Step S24A: dimension compression processing)
The dimension compression unit 119 in each detection device 10 compresses the dimension of the feature amount of the selected image and the feature amount of each gallery image.
Specifically, the dimension compression unit 119 generates a query low-dimensional feature amount by compressing the dimension of the query high-dimensional feature amount, which is the feature amount of the selected image. Also, the dimension compression unit 119 generates gallery low-dimensional feature amounts by compressing the dimensions of the gallery high-dimensional feature amounts, which are the feature amounts of the target gallery images, for each gallery image. For example, the dimension compression unit 119 may
Generate a dimensional query low-dimensional feature and a 2-dimensional gallery low-dimensional feature. Compressing a dimension means reducing the dimension. Methods such as PCA (principal component analysis) and LSA (latent semantic analysis) may be used as methods for compressing dimensions.

(Step S25A: Similarity calculation process)
The search unit 116 in each detection device 10 narrows down gallery images to be searched by comparing the query low-dimensional feature amount generated in step S24A with the gallery low-dimensional feature amount of each gallery image. As a specific example, as shown in FIG. 16, the search unit 116 searches only gallery images having gallery low-dimensional feature amounts having feature amounts in regions adjacent to the area having the feature amount for the query low-dimensional feature amount. Narrow down the target gallery images.
The search unit 116 searches for a gallery image including an object included in the selected image by comparing the query high-dimensional feature amount with the gallery high-dimensional feature amount of the gallery image to be searched. That is, the search unit 116 calculates the similarity between the query high-dimensional feature amount and the gallery high-dimensional feature amount of the gallery image to be searched. Then, in step S26A, the gallery image with the highest degree of similarity is specified.

*** Effect of Embodiment 4 ***
As described above, the detection apparatus 10 according to Embodiment 4 narrows down gallery images to be searched using the query low-dimensional feature amount and the gallery low-dimensional feature amount obtained by compressing the dimension of the feature amount. As a result, the number of times of processing for calculating similarity can be reduced. As a result, it is possible to efficiently search for gallery images.

***Other Configurations***
<Modification 7>
The dimension compression unit 119 may compress the dimension of the feature amount by excluding from the feature amount elements of dimensions in which a large change is not observed even between different objects. Different objects may include not only different objects themselves but also different angles of view of the objects.
In this case, in step S24A of FIG. 15, the dimension compression unit 119 extracts query low-dimensional features by excluding elements that do not change even between different objects from the query high-dimensional feature amount, which is the feature amount of the selected image. produce quantity. Further, the dimensionality compression unit 119 removes elements that do not change even between different objects from the gallery high-dimensional feature amount, which is the feature amount of the gallery image of interest, for each gallery image, and extracts the gallery low-dimensional feature amount. to generate
In step S25A of FIG. 15, the search unit 116 searches for gallery images including the object included in the selected image by comparing the query low-dimensional feature amount with the gallery low-dimensional feature amount of each gallery image. That is, the search unit 116 calculates the similarity between the query low-dimensional feature amount and the gallery low-dimensional feature amount of each gallery image. Then, in step S26A, the gallery image with the highest degree of similarity is identified.

Embodiment 5.
Embodiment 5 differs from Embodiment 2 in that past image data is used in real-time search. In the fifth embodiment, this different point will be explained, and the explanation of the same point will be omitted.

***Description of operation***
The operation of the detection system 100 according to Embodiment 5 will be described with reference to FIGS. 9 and 17. FIG.
The operation procedure of the detection system 100 according to the fifth embodiment corresponds to the detection method according to the fifth embodiment. A program that realizes the operation of the detection system 100 according to the fifth embodiment corresponds to the detection program according to the fifth embodiment.

The search process (step S2 in FIG. 4) according to the second embodiment will be described with reference to FIG.
The processing from step S21 to step S23 and the processing from step S26 to step S27 in FIG. 9 are the same as those in the second embodiment.

(Step S24: Similarity calculation process)
The search unit 116 in each detection device 10 searches gallery images including the object included in the selected image from the gallery images captured before the reference time from the current time based on the feature amount of the selected image. That is, the search unit 116 calculates the similarity between the feature amount of the selected image and the gallery images acquired from the gallery images captured from the current time to the reference time before.

(Step S25: result transmission process)
The search unit 116 in each detection device 10 identifies the gallery image with the highest similarity calculated in step S24. The search unit 116, as shown in FIG. 17, identifies the most recently captured gallery image that includes the same object as the object included in the identified gallery image. The search unit 116 transmits the most recently captured gallery image, the camera ID and image capturing time of the gallery image, and the similarity calculated in step S24 to the search instruction device 20 .
Here, the search unit 116 can identify the most recently captured gallery image that includes the same object as the object included in the identified gallery image, using the results tracked in step S14 of FIG. It is possible.

*** Effect of Embodiment 5 ***
As described above, the detection apparatus 10 according to Embodiment 5 identifies a gallery image with a high degree of similarity using gallery images included from the current time to before the reference time, and selects the same object from the identified gallery images. Identify the most recently captured gallery image containing .
This makes it possible to perform a real-time search more appropriately. For example, when the orientation of the object in the recently captured gallery image is different from the orientation of the object in the query image, the similarity between the recently captured gallery image and the query image is low. It may not be possible to perform an appropriate search. However, in the detection device 10 according to Embodiment 5, past gallery images whose orientation of the object is close to the orientation of the object in the query image are specified, and the most recent gallery image containing the same object as the past gallery image is identified. identified. Therefore, real-time search can be appropriately performed.

<Modification 8>
In the fifth embodiment, the case of real-time search has been described. However, not limited to real-time search, when a time to search is specified, a gallery image with a high degree of similarity is specified using the gallery images included before and after the reference time from the specified specified time, and the specified A gallery image captured at a specified time and containing the same object may be identified from the gallery images obtained. As a result, it is possible to perform a more appropriate search from the gallery images at the specified time.

Note that "unit" in the above description may be read as "circuit", "process", "procedure", "process", or "processing circuit".

The embodiments and modifications of the present disclosure have been described above. Some of these embodiments and modifications may be combined and implemented. Also, any one or some may be partially implemented. It should be noted that the present disclosure is not limited to the above embodiments and modifications, and various modifications are possible as necessary.

100 detection system, 10 detection device, 11 processor, 12 memory, 13 storage, 14 communication interface, 15 electronic circuit, 111 acquisition unit, 112 detection unit, 113 feature amount calculation unit, 114 tracking unit, 115 query extraction unit, 116 search Unit, 117 Output Unit, 118 Statistics Calculation Unit, 119 Dimension Compression Unit, 20 Search Instruction Device, 21 Processor, 22 Memory, 23 Storage, 24 Communication Interface, 25 Electronic Circuit, 211 Query Selection Unit, 212 Image Selection Unit, 30 User terminal 40 Camera 50 Storage device 60 Search screen 61 Input field 62 Query image field 63 Gallery image field 90 Transmission path.

Claims (13)

a query extraction unit for extracting image data including an object from among a plurality of image data captured by a camera and satisfying a condition as a query image for the object;
Based on the selected image selected from the query image extracted by the query extracting unit, a gallery image that is at least part of the image data other than the query image among the plurality of image data is included in the selected image. a search unit for searching gallery images containing objects. The detection device according to claim 1 , wherein the query extracting unit extracts image data that satisfies a moving direction condition of a moving trajectory of the object as the query image. 3. The detection device according to claim 2, wherein the movement direction condition is set by an angle between a movement vector indicating the movement locus and a straight line in the width direction of the image data. 4. The query image according to any one of claims 1 to 3, wherein the query extraction unit extracts the query image from image data obtained by imaging with a specific camera among a plurality of image data obtained by imaging with a plurality of cameras. or the detection device according to item 1. 5. Any one of claims 1 to 4, wherein the search unit searches for gallery images including an object included in the selected image from among the gallery images whose image-taking time is included in a specified period. A detection device according to any one of the preceding paragraphs. The detection device further
A statistic about the feature amount of the object is calculated from the feature amount of the object calculated from each of the selected image and a related query image that is other image data including the same object as the selected image. Equipped with a statistic calculation unit for calculating,
6. The detection according to any one of claims 1 to 5, wherein the search unit searches for gallery images including objects included in the selected image based on the statistics calculated by the statistics calculation unit. Device. The detection device further
The selected image and the related query based on the feature amount of the object calculated from each of the selected image and a related query image that is one or more other image data including the same object as the selected image. A statistic for classifying images into one or more groups, and calculating statistics for the feature amounts of the object calculated from the image data included in the target group for each of the one or more groups. Equipped with a quantity calculation unit,
6. The search unit searches for gallery images including objects included in the selected image based on the statistics for each of the one or more groups calculated by the statistics calculation unit. The detection device according to any one of Claims 1 to 3. The detection device further
A query high-dimensional feature amount, which is a feature amount of the object included in the query image and is represented by a multi-dimensional vector, and a feature amount of the object included in the gallery image, wherein a plurality of a feature amount calculation unit that calculates a gallery high-dimensional feature amount that is a feature amount represented by a dimensional vector;
compressing the dimension of the query high-dimensional feature calculated by the feature calculating unit to generate a query low-dimensional feature, and compressing the dimension of the gallery high-dimensional feature calculated by the feature calculating unit; and a dimensional compression unit that generates a gallery low-dimensional feature amount by
The search unit narrows down the gallery images to be searched by comparing the query low-dimensional feature amount and the gallery low-dimensional feature amount for the selected image generated by the dimension compression unit. and searching for a gallery image including an object included in the selected image by comparing the query high-dimensional feature amount of the selected image with the gallery high-dimensional feature amount of the gallery image to be searched. Item 8. The detection device according to any one of Items 1 to 7. The detection device further
A query high-dimensional feature amount, which is a feature amount of the object included in the query image and is represented by a multi-dimensional vector, and a feature amount of the object included in the gallery image, wherein a plurality of a feature amount calculation unit that calculates a gallery high-dimensional feature amount that is a feature amount represented by a dimensional vector;
A query low-dimensional feature amount is generated by excluding elements that do not change even between different objects from the query high-dimensional feature amount calculated by the feature amount calculation unit, and the query low-dimensional feature amount calculated by the feature amount calculation unit A dimensional compression unit that generates a gallery low-dimensional feature amount by excluding elements that do not change even between different objects from the gallery high-dimensional feature amount,
The search unit compares the query low-dimensional feature amount and the gallery low-dimensional feature amount for the selected image generated by the dimensional compression unit to obtain a gallery image including an object included in the selected image. 8. The sensing device according to any one of claims 1 to 7, which searches for . The search unit searches for a gallery image including an object included in the selected image from among the gallery images captured between a specified time and a reference time before the specified time. 10. The detection device according to any one of claims 1 to 9, which outputs a gallery image including the same object as that contained in the retrieved gallery image and having the image taken at the specified time. extracting, from among a plurality of image data captured by a camera, image data that includes an object and that satisfies a condition as a query image for the object;
Based on a selected image selected from the query image, a gallery image including an object included in the selected image is retrieved from gallery images that are at least part of image data other than the query image among the plurality of image data. detection method. a query extraction process for extracting image data including an object from among a plurality of image data obtained by imaging with a camera, and image data satisfying a condition, as a query image for the object;
Based on the selected image selected from the query image extracted by the query extraction process, from a gallery image that is at least part of the image data other than the query image among the plurality of image data, included in the selected image A detection program that causes a computer to function as a detection device that performs search processing for searching gallery images containing objects. A detection system comprising one or more detection devices and a search instruction device,
each of the one or more sensing devices comprising:
a query extracting unit for extracting, from among a plurality of image data captured by a camera, image data that includes an object and that satisfies a condition as a query image for the object;
The search instruction device is
a query selection unit that selects a selection image from the query images extracted by the query extraction unit of each of the one or more sensing devices;
each of the one or more sensing devices comprising:
Based on the selected image selected by the query selection unit, a gallery image including an object included in the selected image is selected from a gallery image that is at least part of image data other than the query image among the plurality of image data. A detection system comprising a search unit for searching for .

Images