JP2021128540A - Information processing device, information processing method and program - Google Patents

Abstract

To improve the efficiency of a checking work by a user relating to candidate images that are check-up results.SOLUTION: An information processing device includes: an object detecting unit that detects an object from a picked-up video picture; a feature quantity generating unit that generates a feature quantity of the object on the basis of a feature extracted for each part that forms the object; a feature quantity check-up unit that checks up the feature quantity of the object which is a designated check-up source with the feature quantity of the object in the video picture; and a visibility determining unit that determines the visibility of the object in the video picture on the basis of the check-up result by the feature quantity check-up unit and of the information on the part.SELECTED DRAWING: Figure 3

Description

The present invention relates to an information processing device, an information processing method, and a program.

There is a system that enables monitoring of multiple points by installing multiple cameras at convenience stores, shopping malls, airports, etc. and connecting them via a network. For example, there is a usage such as searching for a specific person such as a shoplifter in an image captured by a certain camera a while ago from a plurality of current camera images. This can be achieved by collating features such as the color of clothes extracted from the person in the camera image. For example, by sorting the collation results in descending order according to the score and arranging thumbnail images including the person area as candidate images and presenting them to the user, the user can see whether the candidate person is the same person as the collation source or the candidate image. Can be confirmed.

Although thumbnail images can list a large number of results, the resolution is low, and it is not easy for the user to visually confirm the candidate images. When people overlap in a crowded scene, background objects such as shelves in a convenience store, and hiding by belongings such as umbrellas occur in the image, the area that can be used to confirm the candidate person in the thumbnail image becomes smaller. , Visibility is reduced. Patent Document 1 discloses a technique for assisting user confirmation by displaying a score for each part (eyes, nose, mouth, etc.) of a person in a pie chart or the like. Further, Patent Document 2 discloses a technique of calculating a score for each of a plurality of types of characteristics (face, clothes, gait, etc.) of a person and displaying the collation results for each sorted type in parallel. ..

Japanese Unexamined Patent Publication No. 2014-11523 JP-A-2019-16098

In the technique of Patent Document 1, when the visibility of the candidate image is low, it is easy for the user to confirm the candidate image from the score of each displayed part, but when there are a large number of candidate images, a large number of parts The score will be displayed, and the listability will be reduced. Further, the technique described in Patent Document 2 makes it possible to list a plurality of types of collation results, but the correspondence between the same candidate images is particularly visible because the user needs to compare and judge the collation results. Confirmation work efficiency is reduced in low candidate images.

The present invention has been made in view of such circumstances, and an object of the present invention is to improve the efficiency of confirmation work by a user regarding a candidate image of a collation result.

The information processing apparatus according to the present invention is designated as a detection means for detecting a subject from an captured image and a generation means for generating a feature amount of the subject based on features extracted for each part constituting the subject. Based on the collation means for collating the feature amount of the subject which is the collation source and the feature amount of the subject in the video, the collation result by the collation means, and the information about the parts, the visual recognition of the subject in the video is performed. It has a determination means for determining sex.

According to the present invention, it is possible to improve the efficiency of the confirmation work by the user regarding the candidate image of the collation result.

It is a figure which shows the configuration example of the system which applied the information processing apparatus in this embodiment. It is a figure which shows the hardware configuration example of the information processing apparatus in this embodiment. It is a figure which shows the functional structure example of the information processing apparatus in this embodiment. It is a flowchart which shows the example of the feature amount extraction processing in this embodiment. It is a flowchart which shows the example of the subject collation processing in this embodiment. It is a figure explaining the degree of identification of a part unit. It is a figure which shows an example of the image display in this embodiment. It is a figure which shows another example of the image display in this embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following, the subject to be processed will be described as the whole person (whole body), but the present invention is not limited to this, and the same applies to other subjects such as the face of a person and a vehicle. ..

FIG. 1 is a diagram showing a configuration example of a system to which the information processing apparatus according to the present embodiment is applied. The system in this embodiment includes an imaging device (camera) 101-103 and an information processing device 105. The imaging devices (cameras) 101 to 103 and the information processing device 105 are connected to the network 104, and can communicate data and the like with each other. Note that FIG. 1 shows an example of having three image pickup devices (cameras) 101 to 103, but the present invention is not limited to this, and the number of image pickup devices included in the system is arbitrary.

Each of the image pickup devices (cameras) 101 to 103 has an image pickup lens, an image pickup sensor such as a CCD or CMOS, a video signal processing unit, and the like, and captures an image. Further, the image pickup apparatus (camera) 101 to 103 transmits the captured image via the network 104. The information processing device 105 collates the subject in the camera image with the image received via the network 104. Thereby, the information processing device 105 can detect, for example, the movement of the subject between the cameras.

FIG. 2 is a block diagram showing a hardware configuration example of the information processing device 105 according to the present embodiment. The information processing device 105 in this embodiment includes a CPU 201, a ROM 202, a RAM 203, a secondary storage device 204, an input device 205, a display device 206, and a network I / F 207. The CPU 201, ROM 202, RAM 203, secondary storage device 204, input device 205, display device 206, and network I / F 207 are connected to each other via bus 208 so as to be communicable with each other.

The CPU (Central Processing Unit) 201 executes processing according to a program stored in the ROM 202 or the RAM 203. By executing the program, the CPU 201 controls, for example, each functional unit of the information processing apparatus 105, and performs various processes such as feature amount extraction processing and subject matching processing.

The ROM (Read Only Memory) 202 is a non-volatile memory, and stores a program for executing the process according to the present embodiment and other programs and data necessary for control. The RAM (Random Access Memory) 203 is a volatile memory, and stores temporary data such as frame image data and pattern discrimination results.

The secondary storage device 204 is a rewritable secondary storage device such as a hard disk drive, a solid state drive, and a flash memory, and stores image information, an image processing program, various setting information, and the like. This information is transferred to the RAM 203 and used when the CPU 201 executes the program.

The input device 205 is a keyboard, a mouse, or the like, and receives input from the user. The display device 206 is a cathode ray tube CRT, a liquid crystal display, or the like, and displays a processing result or the like to the user. The network I / F207 is an interface for connecting to a network such as the Internet or an intranet.

The information processing device 105 in the present embodiment reads software (program) that implements processing corresponding to each step of the flowchart described later from the secondary storage device 204, RAM 203, or the like, and executes the software (program) using the CPU 201.

FIG. 3 is a block diagram showing a functional configuration example of the information processing apparatus 105 according to the present embodiment. The information processing device 105 in the present embodiment includes an image acquisition unit 301, an image storage unit 302, a subject detection unit 303, a basic feature extraction unit 304, a mask extraction unit 305, a feature amount generation unit 306, and a feature amount storage unit 307. .. Further, the information processing device 105 includes a collation source designation unit 308, a feature amount collation unit 309, a visibility determination unit 310, and a display unit 311.

The image acquisition unit 301 acquires the image by receiving the image captured by the image pickup device (camera) 101 to 103 from the image pickup device (camera) 101 to 103 via the network 104.

The video storage unit 302 stores the video acquired by the video acquisition unit 301 together with the camera ID and the frame ID. The video storage 302 is composed of, for example, a RAM 203 or a secondary storage device 204. The camera ID is identification information (ID) for identifying an imaging device, and the frame ID is identification information (ID) for identifying a frame image in a video composed of a plurality of frame images captured by the imaging device. be. From the camera ID and the frame ID, it is possible to identify which frame image was taken by which imaging device.

The subject detection unit 303 detects the area of the subject from the image acquired by the image acquisition unit 301. The basic feature extraction unit 304 extracts the basic features in the image of the area of the subject detected by the subject detection unit 303. In the present embodiment, the basic feature extraction unit 304 extracts basic features such as color, edge, and texture from the image of the subject region detected by the subject detection unit 303. The mask extraction unit 305 extracts a mask image for masking an area other than a predetermined part constituting the subject. The parts that make up the subject include, for example, the head, arms, torso, and legs.

The feature amount generation unit 306 masks a feature map based on the basic features extracted by the basic feature extraction unit 304 using the mask image extracted by the mask extraction unit 305, and extracts the features of each part. .. In addition, the feature amount generation unit 306 generates the feature amount of the subject by connecting the extracted features of each part for each subject. At this time, the feature amount generation unit 306 reduces the dimensions of the extracted features of each part and connects them to obtain the feature amount of the subject as a vector having a predetermined dimension. The feature amount storage unit 307 stores the feature amount of the subject generated by the feature amount generation unit 306 in association with the information about the parts, the camera ID, and the frame ID. The feature amount storage unit 307 is composed of, for example, a RAM 203 or a secondary storage device 204.

The collation source designation unit 308 designates a subject to be a collation source in the subject collation process. The feature amount collating unit 309 reads out the feature amount of the subject from the feature amount storage unit 307 and collates it with the feature amount of the subject designated by the collation source designation unit 308. Further, the feature amount collation unit 309 reads a frame image from the video storage unit 302 based on the collation result, cuts out an image of the subject area, and generates a thumbnail image of a predetermined size.

The visibility determination unit 310 determines the visibility of the candidate image to be displayed as the collation result obtained by the feature amount collation unit 309. The display unit 311 displays the collation result by the feature amount collation unit 309 to the user by a display method based on the determination result by the visibility determination unit 310. The display unit 311 displays a thumbnail image according to the collation result by the feature amount collation unit 309 based on the determination result by the visibility determination unit 310. The display unit 311 is composed of, for example, a display device 206.

Next, the processing in the information processing apparatus 105 in this embodiment will be described.
First, a feature amount extraction process for generating a feature amount of a subject from a camera image will be described. FIG. 4 is a flowchart showing an example of the feature amount extraction process in the present embodiment.

In step S401, the image acquisition unit 301 acquires the image captured by the imaging devices (cameras) 101 to 103 via the network 104 in units of frame images together with the camera ID given in advance as the identification information of each camera. ..
Next, in step S402, the frame image acquired in step S401 is stored in the video storage unit 302 together with the camera ID and the frame ID.

Next, in step S403, the subject detection unit 303 detects the subject from the frame image acquired in step S401. As a specific method for detecting the region of the subject from the frame image, for example, there is the method described in Reference 1 below.
(Reference 1) U.S. Patent Application Publication No. 2007/0237387

The method described in Reference 1 scans a detection window of a predetermined size on an input image, and determines whether or not the pattern image obtained by cutting out the image in the detection window is a person or not. For this discrimination, AdaBoost is used to effectively combine many weak discriminators to form a discriminator, and the discriminant accuracy is improved. In addition, these discriminators are connected in series to form a cascade type detector. The weak discriminator is composed of HOG (Histograms of Oriented Gradients) features. Then, the cascade type detector first removes the candidate of the pattern that is clearly not the subject on the spot by using the simple discriminator in the previous stage. Then, only for the other candidates, it is determined whether or not the person is a person by using a complicated discriminator in the latter stage having higher discrimination performance. Since the scanning of the detection window is performed on the entire frame image, when a plurality of subjects are included in the frame image, all the subjects can be detected.

In the present embodiment, the subject is the whole person (whole body), but the above-mentioned method can be applied to other subjects. For example, when it is desired to treat a face as a subject, the discriminator described in Reference 1 may be made for the face. Further, the above-mentioned method for detecting a subject is an example, and the present invention is not limited to this. As a method for detecting the region of the subject, a method using a convolutional neural network (CNN) may be applied, or a background subtraction method or the like may be used.

By the method described above, the subject detection unit 303 can detect the region of the subject from the video (frame image). The subject area is represented by, for example, the x-coordinate and the y-coordinate of two points, the upper left and the lower right of the rectangle surrounding the person who is the subject, with the upper left of the frame image as the origin. The method of indicating the area of the subject is not limited to this, and any method capable of uniquely designating the area can be applied.

Next, in step S404, the basic feature extraction unit 304 extracts the basic features in the image of the area of the subject detected in step S403. The basic features in this embodiment are, for example, basic features such as color, edge, and texture. The basic feature extraction unit 304 uses a CNN as in Resnet described in Reference 2 below, and uses a feature map obtained as an output as a basic feature. The CNN is not limited to this, and may be a CNN of another architecture such as AlexNet or VGG. Alternatively, it may be a color histogram, an LBP (Local Binary Pattern) feature, a HOG feature, a feature extracted using a Gabor filter, a Schmid filter, or the like.

(Reference 2) K.K. He et al. "Deep Learning Learning for Image Recognition", Proceedings of IEEE Computer Vision and Pattern Recognition, pp.770-778, 2016

Next, in step S405, the mask extraction unit 305 extracts a part mask (mask image) related to a predetermined part constituting the subject. For example, as described in Reference 3 below, the mask extraction unit 305 uses the subject image and the parts mask image which is the correct answer data as the learning data, and classifies the parts (K types are assumed in the present embodiment). ) Create a CNN that outputs a score map. Then, a part mask image is obtained by performing threshold processing on the score map of the obtained parts.

(Reference 3) G. Oliveira, A.M. Valada, C.I. Bollen, W. et al. Burgard, and T. et al. Brox. Deep learning for human part discovery in images. ICRA, 2016

The process in step S405 is not limited to this method as long as a part mask image showing a region of the part can be obtained. For example, as described in Reference 4 below, a set of three data sets of a person image of the same person as the collation source person image and a person image of another person is prepared, and a predetermined number (K) of parts detectors are used. There is also a way to learn.

(Reference 4) L. Zhao et al. "Deeply-Learned Part-Aligned Representations for Person Re-Identification," IEEE International Conference on Computer Vision (ICCV), 2017

Further, a method of creating a parts detector by a method as described in Reference 1 can be considered. In this case, since only the circumscribed rectangle of the part can be obtained instead of the part mask image, the pixels other than the part are reflected in the feature amount. However, the processing according to the gist of the present invention is feasible, and the same effect can be obtained.

Next, in step S406, the feature amount generation unit 306 masks the feature map obtained by the basic feature extraction unit 304 in step S404 with the mask obtained by the mask extraction means 305 in step S405, and features of each part. Is extracted. In the example of this embodiment, since there are K parts, K features are extracted.
Subsequently, in step S407, the feature amount generation unit 306 reduces all the K features extracted in step S406 to the dimension number D and concatenates them, so that the subject feature amount of the (K × D) dimensional vector is the subject feature amount. To get.

In the above description, the feature map is generated first and then the parts mask is created. However, the parts mask may be created first and the features may be extracted from the masked image. ..

Next, in step S408, the feature amount of the subject obtained in step S407 is stored in the feature amount storage unit 307 in association with each dimension of the feature amount of the subject and the information of the parts obtained from the feature amount. In the present embodiment, as an example, the part information is a part type ID and a part area. The center position coordinates of the part or the part mask image itself may be used as the part information. Regarding the association between the feature amount and the part information, for example, in the feature amount vector, the 1st dimension to the D dimension are the feature amount information of the part K1, and the (D + 1) dimension to the (2D) dimension are the feature amount information of the part K2. The dimension may be used as an index. In order to create a thumbnail image later, the coordinates of the subject area, the subject ID for identifying the subject, and the frame ID and camera ID of the current processing target are also associated with the feature amount of the subject.

The information processing apparatus 105 repeats the processes of steps S404 to S408 until it is determined in step S409 that the processes for all the subjects detected in step S403 have been completed. That is, the information processing device 105 repeats the processes of steps S404 to S408 until the feature amount of the subject is extracted from all the subjects in the frame image.

Further, the information processing apparatus 105 repeats the processes of steps S401 to S409 until it is determined in step S410 that the processes of all the cameras capable of acquiring images via the network 104 are completed.
The above is the feature amount extraction process for the subject, and it is assumed that all the processes are repeated until the user gives an instruction to end the feature amount.

Next, a subject collation process for collating a subject will be described. FIG. 5 is a flowchart showing an example of subject matching processing in the present embodiment.

In step S501, the collation source designation unit 308 accepts a designation from the user of the person (query subject) to be collated. For the designation from the user, for example, the camera image is displayed on the display device 206, and the instruction is given by using the mouse of the input device 205 or the like. Then, based on the instructed camera ID of the camera and the subject ID of the subject, the feature amount of the subject of the target person is read from the feature amount storage unit 307 and temporarily stored in the RAM 203.

Next, in step S502, the feature amount matching unit 309 reads out the feature amount of the subject from the feature amount storage unit 307 together with the information of the associated parts.
Next, in step S503, the feature amount collating unit 309 collates the feature amount of the subject of the collation source specified by the user in step S501 with the feature amount of the subject read out in step S502.

In the present embodiment, as an example, the Euclidean distance is used for collation of the feature amount, but other distance indexes such as the L1 distance and the cosine distance may be used. Then, the distance between the two features is converted into a score indicating the degree of collation. In this embodiment, the score is represented in the range of 0 to 1000. The minimum value (referred to as d_min) of the distance of a single part is experimentally obtained, and the distance corresponding to the maximum score of 1000 is defined as (K × d_min). The score S can be obtained by the following (Equation 1), assuming that the distance between the subject features is a dust.

Incidentally, the in step S503, for use in the visibility determination to be performed after, the feature checker 309, keep also calculates the score S _K parts units in addition to the score on all parts is an overall person (systemic). Score S _K of each part unit, it can be obtained by substituting the K = 1 in (Equation 1), the distance of the part itself in dist.

In step S504, the feature amount collation unit 309 determines whether or not the feature amount collation processing of steps S502 to S503 has been performed on the feature amounts of all the subjects in the frame image acquired from the camera to be collated. If it is determined that there is a subject in the frame image that has not been subjected to the feature amount matching process (NO), the process returns to step S502, and the feature amount matching unit 309 determines the feature amount of the unprocessed subject. , The feature amount collation processing of steps S502 to S503 is performed. On the other hand, if it is determined that the feature amount collation processing has been completed for all the subjects in the frame image (YES), the process proceeds to step S505.

In step S505, the feature amount collation unit 309 sorts all the candidates of the collation result obtained by the feature amount collation process in descending order of the score. It should be noted that a candidate having a score less than a predetermined threshold value may be determined not to be the same person (not similar) as the collation source and may be excluded from the processing target in the subsequent processing steps.

Next, in step S506, the feature amount collation unit 309 reads out the frame image from the video storage unit 302 based on the camera ID and the frame ID for all the candidates of the collation result. Then, the feature amount collating unit 309 cuts out the image of the subject area from the frame image read from the image storage unit 302 based on the coordinates of the subject area, performs reduction processing and the like, and resizes the image to a predetermined size to display the image. Generate a thumbnail image as a candidate image.

Next, in step S507, the visibility determination unit 310 determines the visibility of the subject in the candidate image based on the collation result obtained by the feature amount collation unit 309. Here, there is a correlation between the high visibility of the candidate image and how much the subject is visible. The visibility of the candidate image is highest when the entire subject is visible, and the more hidden or affected by lighting (such as the shadow of a building), the smaller the clearly visible area. .. If the influence of hiding or the like is large, the score will not increase in the subject matching process and the person will not be determined as the person himself / herself. Therefore, in the present embodiment, it is estimated from the score of each part how much each of the parts constituting the subject is visible. Visibility is evaluated by using the ratio of the sum of the areas of the parts having a high score to the area of the entire subject as an amount indicating how much the subject is visible.

Visibility also depends on the type of part. In general, humans have a high ability to discriminate faces. In addition, the upper body has many variations in clothing compared to the lower body, so it is relatively easy to identify. Therefore, a value (identification ease) indicating the ease of identification is defined in advance for each type of part. The value is specified so that the total sum is 1 as shown in FIG. 6 as an example.

The visibility V representing the degree of visibility in the present embodiment from the above, the area A _K parts, the score S _K parts units, as identification Simplicity E _K parts units, defined as (Equation 2).

The visibility determination unit 310 temporarily stores the calculated visibility V in the RAM 203 in association with the current candidate image.

Next, in step S508, the visibility determination unit 310 determines whether or not the visibility determination processing has been performed on all the candidate images. If it is determined that there is a candidate image for which the visibility determination process has not been performed (NO), the process returns to step S507, and the visibility determination unit 310 performs the visibility determination process for the unprocessed candidate image. On the other hand, if it is determined that the visibility determination process has been completed for all the candidate images (YES), the process proceeds to step S509.

In step S509, the display unit 311 displays the thumbnail image (candidate image) generated in step S506 based on the visibility V calculated in step S507. Here, there are a plurality of methods for displaying the candidate image based on the visibility V, and an example is shown below.

The first method is a method of displaying the candidate images whose visibility V is equal to or less than a predetermined value. For example, if the user rushes to the site immediately after confirming the collation result, it is not necessary to scrutinize all the candidates and there is not enough time. In such a case, reducing the number of candidate images to be displayed reduces the burden on the user. Therefore, candidate images with low visibility are thinned out and displayed based on the visibility V.

The second method is a method of highlighting a candidate image having a visibility V of a predetermined value or less. If you want to investigate the behavior of the person you are collating with, you need to scrutinize all the candidates. In such a case, highlight which candidate image has low visibility and clearly indicate to the user that the candidate image should be carefully confirmed.

The second method will be described with reference to FIG. 7A. In FIG. 7A, 701 is a window for displaying the collation result. Reference numeral 702 is a thumbnail image of the collation source (query). Reference numeral 703 is an area for displaying thumbnail images of candidates (gallery), and in the example shown in FIG. 7A, five candidate images are displayed. In image 704, the head is hidden by an umbrella, and in image 705, a part of the leg is hidden by overlapping with another person, so that the visibility is reduced (the visibility V is equal to or less than a predetermined value). ). The images 704 and 705 are displayed with the marks 706 and 707 indicating that the visibility is reduced.

In the example shown in FIG. 7A, a specific mark is given to the candidate image having low visibility, but another method may be used as long as the user can grasp the candidate image having low visibility. For example, an outer frame may be attached to the thumbnail image. Alternatively, the value of the visibility V may be displayed below each thumbnail image, and if it is less than a predetermined value, it may be changed to a deficit or the like.

The third method is a method of displaying a candidate image having a visibility V of a predetermined value or less and a query image of a collation source in parallel. Since it becomes easy to compare the candidate with low visibility with the collation source, the confirmation work by the user becomes easy.

The third method will be described with reference to FIG. 7 (b). In FIG. 7B, 711 is a window for confirming a candidate image having low visibility, and is displayed as a window different from the window 701 shown in FIG. 7A. Reference numeral 712 is a thumbnail image of a collation source (query), and reference numeral 713 is an area for displaying a thumbnail image of a candidate (gallery) having low visibility. Since the image 712 and the image 713 are displayed next to each other, comparison is easy. Further, the image 712 and the image 713 may be enlarged and displayed. At this time, the super-resolution processing as described in Reference 5 may be performed to further facilitate confirmation. Further, in the third method, the user can continuously perform the confirmation work by displaying a plurality of candidates having low visibility together.

(Reference 5) J. Kim, J.M. K. Lee, and K. M. Lee. "Accurate Image Super Resolution Using Very Deep Convolutional Networks", IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016

In addition, by providing a subject tracking unit that determines which subject detected in the previous frame corresponds to the subject detected in the current frame, an image of the subject area of the candidate front and rear frames with low visibility is displayed. You may try to do it. There are various methods for tracking processing. For example, there is a method of associating the center position of the subject area included in the previous frame with the center position of the subject area included in the current frame. In addition to this, any method may be used as long as the subjects between the frames can be associated with each other, such as a method by pattern matching using the subject area of the previous frame as a matching pattern. Since the same subject ID is assigned to the subject associated between the frames, the image of the subject area of the front and rear frames can be read out based on the subject ID.

The display method in this case will be described with reference to FIG. In FIG. 8, 801 is a window showing details of the candidate image selected by an instruction with a cursor or the like in the image display shown in FIG. 7B, for example. The selected candidate thumbnail images 804 and the thumbnail images 803 and 805 of the same subject extracted from the frames before and after the selected candidate are displayed side by side in chronological order. In a series of movements, in the frame image from which features have not been extracted, there may be a display such as image 805 with less hiding than image 804. In addition, by presenting a series of images, an effect of making it easier to confirm than a single thumbnail image can be expected. Further, the images in the images different in time may be played back as a moving image instead of being displayed side by side at the same time.

In the present embodiment, the whole body of a person is taken as an example as a subject, but it can also be applied to other objects. For example, it is effective in a case where the face is partially covered with accessories such as sunglasses and a mask, and a case where a part of the face is temporarily hidden by a hand or the like.

As described above, according to the present embodiment, it is possible to determine the visibility of the candidate image based on the collation score and the ease of identification of each part constituting the subject. In addition, it becomes possible to display a candidate image having low visibility so that the user can easily check it. This makes it possible to improve the efficiency of the user's confirmation work of the candidate image.

In the above-described embodiment, the image received from the image pickup device via the network is processed by the information processing device, but some functions of the information processing device are incorporated into the image pickup device to distribute the processing. You may. For example, the image pickup device may perform up to the detection of the subject, and the image pickup device may transmit the image of only the subject area from the image pickup device to the information processing device via the network.

(Other Embodiments of the present invention)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

It should be noted that the above-described embodiments are merely examples of embodiment of the present invention, and the technical scope of the present invention should not be construed in a limited manner by these. That is, the present invention can be implemented in various forms without departing from the technical idea or its main features.

101-103: Imaging device 104: Network 105: Information processing device 301: Image acquisition unit 302: Image storage unit 303: Subject detection unit 304: Basic feature extraction unit 305: Mask extraction unit 306: Feature amount generation unit 307: Feature amount Storage unit 308: Collation source designation unit 309: Feature amount collation unit 310: Visibility determination unit 311: Display unit

Claims (15)

A detection means that detects the subject from the captured image,
A generation means for generating the feature amount of the subject based on the features extracted for each part constituting the subject, and
A collation means for collating the feature amount of the subject, which is the designated collation source, with the feature amount of the subject in the image.
An information processing device including a determination unit for determining the visibility of the subject in the video based on a collation result by the collation means and information on the part. The information processing apparatus according to claim 1, wherein the determination means determines the visibility of the subject in the image based on the collation result of each part. The information processing apparatus according to claim 1 or 2, wherein the information about the part includes at least one of the area of the part and the ease of identification of the part. The claim means that the determination means calculates the visibility based on the collation result of each part and the information about the part, and determines the visibility of the subject in the image based on the calculated visibility. Item 3. The information processing apparatus according to item 3. 4. The information processing apparatus according to item 1. The information processing apparatus according to any one of claims 1 to 5, further comprising a display means for displaying an image according to the collation result by the collation means based on the determination result by the determination means. .. The information processing apparatus according to claim 6, wherein the display means displays an image of a subject in the image similar to the subject of the collation source. The information processing apparatus according to claim 7, wherein the display means selects an image to be displayed based on the determination result of the determination means. The information processing device according to claim 7, wherein the display means highlights an image based on the determination result of the determination means. The information processing device according to claim 7, wherein the display means enlarges and displays an image based on the determination result of the determination means. The information processing apparatus according to claim 7, wherein the display means performs super-resolution processing on an image based on the determination result of the determination means. The information according to claim 7, wherein the display means displays in parallel an image of the subject of the collation source and an image of the subject in the video based on the determination result of the determination means. Processing equipment. The information processing apparatus according to claim 7, wherein the display means simultaneously displays images of subjects in images that differ in time based on the determination result of the determination means. A detection process that detects the subject from the captured image,
A generation step of generating a feature amount of the subject based on the features extracted for each part constituting the subject, and
A collation process for collating the feature amount of the subject, which is the designated collation source, with the feature amount of the subject in the image.
An information processing method comprising a determination step of determining the visibility of the subject in the video based on the collation result in the collation step and information on the part. A detection step that detects the subject from the captured image,
A generation step of generating a feature amount of the subject based on the features extracted for each part constituting the subject, and
A collation step for collating the feature amount of the subject, which is the designated collation source, with the feature amount of the subject in the image,
A program for causing a computer to perform a determination step of determining the visibility of the subject in the video based on the collation result in the collation step and information on the part.

Images