JP7240940B2 - Object image extraction device, method, and software program - Google Patents

Description

The present invention relates to a technique for extracting a human image from a video intended for use in deep learning.

In the field of computer vision, a method of recognizing an object from an image using deep learning has attracted attention.

Deep learning is a technique that attempts to realize functions similar to the learning ability that humans naturally perform in computers. Compared to conventional methods such as pattern matching, deep learning has greater flexibility in feature analysis and feature expression, and it does not require humans to define the features of objects targeted for detection. have the advantage of

On the other hand, the recognition accuracy of a recognition model obtained by deep learning is greatly affected by the quantity and quality of teacher images used during learning. If the number of teacher images is small, the discrimination model tends to have a low detection rate that responds only to objects that are very similar to the detection targets included in the teacher images referred to in learning. When a large amount of noise other than the detection target is reflected in the background of the teacher image, the discrimination accuracy of the discrimination model tends to be low.

For this reason, there is a demand for the development of a technique that enables efficient automatic extraction of teacher images with little noise other than detection targets and effective use in deep learning of discrimination models.

Japanese Patent Application Laid-Open No. 2002-200002 discloses a technique of extracting and saving an image group including an object corresponding to the designated object classification from a retained image database based on an object classification (person, etc.) arbitrarily designated by the user. there is

JP 2008-299681 A

The technique disclosed in Patent Document 1 is an image extraction technique that extracts an object arbitrarily designated by a user from a general video group and returns the extraction result in units of frames. The returned frames are likely to contain background noise other than the specified type of object, and may not be suitable for use as training data for deep learning.

One object of the present disclosure is to provide a technique that enables extraction of an image of an object suitable for deep learning.

A target object image extracting device according to one aspect is a target object image extracting device for extracting an image of a target object from frames of an image included in a video, wherein some frames of the video are used as key frames, and the key a key frame object specifying unit for acquiring an image of an object area specified by a rectangle including a portion where the object is displayed in the frame; and an intermediate frame image extracting unit for extracting an image of a rectangular target object area including a portion where the target object is displayed in the intermediate frame, which is a frame that is not the key frame in the above.

According to the present disclosure, an image of an object suitable for deep learning can be extracted.

It is a figure which shows the structural example of the teacher image extraction apparatus which concerns on this embodiment. It is a figure which shows the operation example of a teacher image extraction apparatus. FIG. 4 is a diagram for explaining the relationship between key frames and intermediate frames; FIG. 10 is a diagram showing an example of extracting a person image from key frames and intermediate frames; FIG. 10 is a diagram showing a first example of extracting a plurality of person images from key frames and intermediate frames; FIG. 4 is a diagram for explaining the relationship between a person image extracted from an input video and an extracted image set; FIG. 10 is a diagram showing a second example of extracting a plurality of person images from key frames and intermediate frames; 3 is a block diagram showing details of an intermediate frame image extraction unit and a human area identification unit; FIG. FIG. 10 is a diagram for explaining an operation example of a region vector grouping unit included in the pedestrian processing unit; FIG. 11 is a diagram for explaining an example in which the setting of a person area for a pedestrian is partially cut off; FIG. 10 is a diagram showing an example of correcting a person area based on the tip position of the left leg; FIG. 10 is a diagram showing an example of correcting a person area based on the tip position of the right foot; FIG. 10 is a diagram showing an example of correcting a person area based on tip positions of both feet; It is a figure which shows the operation example of the slight motion vector grouping part contained in a stationary person process part. FIG. 10 is a diagram for explaining an operation example of a adopted image determination unit; FIG. 10 is a diagram showing an example of a learning error rate curve referred to by the adopted image determination unit; 1 is a diagram showing a first example of a teacher image extraction system including a teacher image extraction device; FIG. FIG. 10 is a diagram showing a second example of a teacher image extraction system including a teacher image extraction device;

Embodiments will be described below with reference to the drawings.

FIG. 1 is a diagram showing a configuration example of a teacher image extraction device 10 according to this embodiment. Note that the teacher image extracting device 10 is an example of a target object image extracting device that extracts an image of a target object.

The teacher image extracting device 10 includes a key frame person specifying unit 101 , an intermediate frame image extracting unit 102 , a person region specifying unit 103 , a adopted image determining unit 104 and a teacher image storing unit 105 .

The keyframe person designation unit 101 receives designation of a person to be extracted in a keyframe among a plurality of frames forming an input video (moving image) 100 . A key frame is a frame positioned at predetermined intervals among a plurality of frames. For example, the user manually designates a rectangular area so that the person to be extracted is included in the keyframe.

The keyframe interval may be set arbitrarily. The interval between keyframes may be set to a fixed interval, such as 5 seconds or 10 seconds. For example, if the keyframe interval is set to 5 seconds for a 30 fps video, the number of frames in 5 seconds is 150 (=30 frames×5 seconds). Among them, the first frame is called a key frame, and the 149 frames following the key frame are called intermediate frames. Hereinafter, a set of keyframes and subsequent intermediate frames may be referred to as a keyframe set.

The keyframe intervals may not be constant intervals, and for example, different keyframe intervals may be combined. Alternatively, only the first frame in the entire video may be set as a key frame, and the following remaining frames may be set as intermediate frames.

The intermediate frame image extraction unit 102 tracks the same person as the person specified for the key frame in each intermediate frame following the key frame. Then, the intermediate frame image extracting unit 102 specifies an area including the same person from each intermediate frame, and extracts an image of the specified area. Hereinafter, an area including a person will be referred to as a "person area", and an image obtained by extracting (cutting out) the person area will be referred to as a "person image".

The person region identification unit 103 identifies the person region in the intermediate frame in cooperation with the intermediate frame human part image extraction unit 102 . For example, the human region identification unit 103 identifies the human region so as to include the entire body of the person to be extracted. In other words, the person area identifying unit 103 identifies the person area so that a part of the body of the person to be extracted does not protrude.

The adoption image determination unit 104 determines whether or not to adopt the person image extracted by the intermediate frame image extraction unit 102 as a teacher image for deep learning of the person identification model. For example, the adopted image determination unit 104 determines whether or not an improvement in the accuracy of the person identification model can be expected if the extracted person image is used as a teacher image for deep learning of the person identification model. error rate). Then, the adopted image determination unit 104 adopts the person image determined to be expected to improve accuracy as a teacher image for learning the person identification model.

A teacher image storage unit 105 stores the person image adopted by the employed image determination unit 104 as a teacher image 106 for learning a person identification model. The teacher image storage unit 105 may store, as a teacher image, not only the person image extracted from the intermediate frame, but also the person image extracted from the area specified for the key frame.

FIG. 2 is a diagram showing an operation example of the teacher image extracting device 10. As shown in FIG.

A user inputs a video (moving image) 100 used for extracting a person image to the teacher image extraction device 10 . The input image 100 may be a file stored in a recording medium such as an HDD (Hard Disk Drive). Alternatively, the input image 100 may be an image being captured by a camera or an image received by streaming via a network. Alternatively, the input video 100 may be a collection of a plurality of continuous image files in which all frames forming one moving image are expanded.

A user designates a person to be extracted for a keyframe in the input video 100 through the keyframe person designation unit 101 .

The intermediate frame image extraction unit 102 extracts images (person images) of the same person as the person specified by the key frame person specification unit 101 in cooperation with the person area specification unit 103 , and outputs them as an extraction image set 202 .

Based on the learning error rate, the adopted image determination unit 104 determines whether or not to adopt the extracted image set 202 as teacher images for learning the person identification model. The adopted image determination unit 104 outputs the extracted image set determined to be adopted as the teacher image as the teacher image set 203 .

The teacher image set 203 is used as teacher images for learning a person identification model. The learning of the person identification model may be either learning for generating a new person identification model or re-learning for improving the accuracy of the generated person identification model.

FIG. 3 is a diagram for explaining the relationship between key frames and intermediate frames. Note that FIG. 3 is an example in which the keyframe interval 312 is 6 frames.

The user operates a mouse or the like to designate a person area 310 surrounding the person 308 in the keyframe 300 .

In this way, the user designates the person region 310 surrounding the person 308 for the keyframe 300, thereby improving the tracking accuracy and image extraction accuracy of the same person in intermediate frames following the keyframe. That is, the teacher image extracting apparatus 10 includes a key frame person specifying unit 101 that accepts specification of a person region for a key frame from a user, and an intermediate frame image extracting unit 102 that automatically extracts a person image from each intermediate frame. Therefore, a large number of high-quality human teacher images can be obtained from the input video 100 .

Note that when the keyframe interval is wide, a person who does not exist in the keyframe may newly appear from an intermediate frame in the middle. In this way, a person newly appearing in an intermediate frame may be detected by the motion estimation person detection unit 601 (see FIG. 8) included in the intermediate frame image extraction unit 102 .

In addition, designation of a person region for a key frame is not limited to the above-described manual operation. For example, a person in a key frame may be automatically detected by processing similar to that of the motion estimation person detection unit 601 . Details of the motion estimation person extraction unit 601 will be described later.

FIG. 4 is a diagram showing an example of extracting a person image from key frames and intermediate frames.

The user designates a person area 316 surrounding a person 315 with respect to the key frame 313 . In this case, the intermediate frame image extraction unit 102 automatically traces the person 315 and the same person 317 from the succeeding intermediate frame 314, using the person region 316 specified for the key frame as a base point, and extracts the same person 317. Identify the person region 318 to surround. Then, the intermediate frame image extraction unit 102 extracts a human image from the specified human region 318 .

FIG. 5 is a diagram showing a first example of extracting a plurality of person images from key frames and intermediate frames.

The user designates each person 321 to 323 for the key frame 319 . The intermediate frame image extracting unit 102 automatically tracks each same person in the succeeding intermediate frame 320 using each of the persons 321 to 323 specified for the key frame as a base point, and extracts a person area 324 to 324 surrounding each same person. 326. Then, the intermediate frame image extraction unit 102 extracts a person image from each of the specified person areas 324 to 326. FIG.

FIG. 6 is a diagram for explaining the relationship between a person image extracted from an input video and an extracted image set.

As shown in FIG. 6, the intermediate frame image extraction unit 102 automatically extracts a plurality of person images 409 to 413 and 415 from the intermediate frames 401 to 405 and 407, and saves them as a teacher image set 202 on a recording medium. .

The intermediate frame image extraction unit 102 also stores the human images 408 and 414 specified for the key frames 400 and 406 together with the human images 409 to 413 and 415 extracted from the intermediate frames 401 to 405 and 407, respectively.

FIG. 7 is a diagram showing a second example of extracting a plurality of person images from key frames and intermediate frames.

The first example described with reference to FIG. 5 was an example of extracting a plurality of persons from key frames and intermediate frames for each frame. In contrast, a second example, described with reference to FIG. 7, is an example of tracking and extracting the same person from subsequent intermediate frames for each person specified for a keyframe. As a result, the same person can be extracted with higher accuracy.

As shown in FIG. 7, starting from each of the three persons 503, 506, 509 in the keyframe 500, from subsequent intermediate frames 501, 502, the person images of the same person are tracked and extracted.

For example, the same person is tracked based on the person 503 in the key frame 500, the person image of the same person 504 is extracted from the first intermediate frame 501, and the person image of the same person 505 is extracted from the second intermediate frame 502. do.

Next, the same person is tracked based on the person image 506 of the key frame 500, the person image of the same person 507 is extracted from the first intermediate frame 501, and the person image of the same person 508 is extracted from the second intermediate frame 502. to extract

Next, the same person is tracked based on the person image 509 of the key frame 500, the person image of the same person 510 is extracted from the first intermediate frame 501, and the person image of the same person 511 is extracted from the second intermediate frame 502. to extract

FIG. 8 is a block diagram showing the details of the intermediate frame image extraction unit 102 and the human area identification unit 103. As shown in FIG.

The intermediate frame image extraction unit 102 includes a motion estimation person detection unit 601 , a vector stabilization filter 602 , an area correction unit 609 , an inter-frame area difference determination unit 610 and a person image extraction unit 611 .

Human region identification unit 103 includes region vector grouping unit 603 , whole body map generation unit 604 , front/back frame verification unit 605 , slight movement vector grouping unit 606 , slight movement edge extraction unit 607 , and time series edge strength verification unit 608 .

The motion estimation person detection unit 601 performs motion vector calculation on the input image 100 based on optical flow to detect the motion vector of the person. A motion vector is a vector that indicates the direction and amount of shift between frames of each block that divides a frame into predetermined units. The size of blocks (distance between blocks) is appropriately set according to system operation. Optical flow is an image processing technology that calculates how multiple feature points such as edges move between two images, estimates the movement of the target object, and recognizes the target object. is.

A vector stabilization filter 602 stabilizes the motion vector of the person detected by the motion estimation person detection unit 601 by using a Kalman filter to suppress variation between frames. The Kalman filter is one of state estimation techniques for predicting future states using a plurality of observation data containing errors. Since the Kalman filter has the property of converging the prediction error within a certain range, the vector stabilization filter 602 can utilize this property to stabilize the motion vector output.

The vector stabilization filter 602 outputs the stabilized vector to the pedestrian processing unit 613 when the prediction error is relatively large (for example, a predetermined threshold or more), and the prediction error is relatively small (for example, less than a predetermined threshold). ), it outputs to the stationary person processing unit 614 .

The pedestrian processing unit 613 has an area vector grouping unit 603 and a whole body map generation unit 604 .

A region vector grouping unit 603 bundles (groups) adjacent motion vectors having the same tendency among the plurality of motion vectors stabilized by the vector stabilization filter 602 to generate a vector group.

A whole body map generation unit 604 generates a map showing the whole body of a person (hereinafter referred to as a “whole body map”) from the vector group generated by the region vector grouping unit 603 .

The stationary person processing section 614 has a slight movement vector grouping section 606 , a slight movement edge extraction section 607 and a time series edge strength verification section 608 .

A fine motion vector grouping unit 606 uses, as a marker, an area obtained by bundling fine motion vectors having the same tendency and having a small amount of motion among the plurality of motion vectors stabilized by the vector stabilization filter 602 . Then, the fine movement vector grouping unit 606 bundles the marker groups whose areas overlap between a plurality of adjacent frames to generate a vector group.

A fine movement edge extraction unit 607 extracts an edge image from each of successive frames in the time direction, averages the extracted edge images, and obtains one average edge image.

A time-series edge strength verification unit 608 determines whether or not the average edge image obtained by the fine movement edge extraction unit 607 has an edge component (strength) equal to or greater than a predetermined reference within the vector group frame. Thereby, the presence or absence of a stationary person is determined.

The front/rear frame verification unit 605 smoothes the whole-body map output from the pedestrian processing unit 613 or the edge image output from the stationary person processing unit 614 using the front/rear frames in a predetermined range. Determine the person region in the frame.

When the pedestrian processing 613 is performed, the area correction unit 609 detects the tip positions of the person's feet and corrects the area of the whole body map. Since the amount of motion vector at the tip of the foot in the whole body is observed to be relatively large, this correction can suppress the extraction of a partially missing person image.

The inter-frame region difference determination unit 610 determines an average center-of-gravity movement amount between frames from the center of gravity of the person detected for each frame. Then, the inter-frame region difference determining unit 610 calculates the difference between this average center-of-gravity movement amount and the human center-of-gravity movement amount in the frame of interest, and if the difference is less than a predetermined threshold, selects the human region of the frame of interest. . On the other hand, if the difference is equal to or greater than the predetermined threshold value, the inter-frame area difference determining section 610 does not have to select the person area of the frame of interest.

A human image clipping unit 611 clips a human image from the human region selected by the inter-frame region difference determination unit 610 and outputs 612 it as a teacher image.

FIG. 9 is a diagram for explaining an operation example of the area vector grouping unit 603 included in the pedestrian processing unit 613. As shown in FIG.

In order to identify a moving person (for example, a pedestrian), the area vector grouping unit 603 bundles motion vectors of similar movement direction and movement amount close to each other and predicts the person area of the moving person. It should be noted that adjacent motion vectors may be adjacent blocks of two motion vectors. Also, two motion vectors having the same movement direction may form an angle less than or equal to a predetermined angle. Here, the predetermined angle is appropriately set according to system operation. For example, the predetermined angle may be set to an angle that can be regarded as substantially the same as the motion of each part of the person if the angle formed by the motion vectors is equal to or less than the predetermined angle. Also, motion vectors having the same amount of movement may have a difference in magnitude of two motion vectors that is less than or equal to a predetermined value. Here, the predetermined value is appropriately set according to system operation. For example, the predetermined value may be set to a value that can be regarded as substantially the same as the motion of each part of a person if the difference in motion vector magnitude is equal to or less than the predetermined value.

For example, as shown in FIG. 9A, when a person 701 is moving in a direction 700, the region vector grouping unit 603 performs the following processing. In other words, the area vector grouping unit 603 observes a motion vector group 702 having substantially the same direction and amount of movement as the direction 700 of each feature point by optical flow.

At this time, the region vector grouping unit 603 bundles similar motion vectors that are adjacent within a predetermined range for each feature point (704). Then, the area vector grouping unit 603 sets a virtual circle 705 containing the grouped motion vectors, as shown in FIG. 9(b).

The area vector grouping unit 603 executes the process (b) in FIG. 9 for all feature points included in the person 701 . Region vector grouping section 603 then superimposes virtual circles set as shown in FIG. 9(c) to obtain whole body map 706 representing the whole body.

The region vector grouping unit 603 sets a rectangular human region 707 surrounding the whole body map 706, as shown in FIG. 9(d). Then, the area vector grouping unit 603 extracts the person image 701 from the person area 707 and outputs it as a teacher image.

10 to 13 are diagrams for explaining an operation example of the area correction unit 609 included in the intermediate frame image extraction unit 102. FIG.

In the case of a pedestrian, a relatively large amount of motion vector is observed at the tip of the foot in the whole body. Therefore, the area correction unit 609 detects the tip positions of the feet and corrects the person area of the whole body map.

As illustrated in FIG. 10, the region vector grouping unit 603 may not be able to observe effective feature points. In this case, there is a portion 802 where the virtual circle for generating the whole body map cannot be set, and the whole body map generation unit 804 generates a partially missing whole body map. That is, a rectangular person area 801 is set in which a portion 802 of the person 800 is cut off.

Here, there is a high probability that the upper half of the body during walking is located inside the tips of the feet. Therefore, the area correction unit 609 detects the tip position 804 of the foot, assumes that the detected tip position 804 of the foot is the edge of the person 800 , and corrects the rectangular person area 801 . This increases the probability that the whole body of the person 800 is included inside the rectangular person region 801 .

FIG. 11 shows an example in which the region correction unit 609 performs correction to expand the left portion of the person region 805 of the person 803 based on the detected tip position 804 of the left foot.

FIG. 12 shows an example in which the region correction unit 609 performs correction to expand the right portion of the person region 808 of the person 806 with reference to the detected tip position 807 of the right foot.

FIG. 13 shows an example in which the region correction unit 609 performs correction to expand both the left and right portions of the person region 812 of the person 809 based on the detected left leg tip position 810 and right leg tip position 811. indicates

For example, the area correction unit 609 performs correction to extend the left side of the central axis of the person area if the tip position of the foot is to the left of the central axis of the human body map in the vertical direction before correction. conduct. The area correction unit 609 performs correction to extend the right side of the central axis of the human area if the tip position of the foot is to the right of the central axis of the human body map before correction.

A certain area of the upper body of a pedestrian walking while slouching may protrude outside the tip of the foot. In this case, the region correction unit 609 may determine that the pedestrian is not walking upright, and may not correct the person region based on the above-described foot tip positions.

FIG. 14 is a diagram showing an operation example of the slight motion vector grouping unit 606 included in the stationary person processing unit 614. As shown in FIG.

Humans are generally seldom completely still, except during sleep, and move slightly. This tendency is particularly strong in people who have passed through. A slight motion vector grouping unit 606 detects a person in a stationary state by observing and integrating these slight motions in time series.

In FIG. 14, (a) shows a person 900 at rest. The fine motion vector grouping unit 606 performs the same processing as the processing for pedestrians shown in FIG. 9 also for a stationary person. If a predetermined condition is met, even a stationary person may be able to obtain a person region that includes the entire person. However, in the case of the frame of interest alone, in most cases only a state in which a part of the whole body moves is observed. Therefore, the fine motion vector grouping section 606 continuously observes vector groups in a predetermined frame section (Tn to Tn+4), as shown in FIG. 14(b). For example, the micromotion vector grouping unit 606 observes five vector groups 906-910 for the same person.

In this case, the fine motion vector grouping unit 606 sets a rectangular provisional region that includes the five observed vector groups, as shown in FIG. 14(c). Then, the slight movement vector grouping unit 606 sets the set temporary area as the candidate area 912 of the person 900, as shown in FIG. 14(d).

Next, as shown in (e) in FIG. 14, the fine movement edge extraction unit 607 continuously performs edge extraction processing in a predetermined frame interval Tm to Tm+2 to obtain edge images 916 to 918 for each predetermined frame.

Next, the fine movement edge extraction unit 607 obtains one average edge image 919 from the edge images 916 to 918 in the predetermined frame section Tm to Tm+2, as shown in FIG. 14(f).

Next, as shown in (g) in FIG. 14, the time-series edge strength verification unit 608 determines whether or not there is an edge component strength equal to or greater than a predetermined standard in the candidate region 912 of the average edge image 919. . For example, the time-series edge strength verification unit 608 determines whether pixels with a predetermined brightness value exist in a predetermined area or more. Then, when there is an edge component equal to or greater than a predetermined criterion, time-series edge strength verification section 608 determines candidate region 912 shown in FIG. 14(g) to be person region 920 as shown in FIG. do. Time-series edge strength verification section 608 discards candidate region 912 indicated by (g) in FIG. 14 when the edge component is less than a predetermined criterion.

15A and 15B are diagrams for explaining an operation example of the adopted image determination unit 104. FIG.

Based on the learning error rate, the adopted image determination unit 104 determines whether an improvement in the accuracy of the person identification model can be expected. Then, the adopted image determination unit 104 selects, as a teacher image, the person image in the person region determined to be expected to improve the accuracy of the person identification model.

Adoption image determination unit 104 includes image reading unit 1001 , person identification test model learning unit 1002 , adoption determination unit 1003 , and image storage unit 1007 .

The image reading unit 1001 reads out a predetermined number of human images from the extraction image set 202 . The image reading unit 1001 may read any number of human images from the extracted image set 202 . For example, the image reading unit 1001 may read 2000 person images out of 10000 person images included in the extraction image set 202 .

The person identification test model learning unit 1002 performs deep learning of the person identification test model. Of the 2000 human images read by the image reading unit 1001, the person identification test model learning unit 1002 uses certain 1000 images for the filter coefficient update learning for each epoch of the person identification model, and uses the remaining 1000 images. It may be used for test error rate evaluation per epoch. Note that an epoch is a set of minimum computational units of filter coefficients of a discrimination model, and is a unit for referring to all input images for learning. An epoch is also called a learning training number.

When learning has progressed for a predetermined number of epochs, adoption determining section 1003 compares the learned filter coefficients and reference model error rate 1004 at the epoch with the lowest test error rate. A reference model error rate 1004 is a learning error rate of an existing person identification model.

When the adoption determination unit 1003 determines that the test model error rate is lower than the reference model error rate 1004 (adoption determination 1005: YES), the image storage unit 1007 stores the person image read by the image reading unit 1001 as a teacher image. are stored in the teacher image set 203 as. The teacher image set 203 is used for training a formal person identification model.

On the other hand, when the adoption determination unit 1003 determines that the test model error rate is higher than the reference model error rate 1004 (adoption determination 1005: NO), the person image read by the image reading unit 1001 is discarded.

FIG. 16 is a diagram showing an example of a learning error rate curve referred to by the adopted image determination unit 104. As shown in FIG.

In FIG. 16, the vertical axis indicates the test error rate (unit: %), and the horizontal axis indicates the number of times of learning and training (unit: epoch).

The test error rate is the ratio (error rate) of failures in identifying an unknown image when an unknown image for evaluating the test error rate is input to the learning model output for each epoch. That is. An unknown image is an image different from the image used for learning. The test error rate is expressed in a range from 0% to 100%, and generally, the closer to 0%, the higher the recognition performance of the discriminative model.

In FIG. 16, ER1 indicates the best value of the reference model test error rate 1004. FIG. ER2 indicates the best test error rate in the person identification test model learned by the person identification test model learning unit 1002 .

In the example of FIG. 16, ER2 is smaller than ER1. This indicates that the identification performance of the person identification test model is improved. Therefore, the person images used for learning the person identification test model are stored in the teacher image set.

On the other hand, if ER2 is greater than ER1, it indicates that the identification performance of the person identification test model is degraded. Therefore, the person images used for learning the person identification test model are discarded without being stored in the teacher image set.

FIG. 17 shows a first example of a teacher image extraction system including the teacher image extraction device 10. As shown in FIG. A first example shows an example in which a teacher image extraction system is configured locally.

For example, as shown in FIG. 17, the teacher image extraction system has a camera 1200, a video storage device 1201, a teacher image extraction device 10, a monitor 1203, and a teacher image storage device 1204.

Camera 1200 captures a video including a person.

The image storage device 1201 stores images (moving images) captured by the camera 1200 . Note that the video captured by the camera 1200 may be input directly to the teacher image extraction device 10 without being stored in the video storage device 1201 .

The teacher image extracting device 10 extracts the teacher image from the video input from the video storage device 1201 as described above. The video input to the teacher image extraction device 10 may be arbitrarily selected. The teacher image extracting apparatus 10 described above may have a memory and a CPU (Central Processing Unit), and the CPU may execute a software program stored in the memory to realize processing of each section. In this case, the teacher image extraction device 10 may be a personal computer (PC) that executes the software program.

The teacher image extraction device 10 may display the extracted teacher image on the monitor 1203 . Also, the teacher image extraction device 10 may store the extracted teacher image in the teacher image storage device 1204 .

FIG. 18 shows a second example of a teacher image extraction system including the teacher image extraction device 10. As shown in FIG.

A second example shows an example in which the teacher image extraction system is provided as a network cloud. The teacher image extraction system includes control PC 1208 , teacher image extraction device 10 , video storage device 1209 and teacher image storage device 1212 .

The cloud provides various IT resources such as computing, databases, storage, and/or applications on demand over the Internet 1207 .

For example, as shown in FIG. 18, a camera 1206, a host PC 1213 and a monitor 1214 are provided locally, and a teacher image extraction system is provided as a cloud via a network 1207. FIG.

A camera 1206 captures an image including a person.

The host PC 1213 stores the video captured by the camera 1206 in the video storage device 1209 via the network 1207 and control PC 1208 .

The teacher image extracting device 10 extracts the teacher image from the video input from the video storage device 1201 as described above.

As shown in FIG. 18, a plurality of teacher image extraction devices 10 may be provided. In this case, the plurality of teacher image extraction devices 10 may extract teacher images by parallel processing. Further, the functions and processing of the teacher image extracting device 10 described above may be realized by executing the computer program 1211 stored in the memory of each device 10 by the CPU. In this case, the teacher image device 10 may be a sub-PC that executes the computer program.

A teacher image extracted by the teacher image extraction device 10 may be displayed on a local monitor 1214 via the network 1207 and host PC 1213 . Also, the teacher image extraction device 10 may store the extracted teacher image in the teacher image storage device 1204 .

In the above description, an example in which the extraction target is a person has been described, but the extraction target is not limited to a person. For example, an extraction target may be a building, a vehicle, a home appliance, the sea, a mountain, the sky, a flower, or a tree other than a person.

If the object to be extracted is not a person, the constraint conditions for judging the presence of a person in the intermediate frame image extraction unit including the key frame person designation unit and the area correction unit described above are set appropriately according to the extraction object. may be changed to

The above contents can be expressed as follows.

A teacher image extracting device 10 for extracting an image of a person, which is an example of a target object, from an image frame included in a video includes a key frame person specifying unit 101 and an intermediate frame image extracting unit 102 . The keyframe person specifying unit 101 acquires an image of a person area specified by a rectangle including a portion where a target object is displayed in the keyframe, using a frame of a part of the video as a keyframe. An intermediate frame image extraction unit 102 tracks a person specified by a key frame in the video, and extracts a rectangular human area image including a portion where the person is displayed in the intermediate frame, which is a non-key frame in the video.

According to this configuration, since a rectangular partial image in which a person is displayed is extracted from the frame included in the video, it is possible to extract a teacher image with reduced background noise.

The teacher image extracting apparatus 10 performs human region identification that identifies a human region, which is a portion in which a human is displayed in an intermediate frame, by processing different motion vectors depending on whether the human is moving or stationary. A unit 103 may also be included.

The motion vector of the person's area exhibits different characteristics depending on whether the person is moving or stationary. Therefore, according to this configuration, the human region can be identified by using suitable motion vector processing in each case. can.

When the person is moving, the person area identification unit 103 may identify the person area by synthesizing areas obtained by grouping motion vectors that are close to each other and have the same movement direction and movement amount. .

When the person is moving, the parts of the person move in the same manner. Therefore, according to this configuration, by grouping close equivalent motion vectors and synthesizing them, it is possible to identify the human region well.

When the person is stationary, the person region identifying unit 103 obtains motion vectors by grouping motion vectors indicating similar moving directions and moving amounts in each of a plurality of consecutive frames including the target intermediate frame. A person area may be specified based on an area obtained by synthesizing partial areas over a plurality of frames.

Even if a person is stationary, it is common to see movement in each part. Therefore, according to this configuration, it is possible to satisfactorily identify a person area by synthesizing partial areas obtained by grouping motion vectors over a plurality of frames.

When the person is stationary, the person region identification unit 103 identifies the person based on an area obtained by synthesizing partial regions over a plurality of frames and an edge image extracted from one or more frame images including an intermediate frame. A region may be specified.

According to this configuration, by using the result of edge extraction in addition to the region obtained by synthesizing the partial regions in which movement is observed, it is possible to specify the region of the still person more satisfactorily.

The intermediate frame image extracting unit 102 may correct the person area of the intermediate frame based on motion vectors of one or more frames before or after the intermediate frame.

According to this configuration, since the person area is corrected by the processing using the frames before and after the target intermediate frame, it is possible to reduce the cut-out of the person.

The intermediate frame image extracting unit 102 estimates a region in which the amount of movement of the motion vector in one or more frames before or after the intermediate frame is relatively large as the position of the tip of the person's walking foot. The person region may be corrected to be included.

According to this configuration, the person area is corrected so as to include the area estimated to be the tip of the foot by processing using the motion vectors of the frames before and after the target intermediate frame, so that the tip of the person's foot can be reduced. .

The teacher image extraction device 10 constructs a test model using a human image group including a human image in a key frame and a human image in an intermediate frame for deep learning, evaluates the accuracy of human identification by the test model, and uses it as an evaluation result. It may further include a adopted image determination unit 104 that determines whether or not to adopt the person image group based on the above.

According to this configuration, a test model is constructed from the extracted human image group, the accuracy of human identification is evaluated, and the acceptance or rejection of the human image group is determined. can.

When a plurality of persons are specified in a key frame, the intermediate frame image extracting unit 102 may track each of the plurality of persons and extract a person image of the person area for each person.

According to this configuration, each of a plurality of persons is tracked and a person image in the person area is extracted for each person, so it is possible to extract a large number of person images.

Reference Signs List 10 Teacher image extracting device 100 Input video input 101 Key frame person specifying unit 102 Intermediate frame image extracting unit 103 Human area identifying unit 104 Adopted image determining unit 105 Teacher image saving unit

Claims (7)

An object image extracting device for extracting an image of a person, which is an object, from an image frame included in a video,
a key frame person designating unit that obtains an image of a person area designated by a rectangle including a portion in which a person is displayed in the key frame, using a frame of a part of the video as a key frame;
Intermediate frame image extraction for tracking the person specified by the keyframe in the video, and extracting an image of a rectangular human area including a portion where the person is displayed in an intermediate frame, which is a frame that is not the keyframe in the video. Department and
a human region identifying unit that identifies the human region in the intermediate frame by processing different motion vectors depending on whether the person is moving or stationary;
has
The intermediate frame image extracting unit detects a motion vector indicating a shift direction and a shift amount between frames of each block obtained by dividing a frame into predetermined units, stabilizes the motion vector,
The person region identification unit determines that the person is moving when the prediction error of the motion vector after stabilization is equal to or greater than a predetermined threshold, and determines that the person is moving, and the person is moving in the same direction and the same amount of movement. A human region is identified by synthesizing regions obtained by grouping vectors, and if the prediction error is less than the threshold, the human is determined to be stationary, and the target intermediate frame is included. Identifying the person region based on a region obtained by synthesizing partial regions obtained by grouping motion vectors indicating similar moving directions and amounts of movement in each of a plurality of consecutive frames over the plurality of frames;
Object image extraction device. When the person is stationary, the person area specifying unit divides the area obtained by synthesizing the partial areas over the plurality of frames into an edge image extracted from one or more frame images including the intermediate frame. identifying the person region based on
2. The object image extraction device according to claim 1 . The intermediate frame image extracting unit, when the person region identifying unit determines that the person is moving, determines that the movement amount of the motion vector of one or more frames before or after the intermediate frame is relative. estimating an area with a large area as the tip position of the foot where the person is walking, and correcting the person area so that the tip position of the foot is included;
2. The object image extraction device according to claim 1. A test model is constructed using a human image group including the image of the human region of the key frame and the image of the human region of the intermediate frame for deep learning, the accuracy of human identification by the test model is evaluated, and the evaluation result is used. further comprising a adopted image determination unit that determines whether or not to adopt the person image group based on
2. The object image extraction device according to claim 1. When a plurality of persons are specified in the key frame, the intermediate frame image extracting unit tracks each of the plurality of persons and extracts an image of the person region for each person .
2. The object image extraction device according to claim 1. A target object image extraction method for extracting an image of a person, which is a target object, from an image frame included in a video,
the computer
obtaining an image of a human region specified by a rectangle including a portion in which a person is displayed in the key frame, using a frame of a portion of the video as a key frame;
A method of tracking a person specified by the keyframe in the video, and extracting an image of a rectangular person region including a portion in which the person is displayed in an intermediate frame, which is a frame that is not the keyframe in the video,
Detecting a motion vector indicating the direction and amount of shift between frames of each block obtained by dividing a frame into predetermined units, and stabilizing the motion vector;
If the prediction error of the motion vector after stabilization is equal to or greater than a predetermined threshold, it is determined that the person is moving, and the motion vectors are obtained by grouping motion vectors that are close to each other and have the same movement direction and movement amount. Identify the person area by synthesizing the areas,
When the prediction error is less than the threshold value, the person is determined to be stationary, and movement indicating similar and close moving directions and moving amounts in each of a plurality of consecutive frames including the target intermediate frame. identifying the person region based on a region obtained by synthesizing the partial regions obtained by grouping the vectors over the plurality of frames;
Object image extraction method. A software program for causing a computer to extract an image of a person who is an object from an image frame included in a video,
obtaining an image of a human region specified by a rectangle including a portion in which a person is displayed in the key frame, using a frame of a portion of the video as a key frame;
Tracking the person specified by the keyframe in the video, and extracting an image of a rectangular person region including a portion where the person is displayed in an intermediate frame, which is a frame that is not the keyframe in the video,
Detecting a motion vector indicating the direction and amount of shift between frames of each block obtained by dividing a frame into predetermined units, and stabilizing the motion vector;
If the prediction error of the motion vector after stabilization is equal to or greater than a predetermined threshold, it is determined that the person is moving, and the motion vectors are obtained by grouping motion vectors that are close to each other and have the same movement direction and movement amount. Identify the person area by synthesizing the areas,
When the prediction error is less than the threshold value, the person is determined to be stationary, and movement indicating similar and close moving directions and moving amounts in each of a plurality of consecutive frames including the target intermediate frame. identifying the person region based on a region obtained by synthesizing the partial regions obtained by grouping the vectors over the plurality of frames;
A software program that makes a computer do things.

Images