JP6331270B2 - Information processing system, information processing method, and program - Google Patents

Description

  Some embodiments according to the present invention relate to an information processing system, an information processing method, and a program.

In recent years, for example, various techniques for analyzing video captured by a surveillance camera or the like have been proposed. For example, Patent Document 1 discloses a method of generating a flow line by connecting the reference points in time order after calculating the center point (reference point) of the circumscribed rectangle of the person in the foreground area detected by the background difference. Yes. In this method, a person's state such as a crouching state or a hand-running state is determined from a temporal change of the circumscribed rectangle of the person, and a reference point is calculated by referring to a value of a past circumscribed rectangle according to the state. ing.
Other related techniques are disclosed in Patent Document 2 and Patent Document 3.

JP 2011-243155 A JP 2012-208850 A JP2012-043021A

  However, in the method described in Patent Document 1 described above, since the person rectangle obtained from the background difference is easily affected by changes in environmental factors such as noise and illumination fluctuations, it is difficult to stably determine the state of the person. It is. Further, in the method described in Patent Document 1, it is assumed that the reference point is accurately taken in the state immediately before the state change occurs when calculating the person position (reference point), and immediately before the state change. The reference point at the time of state change is obtained using the coordinate value of. For this reason, for example, when the person rectangle is not correctly obtained before the state change, for example, when the person transitions from the overlapping state to the squatting state, there is a problem that the accuracy is greatly reduced.

  Some aspects of the present invention have been made in view of the above-described problems, and an object thereof is to provide an information processing system, an information processing method, and a program capable of suitably estimating the position of a person. .

  An information processing system according to the present invention includes means for detecting a person area from an image frame constituting a video, and determination means for determining a posture when the person area is included in a posture variation area in the image frame. Calculating means for calculating a foot position of the person using the image frame based on whether or not the posture of the person included in the person area is in a standing state.

  The information processing method according to the present invention includes a step of detecting a person area from an image frame constituting a video, a step of determining a posture when the person area is included in a posture variation area in the image frame, Calculating a foot position of the person using the image frame based on whether or not the posture of the person included in the person region is in an upright state.

  The program according to the present invention includes a process for detecting a person area from an image frame constituting a video, a process for determining a posture when the person area is included in a posture variation area in the image frame, and the person Based on whether or not the posture of the person included in the region is in the standing state, the computer is caused to execute a process of calculating the foot position of the person using the image frame.

  In the present invention, “part”, “means”, “apparatus”, and “system” do not simply mean physical means, but “part”, “means”, “apparatus”, “system”. This includes the case where the functions possessed by "are realized by software. Further, even if the functions of one “unit”, “means”, “apparatus”, and “system” are realized by two or more physical means or devices, two or more “parts” or “means”, The functions of “device” and “system” may be realized by a single physical means or device.

  According to the present invention, it is possible to provide an information processing system, an information processing method, and a program that can preferably estimate the position of a person.

It is a figure for demonstrating the outline | summary of the position estimation method. It is a figure which shows the function structure of the information processing system which concerns on 1st Embodiment. It is a figure for demonstrating the specific example of the determination method of a squatting period. It is a flowchart which shows the flow of a process of the information processing system shown in FIG. It is a block diagram which shows the structure of the hardware which can mount the information processing system shown in FIG. It is a functional block diagram which shows schematic structure of the information processing system which concerns on 2nd Embodiment.

  Embodiments of the present invention will be described below. In the following description and the description of the drawings to be referred to, the same or similar components are denoted by the same or similar reference numerals.

(1 First Embodiment)
1 to 4 are diagrams for explaining the first embodiment. Hereinafter, the present embodiment will be described along the following flow with reference to these drawings. First, in “1.1”, an overview of the present embodiment will be described. Next, an overview of the functional configuration of the system is described in “1.2”, and a processing flow is described in “1.3”. “1.4” shows a specific example of a hardware configuration capable of realizing the information processing system according to the present embodiment. Finally, the effects and the like according to the present embodiment will be described after “1.5”.

(1.1 Overview)
The present embodiment relates to an information processing system that specifies the position (coordinates) of a person's feet in real space from, for example, video captured by a video camera such as a surveillance camera. In particular, the information processing system according to the present embodiment can suitably detect the position of a person's feet even when the person squats down or crouches.

  For example, as a method of extracting a person from a video of a surveillance camera that captures the inside of a store or the like, for example, after detecting the head of the person, the position of the foot in the real space is determined from the result and the height information of the person. An estimation method can be considered. However, with this method, there is a problem that the tracking position tends to shift when the posture changes due to the person squatting or bending.

  This point will be described with reference to FIG. In FIG. 1, a video camera C is used to photograph a person P. Here, the person P is crouching at the position L. If a method of simply estimating the foot based on the position of the head in the video of the video camera C without considering the posture is used, the position L ′ of the person P ′ in the standing posture is changed to the position of the person P. As you might guess. That is, when the person P squats down, an estimated deviation corresponding to the distance from the actual position L to the estimated position L ′ occurs. Thus, if the estimated position of the person is shifted, for example, when it is desired to estimate the time spent on each shelf in the flow analysis of the customer in the store, the correct time and place cannot be obtained.

  Therefore, in the information processing system according to the present embodiment, after determining the posture state of the person, the squatting (the same applies to the bending. Hereinafter, the squatting and the bending are collectively referred to as “squatting”) ( If it is not a standing posture, the foot position is estimated without using the height information of the person. In addition, in order to suppress the influence when a false detection occurs in the determination of the posture state of a person, when a person squatting is determined in advance after specifying an area in which the person is likely to squat in advance, Correct the estimated foot position.

(1.2 System overview)
Hereinafter, the system configuration of the information processing system 100 according to the present embodiment will be described with reference to FIG. FIG. 2 is a block diagram illustrating a system configuration of the information processing system 100. The information processing system 100 includes an input unit 101, a person detection unit 103, a posture variation determination unit 105, posture variation application area information 107, a foot position calculation unit 109, and an output unit 111.

  The input unit 101 receives video input from, for example, a monitoring camera (not shown) or a storage medium that records video captured by the monitoring camera. The input unit 101 outputs the input video to the person detection unit 103, the posture variation determination unit 105, and the foot position calculation unit 109 for each image frame.

  The person detection unit 103 outputs person detection information by detecting a person from the input video. Here, the person detection unit 103 detects a person by, for example, head detection, face detection, and upper body detection on the image. Here, the head, face, upper body, and the like can be detected based on the color, shape, and the like in the image, respectively. The person detection information output by the person detection unit 103 includes, for example, position information of the detected person in the image (for example, rectangular information including a person region).

  The posture variation determination unit 105 detects posture variation for each person area included in the person detection information output from the input unit 101 (detects whether or not it is a bent posture (squatting posture)). At this time, the posture variation determination unit 105 does not determine the posture based only on the person detection information, but also detects the posture variation using the video input from the input unit 101. As a posture detection method by the posture variation determination unit 105, for example, a discriminator in which postures of various people are learned in advance is constructed, and this is applied to the vicinity of a human region in an image frame to be processed in a video. Thus, it can be determined whether or not the person shown in the person area is in a squatting position (whether it is a standing posture). The construction of this classifier is, for example, a method in which an image of a person in a posture to be detected and other images are collected in advance, and the image itself is learned by a neural network, a luminance gradient feature from the collected image, etc. After the feature amount is extracted, the feature amount can be realized by a classifier such as SVM (Support Vector Machine) or GLVQ (Generalized Learning Vector Quantization). The posture variation determination unit 105 outputs the posture determination result to the foot position calculation unit 109 as posture variation detection information.

  Note that if a detection error occurs during posture detection, for example, when a part of the person area is shielded by an obstacle, it may greatly affect the calculation of the person's foot position. In view of this, in the information processing system 100 according to the present embodiment, posture variation application region information 107 in which an image region to which posture variation detection is applied is set in advance. The posture variation determination unit 105 determines the posture of the person with respect to the posture variation application region specified by the posture variation application region information 107 in the image frame related to the input video.

  In this embodiment, the posture variation determination unit 105 performs crouching detection for each image frame. However, even when the person is in a squatting posture, the squatting detection is not always performed correctly in all frames, and there is a possibility that the squatting is not detected. Therefore, as shown in FIG. 3, when the posture variation determination unit 105 detects squatting in a certain image frame (in the case of FIG. 3, three times of squatting postures at times t1, t2, and t3 are detected), then It can also be determined that the squatting posture continues for a predetermined time T. In that case, the posture variation determination unit 105 can also determine that the squatting continues for a certain period (time T) from the detection of the last squatting posture (time t3 in FIG. 3).

  The foot position calculation unit 109 uses the image frame of the video input from the input unit 101, the person detection information output from the person detection unit 103, and the posture variation detection information output from the posture variation determination unit 105. Estimate the position of the person's feet in real space. The foot position calculation unit 109 outputs the estimated foot position of the person in real space to the output unit 111.

  More specifically, when the person is standing (when the squatting posture is not detected), the foot position calculation unit 109 is included in the person detection information, for example, as a result of head detection. The foot position of the person is estimated using the obtained position information of the top of the head and the height information given to the person obtained by dynamic analysis of the person image of the detected person. At this time, it is necessary to calculate the apparent length on the image from the height, but this can be calculated using camera parameters obtained by camera calibration.

  On the other hand, when it is detected that the person is in a squatting state, the foot position calculation unit 109 estimates the foot position of the person through image analysis. At this time, the foot position calculation unit 109 may take a difference between a background image (an image in which no person is shown) and an image in which a person to be processed is shown, and use the lower end related to the difference as the foot position. However, there are many cases where the foot position cannot be obtained correctly due to the influence of the shadow reflected on the foot of the person. Thus, it is also conceivable to detect the foot position in the image using a foot detector generated by learning images of the foot of a person in various squatting postures. At this time, the foot detector detects the position of the foot by, for example, estimating the image region where the foot exists from the position of the top of the head and applying the foot detector to the foot region. At this time, when the person is crouching, the position of the foot estimated from the height often deviates greatly, so the foot position calculation unit 109 searches for a wider range as the foot area than when standing. You may do it. The foot detector can also be constructed by causing a discriminator such as a neural network or SVM to learn the image of the foot region and the feature amount of the image, similarly to the detector that detects the squatting posture.

  The output unit 111 outputs the calculation result of the foot position in the real space in each image frame of each person shown in the video by the foot position calculation unit 109. As an output destination of the person position calculation result by the output unit 111, for example, a display device, a storage medium such as an HDD (Hard Disk Drive), or the like can be considered.

(1.3 Process flow)
Next, a processing flow of the information processing system 100 will be described with reference to FIG. FIG. 4 is a flowchart showing a processing flow of the information processing system 100 according to the present embodiment.

  Each processing step to be described later can be executed in any order or in parallel as long as there is no contradiction in processing contents, and other steps can be added between the processing steps. good. Further, a step described as a single step for convenience can be executed by being divided into a plurality of steps, and a step described as being divided into a plurality of steps for convenience can be executed as one step.

  The input unit 101 receives video input from, for example, a monitoring camera or a storage medium that stores video of the monitoring camera (S401). The input unit 101 sequentially outputs the image frames constituting the video to the person detection unit 103, the posture variation determination unit 105, and the foot position calculation unit 109.

  The person detection unit 103 detects a person from the image frame received from the input unit 101 (S403). As described above, head detection, upper body detection, face detection, and the like are conceivable as a person detection method.

  The posture variation determination unit 105 determines whether or not the detected person is in the posture variation application area in the image frame (S405). As a result, when the person is in the posture variation application area (Yes in S405), the posture variation determination unit 105 detects the posture of the person (S407). As a result, if it is determined that the target person is standing (not squatting) (Yes in S409), the foot position calculation unit 109 calculates the head position and the height information of the person. The foot position in the real world of the person is calculated based on the original (S411). The same applies to the case where it is determined in S405 that the position of the person is not the posture variation application area (No in S405).

  On the other hand, if it is determined that the person is crouching (not standing) (No in S409), the foot position calculation unit 109 estimates the foot position from the image frame, for example, by the method described above. This is performed (S413).

(1.4 Hardware configuration)
Hereinafter, an example of a hardware configuration when the information processing system 100 described above is realized by a computer will be described with reference to FIG. Note that the functions of the information processing system 100 can also be realized by a plurality of information processing apparatuses.

  As illustrated in FIG. 5, the information processing system 100 includes a processor 501, a memory 503, a storage device 505, an input interface (I / F) 507, a data I / F 509, a communication I / F 511, and a display device 513.

  The processor 501 controls various processes in the information processing system 100 by executing a program stored in the memory 503. For example, the processing related to the input unit 101, the person detection unit 103, the posture variation determination unit 105, the foot position calculation unit 109, and the output unit 111 described in FIG. 2 is temporarily stored in the memory 503 and then mainly the processor 501. It can be realized as a program that operates above.

  The memory 503 is a storage medium such as a RAM (Random Access Memory). The memory 503 temporarily stores a program code of a program executed by the processor 501 and data necessary for executing the program. For example, a stack area necessary for program execution is secured in the storage area of the memory 503.

  The storage device 505 is a non-volatile storage medium such as a hard disk or a flash memory. The storage device 505 stores an operating system, various programs for realizing the input unit 101, the person detection unit 103, the posture variation determination unit 105, the foot position calculation unit 109, and the output unit 111, various data, and the like. Programs and data stored in the storage device 505 are referred to by the processor 501 by being loaded into the memory 503 as necessary.

  The input I / F 507 is a device for receiving input from the user. The posture variation application area information 107 described with reference to FIG. 2 may be input from the user by the input I / F 507. Specific examples of the input I / F 507 include a keyboard, a mouse, and a touch panel. The input I / F 507 may be connected to the information processing system 100 via an interface such as USB (Universal Serial Bus).

  The data I / F 509 is a device for inputting data from outside the information processing system 100. Specific examples of the data I / F 509 include a drive device for reading data stored in various storage media. The data I / F 509 may be provided outside the information processing system 100. In that case, the data I / F 509 is connected to the information processing system 100 via an interface such as a USB.

  The communication I / F 511 is a device for performing data communication with a device external to the information processing system 100, such as a monitoring camera, by wire or wireless. It is conceivable that the communication I / F 511 is provided outside the information processing system 100. In that case, the communication I / F 511 is connected to the information processing system 100 via an interface such as a USB.

  The display device 513 is a device for displaying various information. Specific examples of the display device 513 include a liquid crystal display and an organic EL (Electro-Luminescence) display. The display device 513 may be provided outside the information processing system 100. In that case, the display device 513 is connected to the information processing system 100 via, for example, a display cable.

(1.5 Effects according to this embodiment)
As described above, in the information processing system 100 according to the present embodiment, after detecting the posture of the person shown in the video, based on whether or not the person is in the squatting position (whether the person is standing or not), The position calculation method is changed. Thereby, the position of a person can be estimated suitably.
Further, in the information processing system 100 according to the present embodiment, the posture of the person is determined using not only the detection result of the person but also the video, and the foot position is also separately detected from the image. This makes it possible to estimate the position with high accuracy even when there is an external environmental change.

(2 Second Embodiment)
The second embodiment will be described below with reference to FIG. FIG. 6 is a block diagram illustrating a functional configuration of the information processing system 600. As illustrated in FIG. 6, the information processing system 600 includes a detection unit 610, a determination unit 620, and a calculation unit 630.
The detection unit 610 detects a person area from the image frames constituting the video.
The determination unit 620 determines the posture of a person included in the person area.
The calculation unit 630 calculates the foot position of the person using the image frame based on whether or not the posture of the person is standing.
By mounting in this way, the information processing system 600 according to the present embodiment can preferably estimate the position of the person.

(3 Additional notes)
Note that the configurations of the above-described embodiments may be combined or some of the components may be replaced. The configuration of the present invention is not limited to the above-described embodiment, and various modifications may be made without departing from the scope of the present invention.

  A part or all of each of the above-described embodiments can be described as in the following supplementary notes, but is not limited thereto. Moreover, the program of this invention should just be a program which makes a computer perform each operation | movement demonstrated in said each embodiment.

(Appendix 1)
Using the image frame based on means for detecting a person area from image frames constituting the video, determination means for determining the posture of the person included in the person area, and whether or not the posture of the person is standing An information processing system comprising: a calculating means for calculating a foot position of the person.

(Appendix 2)
The information processing system according to supplementary note 1, wherein the determination unit determines a posture of a person when the person region is included in a posture variation region in the image frame.

(Appendix 3)
The calculation means calculates the foot position of the person based on whether the person area is in a predetermined area and the posture of the person included in the person area is in an upright state. The information processing system described.

(Appendix 4)
Any one of appendix 1 to appendix 3, wherein when the person included in the person area is determined not to be in a standing state, the calculating means calculates a foot position on the assumption that the person is not in a standing state for a certain period of time. 1. An information processing system according to item 1.

(Appendix 5)
The information processing system according to any one of appendix 1 to appendix 4, wherein when the person included in the person area is determined to be standing, the calculation means calculates a foot position based on a height. .

(Appendix 6)
Any one of appendix 1 to appendix 5, wherein the calculation means calculates a foot position when a person included in the person region is not in a standing state, using a learning result related to a foot position of a person who is not in a standing state. Information processing system according to item.

(Appendix 7)
Using the image frame based on detecting a person region from image frames constituting the video, determining a posture of a person included in the person region, and whether or not the posture of the person is standing An information processing method in which an information processing system performs the step of calculating the foot position of the person.

(Appendix 8)
The information processing method according to appendix 7, wherein the posture of the person is determined when the person region is included in the posture variation region in the image frame.

(Appendix 9)
The information processing method according to appendix 7 or appendix 8, wherein the person area is in a predetermined area, and the foot position of the person is calculated based on whether or not the posture of the person included in the person area is in an upright state. .

(Appendix 10)
The information processing method according to appendix 7 to appendix 9, wherein when it is determined that the person included in the person area is not standing, the foot position is calculated by assuming that the person is not standing for a certain period.

(Appendix 11)
The information processing method according to any one of appendix 7 to appendix 10, wherein when it is determined that a person included in the person area is standing, a foot position is calculated based on a height.

(Appendix 12)
The information processing according to any one of appendix 7 to appendix 11, wherein a foot position when a person included in the person region is not in a standing state is calculated using a learning result related to the foot position of a person who is not in a standing state. Method.

(Appendix 13)
Using the image frame based on a process for detecting a person area from image frames constituting a video, a process for determining the posture of a person included in the person area, and whether or not the posture of the person is standing The program which makes a computer perform the process which calculates the said person's foot position.

(Appendix 14)
The program according to appendix 13, wherein the posture of the person is determined when the person region is included in the posture variation region in the image frame.

(Appendix 15)
The program according to supplementary note 13 or supplementary note 14, wherein the person region is in a predetermined region, and the foot position of the person is calculated based on whether or not the posture of the person included in the person region is in an upright state.

(Appendix 16)
The program according to appendix 13 to appendix 15, wherein when it is determined that the person included in the person area is not in the standing state, the foot position is calculated by assuming that the person is not in the standing state for a certain period of time.

(Appendix 17)
The program according to any one of appendix 13 to appendix 16, wherein when it is determined that the person included in the person area is standing, the foot position is calculated based on the height.

(Appendix 18)
18. The program according to any one of appendix 13 to appendix 17, wherein a foot position when a person included in the person area is not in a standing state is calculated using a learning result related to the foot position of a person who is not in a standing state.

DESCRIPTION OF SYMBOLS 100 ... Information processing system, 101 ... Input part, 103 ... Person detection part, 105 ... Attitude fluctuation determination part, 107 ... Attitude fluctuation application area information, 109 ... Foot position calculation part 111, output unit, 501 ... processor, 503 ... memory, 505 ... storage device, 507 ... input interface, 509 ... data interface, 511 ... communication interface, 513. ··· Display device, 600 ··· Information processing system, ······································································· ··person

Claims (7)

Means for detecting a human region from image frames constituting a video;
Determining means for determining the posture of the person included in the person area;
Calculating means for calculating a foot position of the person using the image frame based on whether or not the posture of the person is standing ;
The calculation means calculates a foot position when a person included in the person region is not in a standing state, using a learning result relating to the foot position of a person who is not in a standing state.
Information processing system. The determination means determines the posture of the person when the person region is included in the posture variation region in the image frame;
The information processing system according to claim 1. The calculating means calculates the foot position of the person based on whether the person area is in a predetermined area and the posture of the person included in the person area is in an upright state;
The information processing system according to claim 1 or 2. The calculation means calculates a foot position, assuming that the person is not in a standing state for a certain period when it is determined that the person included in the person area is not in a standing state.
The information processing system according to any one of claims 1 to 3. The calculating means calculates a foot position based on a height when it is determined that the person included in the person area is in a standing state;
The information processing system according to any one of claims 1 to 4. Detecting a human region from image frames constituting the video;
Determining the posture of a person included in the person area;
Based posture of the person on whether standing state, have rows information processing system and calculating a foot position of the person using the image frame,
Using the learning result relating to the foot position of the person who is not in the standing state, the foot position when the person included in the person region is not in the standing state is calculated.
Information processing method. A process for detecting a human area from image frames constituting a video;
A process of determining the posture of a person included in the person area;
Based on whether or not the posture of the person is in a standing state, the computer executes a calculation process for calculating the position of the person's foot using the image frame ,
In the calculation process, by using a learning result related to a foot position of a person who is not in a standing state, a foot position when a person included in the person region is not in a standing state is calculated.
program.

Images