WO2022025127A1 - Video analysis device and video analysis method - Google Patents

Abstract

In order to enable calculation of long-term staying time, the present invention is characterized by comprising: a video data processing unit (11) that acquires a video from a video recording device (3); a frame image analysis unit (13) that detects a person image in each of the images constituting the acquired video; a movement line analysis unit (14) that extracts a line of movement from the detected person images, detects movement line information which is information about the extracted line of movement, acquires multiple pieces of movement line information detected, and determines whether the multiple pieces of movement line information acquired represent the line of movement of the same person on the basis of person features in the lines of movement; a total staying time analysis unit (21) that calculates, on the basis of the pieces of movement line information determined as being derived from the same person, the total staying time of that person; and an output server (5) that outputs the calculated total staying time to an output unit.

Description

Video analysis device and video analysis method

The present invention relates to a technique of a moving image analysis device and a moving image analysis method.

Currently, in order to search for a person who has been reflected on the same camera for a certain period of time or longer (hereinafter referred to as a stagnant person), a stagnant detection using a flow line detection is performed.

The techniques described in Patent Document 1 and Patent Document 2 are disclosed as those for detecting retention by detecting a flow line.
Patent Document 1 includes a processor 12 that "acquires captured images 31 for each of a plurality of cameras 1 and acquires tracking information by analyzing the captured images 31 to generate a surveillance image including an image relating to a prowling person. The processor 12 has, for each camera 1, a first person rectangular 32 indicating that the person detected from the captured image 31 is a prowling person candidate based on the first and second tracking tables, and a prowling person. A person behavior monitoring device and a person behavior monitoring system for generating a monitoring image in which a second person rectangle 32 indicating that the image is superimposed on the captured image 31 are disclosed (see summary).

Patent Document 2 states, "In an image processing device, an area for detecting a subject is set in an image to be processed, a plurality of images to be processed that are continuous in time are analyzed to detect the subject, and the subject is continuously detected in time. Based on the detection results of the subject in the plurality of images to be processed, the residence time during which the subject stays in the area is determined, and in the determination, the subject moves from the inside of the area to the outside of the area and returns to the inside of the area. In this case, the time when the subject is located outside the area is included in the residence time. ”The image processing device, the image processing system, the control method of the image processing device, and the program are disclosed (see summary).

Japanese Unexamined Patent Publication No. 2020-074506 Japanese Unexamined Patent Publication No. 2020-0916449

Retention detection using flow line detection as shown in Patent Document 1 and Patent Document 2 has the following two problems and is not suitable for long-term retention tracking.
(A) Generally, when the flow line is interrupted, it is determined that the staying person has disappeared. Therefore, if there is a tracking error due to the interruption of the flow line, the total residence time of the person cannot be calculated. That is, if the stagnant person is blocked by an object or the like and most of the person is not reflected, the detection of the flow line is interrupted. Similarly, when the person to be tracked for detecting the flow line is framed out, the flow line is interrupted. As a result, even if the same person is reflected again, another person will be retained, and the total residence time cannot be calculated accurately.

(B) If an occlusion (overlap with another person) occurs for the staying person, erroneous tracking will occur.

The present invention has been made in view of such a background, and it is an object of the present invention to make it possible to calculate the residence time over a long period of time.

In order to solve the above-mentioned problems, the present invention relates to a moving image acquisition unit that acquires a moving image from an imaging unit, a person detecting unit that detects a person image in each of the acquired images constituting the moving image, and the detected person image. , The flow line detection processing unit that extracts one flow line and detects the flow line information that is the information about the extracted flow line, and the flow line information that is once interrupted detected by the flow line detection processing unit. And a determination unit that determines whether or not each of the acquired plurality of the flow line information is the same person's flow line based on the person's characteristic amount in the flow line, and the same person. It is characterized by having a total residence time calculation unit that calculates the total residence time of the person based on the flow line information, and an output processing unit that outputs the calculated total residence time to the output unit. ..
Other solutions will be described as appropriate in the embodiments.

According to the present invention, it is an object to make it possible to calculate the residence time over a long period of time.

It is a figure which shows the structural example of the moving image analysis system which concerns on 1st Embodiment. It is a figure which shows the detailed structure of the frame image analysis part in 1st Embodiment. It is a figure which shows the detailed structure of the flow line analysis part in 1st Embodiment. It is a figure which shows the detailed structure of the total residence time analysis part in 1st Embodiment. It is a figure which shows the hardware block diagram of the arithmetic unit. It is a figure which shows the example of the person data. It is a figure which shows the example of the flow line data. It is a figure which shows the example of a frame data. It is a figure which shows the example of the flow line rediscovery data. It is a figure which shows the example of the total residence time data. It is a flowchart which shows the procedure of the flow line collection process performed in 1st Embodiment. It is a flowchart which shows the procedure of the total residence time extraction processing performed in 1st Embodiment. It is a flowchart which shows the detailed procedure of the moving image analysis processing performed in 1st Embodiment. It is a flowchart which shows the detailed procedure of the flow line detection process performed in 1st Embodiment. It is a flowchart which shows the detailed procedure of the flow line re-detection process which concerns on 1st Embodiment. It is a flowchart which shows the detailed procedure of the total residence time calculation process performed in 1st Embodiment. It is a flowchart which shows the detailed procedure of the total residence time detailed screen display processing performed in 1st Embodiment. It is a figure which shows the surveillance moving image display screen. It is a figure which shows the total residence time detailed screen. It is a figure which shows the personal stay information display screen. It is a figure explaining the flow line transfer. It is a flowchart which shows the procedure of the total residence time extraction processing performed in 2nd Embodiment. It is a figure which shows the result of having performed the total residence time extraction process performed in 2nd Embodiment. It is a figure which shows the detailed structure of the flow line analysis part in 3rd Embodiment. It is a figure which shows the example of the flow line data used in 3rd Embodiment. It is a flowchart which shows the procedure of the flow line detection process performed in 3rd Embodiment. It is a flowchart which shows the detailed procedure of the flow line coupling process performed in 3rd Embodiment. It is a figure which shows the specific example of the flow line coupling process.

Next, an embodiment for carrying out the present invention (referred to as "embodiment") will be described in detail with reference to the drawings as appropriate.

[First Embodiment]
First, the first embodiment of the present invention will be described with reference to FIGS. 1 to 20.
<System configuration diagram>
FIG. 1 is a diagram showing a configuration example of the moving image analysis system Z according to the first embodiment.
The moving image analysis system Z includes a moving image analysis server 1, a total residence time analysis server 2, a moving image shooting device (imaging unit) 3 such as a camera, a database 4, an output server 5, and an output device (output unit) 6.
The moving image analysis server 1 has a moving image data processing unit (moving image acquisition unit) 11, a temporary storage unit 12, a frame image analysis unit 13, and a flow line analysis unit 14.
The moving image data processing unit 11 extracts frame image data (hereinafter referred to as frame image data) constituting the moving image data acquired from the moving image shooting device 3, and stores the extracted frame image data in the temporary storage unit 12. The temporary storage unit 12 is a storage area formed in the memory 71 (see FIG. 5).

The frame image analysis unit 13 extracts a person image from the extracted frame image data, and stores the information about the extracted person image (person information 411 in FIG. 2) in the temporary storage unit 12 as the person data 41 (see FIG. 6). ). In addition, the frame image analysis unit 13 stores the person information 411 in the person data 41 of the database 4.
The flow line analysis unit 14 tracks the movement trajectory of the person image in each continuous frame image data using the result analyzed by the frame image analysis unit 13, and extracts the flow line data 42. Then, the flow line analysis unit 14 stores the information regarding the extracted flow lines in the flow line data 42 of FIG. 7 stored in the database 4.

The total residence time analysis server 2 has a total residence time analysis unit 21.
The total residence time analysis unit 21 once re-searches whether or not the information regarding the flow line stored in the database 4 is the flow line of the same person. Then, the total residence time analysis unit 21 calculates the total residence time of each person based on the result of the re-search. Further, the total residence time analysis unit 21 determines whether or not an alarm notification is necessary based on the calculated total residence time.

The output server 5 outputs the total residence time calculated by the total residence time analysis unit 21 to the output device 6, and issues an alarm to the output device 6 when the total residence time analysis unit 21 determines that alarm notification is necessary. Output.

The database 4 stores the person data 41 (see FIG. 6), the flow line data 42 (see FIG. 7), the flow line rediscovery data 44 (see FIG. 9), and the total residence time data 45 (see FIG. 10). There is. Further, the temporary storage unit 12 stores the person data 41 (see FIG. 6) and the frame data 43 (see FIG. 8). That is, the same person data 41 is stored in the temporary storage unit 12 and the database 4. The respective data 41 to 45 stored in the database 4 and the temporary storage unit 12 will be described later.

In the example shown in FIG. 1, the video analysis server 1, the total residence time analysis server 2, the database 4, and the output server 5 are separate devices, but they may be integrated devices. Alternatively, two or more devices may be installed as one device.

(Frame image analysis unit 13)
FIG. 2 is a diagram showing a detailed configuration of the frame image analysis unit 13 in the first embodiment. In FIG. 2, the same components as those in FIG. 1 are designated by the same reference numerals and description thereof will be omitted.
The frame image analysis unit 13 has a frame image data acquisition unit 131 and a person extraction processing unit 132.
The frame image data acquisition unit 131 acquires the frame image data stored in the temporary storage unit 12 and passes it to the person extraction processing unit 132.
The person extraction processing unit 132 extracts a person image from the frame image data acquired by the frame image data acquisition unit 131. Then, the person extraction processing unit 132 stores the information (person information 411) related to the extracted person image in the person data 41 (see FIG. 6) of the temporary storage unit 12. In addition, the person extraction processing unit 132 stores the person information 411 in the person data 41 of the database 4.

(Flow line analysis unit 14)
FIG. 3 is a diagram showing a detailed configuration of the flow line analysis unit 14 in the first embodiment. In FIG. 3, the same components as those in FIG. 1 are designated by the same reference numerals and description thereof will be omitted.
The flow line analysis unit 14 has a person information acquisition unit 141 and a flow line detection processing unit (determination unit) 142.
The person information acquisition unit 141 acquires the person information 411 from the person data 41 (see FIG. 6) stored in the temporary storage unit 12.
The flow line detection processing unit 142 detects the flow line, which is the locus of each person in the continuous frame image data, based on the person information 411 acquired by the person information acquisition unit 141.
The flow line information (flow line information 421) generated as a result of processing by the flow line detection processing unit 142 is stored in the flow line data 42 stored in the database 4.

(Total residence time analysis unit 21)
FIG. 4 is a diagram showing a detailed configuration of the total residence time analysis unit 21 in the first embodiment. In FIG. 4, the same components as those in FIG. 1 are designated by the same reference numerals, and the description thereof will be omitted.
The total residence time analysis unit 21 includes a periodic search unit 211, a flow line re-detection unit 212, a total residence time calculation unit 213, and an alarm determination processing unit 214.
When the user is set to periodically search for the total residence time, the periodic search unit 211 determines whether or not the search time for the total residence time has been reached. The periodic search unit 211 can be omitted.
The flow line re-detection unit 212 once re-searches the information regarding the flow line stored in the database 4 to see if it is the flow line of the same person. The flow line rediscovery information 441 and the like, which are the results of the re-search, are stored in the flow line rediscovery data 44 (see FIG. 9) of the database 4.
The total residence time calculation unit 213 calculates the total residence time of each person based on the flow line rediscovery data 44 stored in the database 4.
The alarm determination processing unit 214 determines whether or not an alarm notification is necessary based on the calculated total residence time. The total residence time calculated by the total residence time calculation unit 213 and the necessity determination result of the alarm notification by the alarm determination processing unit 214 are stored in the total residence time data 45 (see FIG. 10) stored in the database 4. To.

(Hardware configuration)
FIG. 5 is a diagram showing a hardware configuration diagram of the arithmetic unit 7.
Here, the arithmetic unit 7 constitutes a moving image analysis server 1, a total residence time analysis server 2, and an output server 5.
The arithmetic unit 7 includes a memory 71, a CPU 72 (Central Processing Unit), a storage device 73 such as an HD (Hard Disk), a communication device 74, and an input device 75.
The communication device 74 transmits / receives information to / from another device and the database 4.
The input device 75 is a keyboard, a mouse, or the like.
Further, the program stored in the storage device 73 is loaded into the memory 71. Then, the loaded program is executed by the CPU 72. As a result, the moving image data processing unit 11, the frame image analysis unit 13, the flow line analysis unit 14, the total residence time analysis unit 21, and the frame image analysis unit 13, the flow line analysis unit 14, and the total residence time analysis unit 21 are configured. Each part 131 to 132, 141 to 142, 211 to 214 is embodied.

<Database 4>
Next, each data 41 to 45 stored in the database 4 and the temporary storage unit 12 will be described with reference to FIGS. 6 to 10.
(Person data 41)
FIG. 6 is a diagram showing an example of person data 41.
The person data 41 has columns of "person ID", "frame ID", "rectangular information", "time stamp", "person feature amount", "traffic line ID", and "person image".
In the "person ID" field, an ID uniquely assigned to the person image detected in the frame image data is stored. Even for the same person, if the frame image data of the detection source is different, different person IDs are given.
In the "frame ID" column, an ID assigned to the frame image data in which the person image indicated by the "person ID" is detected is stored.
Information about a rectangle including a person image is stored in the "rectangle information" column.
In the "time stamp" column, information regarding the date and time when the frame image data indicated by the "frame ID" was captured is stored.
In the "person feature amount" column, information about the feature indicated by the detected person image is stored.
In the "traffic line ID" column, the ID given by the flow line analysis unit 14, and the flow line ID in which the ID connecting the person IDs determined to be the same person is stored, is traced to the flow line ID of a certain person. You can follow the flow line (trajectory of movement).
The detected person image data is stored in the "person image" column.

(Flow line data 42)
FIG. 7 is a diagram showing an example of the flow line data 42.
The flow line data 42 has columns for "flow line ID", "list of person IDs", "start frame ID", "end frame ID", and "presence / absence of frame out".
In the "traffic line ID" column, the same data as the "traffic line ID" of the person data 41 is stored.
In the "list of person IDs" column, person IDs to which the same flow line ID is assigned are stored as a list.
In the "start frame ID" field, the frame ID of the first frame in which the person image to which the target flow line ID is assigned is detected is stored.
In the "end frame ID" column, the frame ID of the last frame in which the person image to which the target flow line ID is assigned is detected is stored.
Information regarding the presence / absence of flameout is stored in the "presence / absence of frameout" column. Whether or not there is a frame out is determined by whether or not the person image is at the edge of the frame image in the last frame of the flow line. If the person image is at the end of the frame image in the last frame of the flow line, "Yes" is stored in the "Presence / absence of frame out" column.

Note that, in FIG. 7, the flow lines indicated by the respective flow line IDs are recognized as the flow lines of different persons even if there is a flow line of the same person.

(Frame data 43)
FIG. 8 is a diagram showing an example of frame data 43.
The frame data 43 has columns for "frame ID", "frame image", and "thumbnail image".
The "frame ID" column is an ID uniquely assigned to each frame image.
Frame image data is stored in the "frame image" field.
In the "thumbnail image" field, a thumbnail image in which the size of the frame image data in the "frame image" is reduced is stored.

(Flow line rediscovery data 44)
FIG. 9 is a diagram showing an example of the flow line rediscovery data 44.
The flow line rediscovery data 44 includes "flow line ID", "retention start time", "retention end time", "retention time (seconds)", "personal feature amount", "presence or absence of the same person", and "same person ID". It has each column of.
In the "flow line ID" column, the same as the flow line ID in the person data 41 of FIG. 6, an ID connecting the person IDs determined to be the same person is stored.
The "retention start time" column is the time indicated by the first frame image in a series of frame images including the person ID connected by the flow line ID (that is, the frame image connected by the flow line ID).
The "retention end time" column is the time indicated by the last frame image in the series of frame images including the person ID connected by the flow line ID (that is, the frame image connected by the flow line ID).
In the "retention time (seconds)" column, the time obtained by subtracting the "retention start time" from the "retention end time" is stored.
In the "person feature amount" column, the person feature amount of the person image associated with the flow line ID in the flow line data 42 of FIG. 7 is stored.
In the "presence / absence of same person" column, information indicating whether or not there is another flow line ID of the same person as the target flow line ID in the flow line stored in the flow line rediscovery data 44 is stored. The flow.
In the "same person ID" column, a common ID given to a flow line determined to be the same person is stored. In the example of FIG. 9, each of the flow line IDs to which "A" is given as the "same person ID" is a flow line ID determined to be the same person. Further, each of the flow line IDs to which "B" is assigned is a flow line ID that is determined to be the same person, although they are different from "A". The information stored in the "same person ID" is generally not the name of the person but the ID number.

(Total residence time data 45)
FIG. 10 is a diagram showing an example of total residence time data 45.
The total residence time data 45 has columns for "same person ID", "total residence time", and "excess determination".
The same person ID as in FIG. 9 is stored in the "same person ID" column.
In the "total residence time" column, the total residence time obtained by adding all the residence times of the flow line IDs to which the same person ID is given in the flow line rediscovery data 44 of FIG. 9 is stored.
In the "excess determination" column, whether the total residence time exceeds a predetermined time (first threshold value V1) ("excess") or the total residence time is a predetermined time (second threshold value V2; V1> V2). Information such as whether it exceeds (“warning”) is stored. Further, when the total residence time does not exceed a predetermined time (second threshold value V2), "normal" is stored in the "excess determination" column.

<Flow chart>
(Overall processing)
11 and 12 are flowcharts showing the procedure of the long-term resident detection process performed in the first embodiment.
First, the flow line collection process will be described with reference to FIG.
FIG. 11 is a flowchart showing the procedure of the flow line collection process performed in the first embodiment.
First, the moving image data processing unit 11 and the frame image analysis unit 13 perform moving image analysis processing (S1). Details of the video analysis process will be described later.
Next, the flow line analysis unit 14 performs the flow line detection process (S2). Details of the flow line detection process will be described later. Information about the flow line is collected by steps S1 and S2.

Next, with reference to FIG. 12, the total residence time extraction process performed in response to the process of FIG. 11 will be described.
FIG. 12 is a flowchart showing the procedure of the total residence time extraction process performed in the first embodiment.
First, the total residence time analysis unit 21 performs the flow line re-detection process (S4). Details of the flow line rediscovery process will be described later. In step S4, it is searched in steps S1 and S2 whether or not the flow lines recognized as belonging to different persons are the flow lines of the same person.
Subsequently, the total residence time calculation unit 213 performs a total residence time calculation process for calculating the total residence time for each person using the result of step S4 (S5). Details of the total residence time calculation process will be described later.
After that, the output processing unit 51 performs screen display processing for displaying the total residence time calculated by the total residence time calculation processing on the display device (S6). Details of the screen display process will be described later.

(Video analysis processing)
FIG. 13 is a flowchart showing a detailed procedure of the moving image analysis process (step S1 in FIG. 11) performed in the first embodiment.
First, the moving image data processing unit 11 acquires the moving image data captured by the moving image shooting device 3 (S101).
Next, the moving image data processing unit 11 acquires each of the frame image data constituting the moving image data from the acquired moving image data (S102), and stores each frame image data in the temporary storage unit 12 (S103). The moving image data processing unit 11 assigns a frame ID to each of the acquired frame image data, and then stores the frame image data in the frame data 43 (see FIG. 8) of the temporary storage unit 12.

Subsequently, the frame image data acquisition unit 131 acquires frame image data from the temporary storage unit 12 (S111).
Next, the person extraction processing unit 132 extracts a person image from the frame image data acquired based on the person feature amount (S112). Here, all the person images existing in the frame image are extracted.
Then, the person extraction processing unit 132 assigns a person ID to each of the extracted person images, and the person data 41 (see FIG. 6) for each person feature amount, data related to the frame image, and the person image from which the person image data is extracted. ) (S113).

Subsequently, the flow line analysis unit 14 determines whether or not the extraction of the person images of all the persons has been completed in the frame image to be processed (completed for all persons ?; S114).
When the extraction of the human image of all the persons is not completed (S114 → No), the flow line analysis unit 14 returns the process to step S111.
When the extraction of the human image of all the persons is completed (S114), the moving image analysis server 1 returns the process to step S2 in FIG.

(Flow line detection processing)
FIG. 14 is a flowchart showing a detailed procedure of the flow line detection process (step S2 in FIG. 11) performed in the first embodiment.
First, the person information acquisition unit 141 acquires the person information 411 from the person data 41 (S201). As described above, the person information 411 indicates a record of the person data 41 shown in FIG.
Subsequently, the movement line detection processing unit 142 compares the person feature amount in the frame image immediately before the current frame image with the person feature amount calculated in the extraction of the person image in step S112 in the person data 41. By doing so, it is determined whether or not the person information 411 (similar person information 411) having a similar person feature amount exists in the previous frame image (S202). The similar person feature amount means that the difference between the person feature amounts has a value within a predetermined range (person feature amount threshold value). The person information 411 is a record of the person data 41 shown in FIG.

When the similar person information 411 does not exist in the previous frame image (S202 → No), the flow line detection processing unit 142 newly assigns the flow line ID to the corresponding person information 411 (S203).
Further, the flow line detection processing unit 142 stores the flow line ID newly assigned in step S202 in the "flow line ID" column of the flow line data 42 shown in FIG. 7, and processes it in the "person ID list" column. The person ID of the target person information 411 is stored (S204). In step S203, the flow line detection processing unit 142 stores the frame ID of the person information 411 to be processed in the "start frame ID" column of the flow line data 42 of FIG. 7. At this time, the flow line detection processing unit 142 also stores the flow line ID assigned to the flow line ID column of the person data 41.

When there is similar person information 411 in the previous frame image (S202 → Yes), the flow line detection processing unit 142 determines whether or not the person image is being tracked for a predetermined time or longer (S211). .. In step S211 first, the flow line detection processing unit 142 acquires the flow line information 421 having the same flow line ID as the flow line ID in the similar person information 411 in the flow line data 42 shown in FIG. 7. Here, the flow line information 421 is a record in the flow line data 42. Subsequently, the flow line detection processing unit 142 acquires the frame ID stored in the "start frame ID" column of the acquired flow line information 421. Here, the acquired frame ID is used as the start frame ID. Then, the flow line detection processing unit 142 searches for the person data 41 using the acquired start frame ID as a key, and acquires a time stamp stored in association with the frame ID corresponding to the start frame ID. Here, the acquired time stamp is referred to as a start time stamp. Then, by comparing the time stamp stored in the person information 411 acquired in step S201 with the start time stamp, it is determined whether or not the tracking of the person image is performed for a predetermined time or more.

When the tracking of the person image is performed for a predetermined time or longer (S211 → Yes), the flow line detection processing unit 142 temporarily cuts off the flow line by generating a new flow line ID (S203). If the flow line continues indefinitely, the flow line re-detection process described later cannot be started for the person image, but such a problem can be solved by temporarily cutting off the flow line at regular intervals. be able to.

When the tracking of the person image has not been performed for a predetermined time or more (S211 → No), the flow line detection processing unit 142 uses the flow line ID in the similar person information 411 as the flow line of the person information 411 to be processed. Store in the ID field (S212).
Further, in the flow line data 42 shown in FIG. 7, the flow line detection processing unit 142 is the person to be processed in the "person ID list" column of the flow line ID record to be processed in step S204. The person ID of the information 411 is stored (S213).

Then, the flow line detection processing unit 142 determines whether or not the association with the flow line ID is completed for all the person information 411 (completed for all the person information 411; S214).
When the association with the flow line ID is not completed for all the person information 411 (S214 → No), the flow line detection processing unit 142 returns the processing to step S201.
When the association with the flow line ID is completed for all the person information 411 (S214 → Yes), the flow line detection processing unit 142 detects the end frame image for each flow line ID, and is shown in FIG. 7. The frame ID of the detected end frame image is stored in the "end frame ID" column of the flow line data 42 (S215). After that, the moving image analysis server 1 returns to the process of FIG.

Although not shown in FIG. 14, the flow line detection processing unit 142 searches for the end frame image based on the end frame ID of the flow line data 42. Then, when the person image is at the end of the frame image in the end frame image, the flow line detection processing unit 142 stores “Yes” in the “Presence / absence of frame out” column of the flow line data 42.

(Flow line rediscovery process)
FIG. 15 is a flowchart showing a detailed procedure of the flow line rediscovery process (step S4 in FIG. 12) according to the first embodiment.
First, the periodic search unit 211 determines whether or not the time set by the user has elapsed from the time stamp referred to in the previous retention search (S401). In the present embodiment, it is said that the process of step S402 or less is performed when the time set by the user elapses, but the process of step S402 or less is performed when the user instructs to start the search for the total residence time. May be good.
If a certain period of time has not elapsed (S401 → No), the periodic search unit 211 returns the process to step S401.
When the time set by the user has elapsed (S401 → Yes), the flow line re-detection unit 212 acquires all the flow line IDs within the specified time range in the flow line data 42 from the flow line data 42 (S403). Here, the designated time range is input by the user via the input device 75. The flow line rediscovery unit 212 stores the flow line ID acquired in step S403 in the “flow line ID” column of the flow line rediscovery data 44 shown in FIG.

Then, the flow line re-detection unit 212 selects one flow line ID from the acquired flow line IDs as the flow line re-detection target (S404).
Then, the flow line re-detection unit 212 searches for the person image in the flow line ID acquired in step S404 and the person image considered to be the same person among the person images in the flow line ID acquired in step S403 (same person). Search; S405). Specifically, the flow line re-detection unit 212 acquires the person ID stored in the flow line information 421 having the flow line ID acquired in step S403. This person ID is referred to as a flow line rediscovery person ID. Subsequently, the flow line re-detection unit 212 searches for the person data 41 and acquires the person feature amount of the flow line re-detection person ID. Here, the acquired person feature amount is referred to as a flow line rediscovered person feature amount. Further, a person having a flow line rediscovered person feature amount is referred to as a flow line rediscovered person.

Subsequently, the flow line re-detection unit 212 acquires the person ID in the flow line information 421 having the flow line ID acquired in step S403. Then, the flow line re-detection unit 212 compares the person feature amount acquired from the person data 41 based on this person ID with the flow line re-detection person feature amount.

As a result of the search, the flow line re-detection unit 212 determines whether or not there is a person image that can be considered to be the same person as the flow line rediscovered person among the person images in the flow line ID acquired in step S403 (same). Person existence; S411). Specifically, the flow line re-detection unit 212 determines whether or not there is a person feature amount similar to the flow line rediscovered person feature amount (within a predetermined threshold value).
When there is no person image considered to be the same person (S411 → No), the flow line re-detection unit 212 displays the flow line re-detection information 441, which is information on the flow line re-detection person special amount, as a single person in FIG. It is stored in the flow line rediscovery data 44 (S412), and the process proceeds to step S414.
When there is a person image considered to be the same person (S411 → Yes), the flow line re-detection unit 212 stores the flow line ID considered to be the same person in the flow line re-detection data 44 shown in FIG. 9 (S413). ..

Here, the details of step S412 and step S413 will be described with reference to the flow line rediscovery data 44 shown in FIG.
(A) When the same person does not exist.
The flow line re-detection unit 212 stores information in each column of the "retention start time", "retention end time", and "person feature amount" of the record corresponding to the flow line ID of the flow line re-detection person. Incidentally, for the "retention start time" and the "retention end time", the "start frame ID" and the "end frame ID" of the flow line data 42 are acquired by using the flow line ID as a key. Further, the flow line rediscovery unit 212 searches for the person data 41, acquires the acquired time stamps of the “start frame ID” and the “end frame ID”, and uses the acquired time stamps as the flow line rediscovery data. It is stored in the "retention start time" and "retention end time" columns of 44.

Further, the flow line re-detection unit 212 stores the flow line re-detection person ID in the “person feature amount” column.
Further, the flow line re-detection unit 212 stores "none" in the "presence / absence of the same person" column, and stores the same person ID in the "same person ID" column so as not to be common with other person images. ("C" in the example of FIG. 9).

(B) When the same person exists The flow line re-detection unit 212 includes the flow line ID of the flow line rediscovered person and the flow line ID of the person image determined to be the same person as the flow line rediscovered person in step S405. Information is stored in each column of "retention start time", "retention end time", and "personal feature amount" in each of the records corresponding to. Since the information in these columns is the same as in (A) the case where the same person does not exist, the description here will be omitted. Further, "Yes" is stored in the "Presence / absence of the same person" column, and the same person ID that is not common to other person images is stored in the "Same person ID".
At this stage, the flow line re-detection unit 212 calculates the residence time for each flow line ID by calculating the difference between the retention end time and the retention start time (S414), and the calculated residence time is used as the flow line. It is stored in the "residence time" column of the rediscovery data 44. The residence time calculated in step S414 is a residence time corresponding to each flow line.

After step S414, the flow line rediscovery unit 212 excludes the flow line ID of the flow line rediscovered person and the flow line ID determined to be the same person as the flow line rediscovered person (S415). ..
Then, the flow line re-detection unit 212 determines whether or not the flow line ID acquired in step S202 remains (S416).
When the flow line ID remains (S416 → Yes), the process returns to step S403, and the flow line re-detection unit 212 selects a new flow line ID from the remaining flow line IDs.
When the flow line ID does not remain (S416 → No), the total residence time analysis unit 21 returns the process to step S5 in FIG.

With conventional technology, a flow line that is once interrupted is recognized as another person's flow line. Therefore, it was difficult to obtain an accurate total residence time. On the other hand, in the first embodiment, as shown in FIG. 15, data on all the flow lines is collected, and whether or not the same person's flow lines exist for each flow line is searched again. As a result, even if the flow line of the same person is interrupted once, it can be recognized as the same person, so that the calculation accuracy of the total residence time described later can be improved.

(Total residence time calculation process)
FIG. 16 is a flowchart showing a detailed procedure of the total residence time calculation process (step S5 in FIG. 12) performed in the first embodiment.
First, the total residence time calculation unit 213 refers to the flow line rediscovery data 44 shown in FIG. 9 and selects one flow line ID (S501).
After that, the total residence time calculation unit 213 refers to the "same person ID" column of the flow line rediscovery data 44, and has the same person ID as the same person ID associated with the flow line ID selected in step S501. All the flow line IDs having the above are acquired (S502).

Next, the total residence time calculation unit 213 refers to the “residence time” column of the flow line rediscovery data 44 shown in FIG. 9, and the residence time associated with the flow line ID acquired in steps S501 and S502. (Total residence time) is calculated (S503). The total residence time calculation unit 213 has the same person ID and the total residence time calculated in step S503 in the "same person ID" column and the "total residence time" column of the total residence time data 45 shown in FIG. Is stored in association with each other.

Subsequently, the total residence time calculation unit 213 determines whether or not the calculated total residence time T exceeds a predetermined threshold value V (T> V ?; S511).
When the total residence time does not exceed a predetermined threshold value (S511 → No), the total residence time calculation unit 213 proceeds to step S513.
When the total residence time exceeds the processing threshold value (S511 → Yes), the output processing unit 51 outputs an alarm by displaying the pop-up screen 812 on the monitoring moving image display screen 810 shown in FIG. 18 (S512).

Then, the total residence time calculation unit 213 searches the flow line rediscovery data 44 and determines whether or not the unprocessed flow line ID exists (S513).
When the unprocessed flow line ID exists (S513 → Yes), the total residence time calculation unit 213 returns the process to step S501.
When the unprocessed flow line ID does not exist (S513 → No), the total residence time analysis unit 21 returns the process to step S6 in FIG.

(Total residence time detail screen display processing)
FIG. 17 is a flowchart showing a detailed procedure of the total residence time detailed screen display process (step S6 in FIG. 12) performed in the first embodiment.
The total residence time detail screen display process shown in FIG. 17 is a process for displaying the total residence time detail screen 820 shown in FIG.
When the display instruction of the total residence time detail screen 820 shown in FIG. 19 is given, the output processing unit 51 increases the total residence time stored in the "total residence time" column of the total residence time data 45 shown in FIG. 10 in descending order. Sort (S601). The display instruction of the total residence time detail screen 820 will be described later.
Subsequently, the output processing unit 51 selects one same person ID of the total residence time data 45 (S602).
Next, the output processing unit 51 refers to the "total residence time" column of the total residence time data 45, and determines whether or not the total residence time T corresponding to the selected same person ID exceeds the first threshold value V1. (T> V1 ?; S603).
When the first threshold value V1 is exceeded (S603 → Yes), the output processing unit 51 stores “excess” in the “excess determination” column in the corresponding record of the total residence time data 45 (see FIG. 10) (S611). .. Then, the bar graph 821 of the person to be processed on the total residence time detail screen 820 shown in FIG. 19 is displayed in, for example, “red” (S612). That is, an output indicating that the total residence time exceeds a predetermined threshold value is output.

When the first threshold value V1 is not exceeded (S603 → No), the output processing unit 51 determines whether or not the total residence time T corresponding to the selected same person ID exceeds the second threshold value V2 (T> V2). S621). However, the first threshold value V1> the second threshold value V2.
When the second threshold value V2 is exceeded (S621 → Yes), the output processing unit 51 stores “warning” in the “excess determination” column in the corresponding record of the total residence time data 45 (S622). Then, the bar graph 821 of the person to be processed on the total residence time detail screen 820 shown in FIG. 19 is displayed in, for example, “orange” (S623).

When the second threshold value V2 is not exceeded (S621 → No), the output processing unit 51 stores “normal” in the “excess determination” column in the corresponding record of the total residence time data 45 (S631). Then, the bar graph 821 of the person to be processed on the total residence time detail screen 820 shown in FIG. 19 is displayed in, for example, “green” (S632).

After steps S612, S623, and S632, the output processing unit 51 determines whether or not the processing has been completed for all the same person IDs (S633).
When the processing is not completed for all the same person IDs (S633 → No), the output processing unit 51 returns the processing to step S602 and acquires one unprocessed same person ID (S602).
When the processing is completed for all the same person IDs (S633 → Yes), the output processing unit 51 returns to the processing of FIG.

In this embodiment, the excess determination is shown in three stages of "excess", "warning", and "normal", but the present invention is not limited to this. For example, the excess determination may be indicated in four stages of "excess", "warning", "caution", and "normal".

<Screen example>
(Surveillance video display screen 810)
FIG. 18 is a diagram showing a surveillance moving image display screen 810.
The monitoring video display screen 810 has a monitoring video display unit 811, a total residence time detail screen display button 813, and the like.
The surveillance moving image display unit 811 displays the moving image captured by the moving image shooting device 3.
When the user selects and inputs the total residence time detail screen display button 813 (display instruction of the total residence time detail screen 820), the total residence time detail screen 820 shown in FIG. 19 is displayed on the display unit.

Further, in the total residence time data 45 shown in FIG. 10, if there is a person (“excess”) whose total residence time exceeds a predetermined time, the total residence time is exceeded as shown in FIG. A pop-up screen 812 notifying that there is a person is displayed. When the user selects and inputs the pop-up screen 812, the total residence time detail screen 820 shown in FIG. 19 is displayed on the display unit (display instruction of the total residence time detail screen 820).

In the example shown in FIG. 18, the pop-up screen 812 notifies the person who is determined to be "excessive" in the total residence time data 45 shown in FIG. However, the present invention is not limited to this, and a notification by the pop-up screen 812 may be given to a person who is determined to be "alert" or "attention".

(Total residence time detail screen 820)
FIG. 19 is a diagram showing a total residence time detail screen 820.
As described above, the total residence time detail screen 820 is displayed by selecting and inputting the total residence time detail screen display button 813 and the pop-up screen 812 in FIG.
The total residence time detail screen 820 has a total residence time display unit 825 and a person image display unit 823.
The total residence time display unit 825 displays the total residence time for each person as a bar graph 821 based on the "total residence time" stored in the total residence time data 45 shown in FIG. In the example shown in FIG. 19, the total residence time of four persons (retention persons) of "A", "B", "C", and "D" is displayed. Note that "A", "B", "C", and "D" are not the names of the respective persons, but the identification numbers and the like. When the user selects and inputs the bar graph 821 from any of them via the input device 75, the personal residence information display screen 830 shown in FIG. 20 is displayed, and the person associated with the selected and input bar graph 821 is displayed. The stagnation information is displayed. Here, the broken line 822 indicates the excess determination threshold value (first threshold value V1 in FIG. 17), and the bar graph 821a of the person “A” whose total residence time exceeds the broken line 822 is displayed in red, for example. Further, although the total residence time does not exceed the broken line 822, the total residence time is close to the broken line 822 (in FIG. 17, the total residence time T is V1>T>V2; V2 is the second threshold value). 821b is displayed in orange, for example. The bar graphs 821c of the other persons "C" and "D" are displayed in green indicating that the total residence time is normal. The color of the bar graph 821 is determined by the output processing unit 51 searching for the total residence time data 45 (see FIG. 10) as described above. The normal total residence time here means that the total residence time is not long-term.

The person image display unit 823 displays an image (person image) of a person whose total residence time is displayed on the total residence time display unit 825. This person image is acquired by the output processing unit 51 referring to the "person image" column of the person data 41 in FIG. The output processing unit 51 acquires the flow line ID associated with the “same person ID” column for the person whose total residence time in the flow line rediscovery data 44 shown in FIG. 9 is displayed. Subsequently, the output processing unit 51 refers to the “list of person IDs” column of the flow line data 42 shown in FIG. 7, and acquires the start frame ID or the end frame ID associated with the flow line ID). Here, it is assumed that the start frame ID is acquired. At this time, there are a plurality of candidates for the start frame ID to be acquired, but only one of them (for example, the start frame ID stored first) may be acquired.

(Personal retention information display screen 830)
FIG. 20 is a diagram showing a personal residence information display screen 830.
The personal residence information display screen 830 is a screen that displays detailed information about the residence time of a person determined to be the same by the moving image analysis server 1. As shown in FIG. 20, the personal residence information display screen 830 includes a residence time information display unit 831, a person image display unit 836, a check box 837, a delete button 838, a total residence time display unit 839B, and a search start time display unit 839A. Have. In the example of FIG. 20, the search start time of the past 4 hours from the current time (13:00:00), that is, the residence time is set to 9:00:00.

In the residence time information display unit 831, the residence time of the person determined by the video analysis server 1 to be the same is displayed for each flow line. That is, each of the flow line lines 832 displayed on the residence time information display unit 831 corresponds to each of the flow line IDs in FIG. 7. Further, the line 834A in the residence time information display unit 831 indicates the search start time, and the line 834B indicates the current time. That is, the moving image analysis server 1 searches for the residence time for the time between the line 834A and the line 834B (arrow 835). Further, the frame-out display 833 indicates the time when the corresponding person is frame-out. The frame-out display 833 is displayed by the moving image analysis server 1 referring to the “presence / absence of frame-out” column shown in FIG. That is, the output processing unit 51 refers to the “presence / absence of frame out” column of the flow line data 42. If the frame out is "Yes", the output processing unit 51 displays a frame out display 833 between the flow line 832 indicated by the flow line information 421 and the next flow line 832.

The person image display unit 836 displays the person images 836a and 836b associated with the flow line line 832. That is, the person image display unit 836 displays the person images 836a and 836b corresponding to the person ID in the "list of person IDs" column associated with the flow line ID shown by the flow line line 832 in FIG. 7. ing. Further, check boxes 837 are associated with and displayed on the respective person images 836a and 836b. It will be described later that the person image 836a is different from the other person images 836b.

The search start time is displayed on the search start time display unit 839A. The total residence time display unit 839B displays the total residence time calculated in step S503 of FIG. That is, the total residence time display unit 839B displays the total time indicated by each of the flow line lines 832 displayed on the residence time information display unit 831.

Here, the person image 836a is different from other person images 836b. This indicates that the moving image analysis server 1 erroneously recognizes the person in the person image 836a and the person in the other person image 836b as the same person. In such a case, as shown in FIG. 20, the user checks the check box 837 (check box 837a in the example of FIG. 20) which is displayed in association with the person image 836b via the input device 75. input. After that, when the user selects and inputs the delete button via the input device 75, the person image 836a and the flow line 832a associated with the person image 836a are deleted. In addition, although one person image 836a and the flow line 832a are deleted in FIG. 20, a plurality of person images 836a and the flow line 832a are deleted by inputting a check in a plurality of check boxes 837. It is possible. That is, the information of the predetermined flow line can be deleted. After deleting the person image 836a and the flow line 832a, the moving image analysis server 1 recalculates the total sum (total residence time) of the flow line 832 displayed on the personal residence information display screen 830.

According to the first embodiment, it is determined whether or not the person is the same person based on the person feature amount for a plurality of interrupted flow lines, and if it is determined to be the same person, it is regarded as the same person's flow line. Calculate the total residence time. By doing so, it is possible to calculate the residence time over a long period of time by connecting the flow lines of the same staying person who has been interrupted by the person feature amount. Further, according to the first embodiment, even if a frameout occurs, it is determined whether or not the person is the same person based on the person feature amount for a plurality of interrupted flow lines, so that the same person can be used. The flow line can be detected. Therefore, the total residence time can be calculated even if flameout occurs.

As shown in FIG. 20, when a different person is erroneously recognized as a person for which the total residence time is calculated, a check is entered in the check box 837 of the different person, and the delete button 838 is selected and input. It is possible to exclude the flow line of the recognized person. As a result, since the error of the tracking error does not accumulate, it is possible to monitor the residence for a long time.

Further, as shown in FIG. 19, the total residence time for each person is shown by a bar graph 821 together with the image of the person, and the total residence time calculated by changing the color of the bar graph 821 of the person exceeding a predetermined threshold value is shown. When it exceeds a predetermined threshold value, it indicates that the total residence time exceeds the predetermined threshold value. By doing so, the user can easily recognize a person whose total residence time exceeds a predetermined threshold value.

[Second Embodiment]
Next, a second embodiment of the present invention will be described with reference to FIGS. 21 to 23. The second embodiment is an invention relating to prevention of flow line transfer.
FIG. 21 is a diagram illustrating a flow line transfer.
In FIG. 21, the person H1 is detected at first, but it is detected in a state where the person H2 overlaps the person H1 at the time T1. If the feature quantities of the person H1 and the person H2 are close to each other, the moving image analysis server 1 confuses the person H2 with the person H1, and after the time T1, the person H2 is regarded as the person H1 and the flow line is detected. That is, the flow line shift occurs (thick solid line arrow 901). In the second embodiment, it is an object to prevent such a flow line transfer. It is assumed that the person H1 stops for a while at the time T1 and then starts moving. As a result, after the time T1, the person H1 is detected as a flow line (thick broken line arrow 902) different from the thick solid line arrow 901.

<Flow chart>
FIG. 22 is a flowchart showing the procedure of the total residence time extraction process performed in the second embodiment.
In FIG. 22, the same processing as in FIG. 12 is assigned the same step number and the description thereof will be omitted.
In FIG. 22, the process different from that in FIG. 12 is that the user first sets the person feature amount threshold value to be less than a predetermined value via the input device 75 (step S3). At this time, the person feature amount threshold value is set to be as small as possible.
By setting the person feature amount threshold value small in this way, the probability of being determined as “No” in step S202 of FIG. 14 increases. That is, the flow line is intentionally made easy to be interrupted. The effect of doing so will be described with reference to FIG.

FIG. 23 is a diagram showing the result of performing the process shown in FIG. 22.
In FIG. 23, as in FIG. 21, the person H1 is detected (traffic line M0), but it is detected in a state where the person H2 overlaps the person H1 at the time T1. However, since the person feature amount threshold value is small, it is possible to distinguish between the person H1 before the time T1 and the person H2 overlapping the person H1 at the time T1. Therefore, the flow line is interrupted at time T1. The flow line at time T1 is the first of the flow lines M1.

Then, after that, the flow line re-detection process shown in FIG. 15 is performed, and each of the flow line M1 and the flow line M2 is compared with the person of the other flow line, and the flow lines of the person considered to be the same are compared with each other. It is summarized as.

With the conventional technology, if it is determined that the flow lines are different, even the flow lines of the same person are actually recognized as the flow lines of different people. Therefore, in the conventional technique, if the person feature amount threshold is set small as in the present embodiment, the flow line is interrupted when the person feature amount of the same person changes due to, for example, the amount of light, and the flow line is different. There is a risk that it will be recognized as the flow line of the person. Therefore, with the conventional technology, it is not possible to set the person characteristic tension threshold value small.

On the other hand, in the second embodiment, even if the flow line is interrupted, the flow line of the same person is collected by the flow line re-detection process shown in FIG. 15, so that the person feature amount threshold value is set small. be able to. Thereby, according to the second embodiment, it is possible to prevent the flow line transfer.

[Third Embodiment]
Next, a third embodiment of the present invention will be described with reference to FIGS. 24 to 28. In the third embodiment, when it is determined that two interrupted flow lines belong to the same person, the two flow lines are combined so that they can be recognized as one flow line. .. In the first embodiment, the interrupted flow line collects the flow lines of the same person while remaining interrupted. On the other hand, in the third embodiment, if the person feature amount and the position of the person in the image are close to each other, the flow lines are combined and treated as one flow line.

<Flow line analysis unit 14b>
FIG. 24 is a diagram showing a detailed configuration of the flow line analysis unit 14b according to the third embodiment. In FIG. 24, the same components as those in FIG. 3 are designated by the same reference numerals and description thereof will be omitted.
The flow line analysis unit 14b of the moving image analysis server 1b is different from the flow line analysis unit 14 shown in FIG. 3 in that it has a flow line connection unit 144 that connects the flow lines determined to be the same person. ..
Since the configuration of the moving image analysis server 1b other than the flow line coupling portion 144 is the same as the configuration shown in FIGS. 1, 2, and 4, the illustration and description will be omitted.

<Flow line data 42b>
FIG. 25 is a diagram showing an example of the flow line data 42b used in the third embodiment.
In FIG. 25, the difference from the flow line data 42 shown in FIG. 7 is that it has a “next flow line ID” column.
For example, it is assumed that the flow line indicated by the flow line ID "00002" is determined to be the flow line of the same person as the flow line indicated by the flow line ID "00007". In such a case, the flow line ID "00007" is stored in the "next flow line ID" column in the record of the flow line ID "00002". As a result, the flow line of the flow line ID "00002" and the flow line of the flow line ID "00007" are combined and recognized as one flow line. In the example of FIG. 25, the flow line of the flow line ID “00015” is connected to the flow line of the flow line ID “000019”, and further is connected to the flow line of the flow line ID “00023”.

<Flow chart>
(Flow line detection processing)
FIG. 26 is a flowchart showing the procedure of the flow line detection process performed in the third embodiment. In FIG. 26, the same process as in FIG. 13 is assigned the same step number and the description thereof will be omitted.
In FIG. 26, the difference from FIG. 13 is that the flow line coupling process (S221) is performed after step S215. Details of the flow line coupling process will be described later.

(Flow line connection processing)
FIG. 27 is a flowchart showing a detailed procedure of the flow line coupling process (S221 in FIG. 26) performed in the third embodiment.
First, the flow line coupling unit 144 selects two flow line IDs from the flow line data 42 (S701). Here, the selected flow line ID is such that the time of the last frame image of one flow line ID and the time of the first frame image of the other flow line ID exist within a predetermined range. be.
Then, among the flow lines indicated by the two selected flow line IDs, rectangular information is acquired from the last frame image of the previous flow line (referred to as the first flow line) in time (S702). The rectangular information acquired in step S702 is referred to as first rectangular information.
Next, the flow line coupling unit 144 starts from the first frame image of the flow line (referred to as the second flow line) in which the first frame image exists within a predetermined time with respect to the last frame image of the first flow line. Acquire rectangle information (S703). The rectangular information acquired in step S703 is referred to as a second rectangular information. The first rectangle information and the second rectangle information are acquired from the flow line data 42 and the person data 41.

Subsequently, the flow line coupling unit 144 determines whether or not the distance D between the position P1 of the first rectangular information and the position P2 of the second rectangular information is equal to or less than a predetermined value D1 (D ≦ D1; S711). ). Here, the position of each rectangle information is the center of the rectangle or the like, and can be appropriately set by the user.

When the distance D is larger than the predetermined value D1 (S711 → No), the flow line coupling unit 144 proceeds to step S721.
When the distance D is equal to or less than a predetermined value D1 (S711 → Yes), the flow line connecting portion 144 has a person feature amount in the first rectangular information (referred to as the first person feature amount C1) and a person in the second rectangular information. The feature amount (referred to as the second person feature amount C2) is compared. Then, the flow line coupling portion 144 determines whether or not the absolute value of the difference between the first person feature amount C1 and the second person feature amount C2 is equal to or less than a predetermined value C (| C1-C2 | ≦ C). S712).

When the absolute value of the difference between the first person feature amount C1 and the second person feature amount C2 is larger than the predetermined value C (S712 → No), the flow line connecting portion 144 proceeds to step S721.
When the absolute value of the difference between the first person feature amount C1 and the second person feature amount C2 is a predetermined value C or less (S712 → Yes), the flow line connecting portion 144 has the first flow line and the second flow line. Is determined to be the flow line of the same person, and the first flow line and the second flow line are combined (S713). In the flow line data 42b shown in FIG. 25, the flow line coupling unit 144 stores the flow line ID of the second flow line in the “next flow line ID” of the flow line information 421 indicating the first flow line.

After step S711 "No", step S712 "No", and step S713, the flow line coupling unit 144 determines whether or not the processing of steps S701 to S713 has been completed for all the flow line IDs (S721).
When the processing of steps S701 to S713 is not completed for all the flow line IDs (S721 → No), the flow line connecting unit 144 returns the processing to step S701.
When the processes of steps S701 to S713 are completed for all the flow line IDs (S721 → Yes), the flow line analysis unit 14b returns to the process of FIG. 26.

<Specific example of flow line coupling processing>
FIG. 28 is a diagram showing a specific example of the flow line coupling process.
The frame image F1 shows the last frame image of the frame image constituting a certain flow line. Further, the frame image F2 shows a frame image within a predetermined time from the time indicated by the frame image F1, and shows the first frame image among the frame images constituting a flow line different from the flow line to which the frame image F1 belongs. ing. Further, the rectangle R1 includes a person image of the person H11 in the frame image F1. The rectangle R2 and the rectangle R3 include an image of a person (personal image) included in the frame image F2.

Here, the flow line connecting portion 144 compares the distance between the position of the rectangle R1 and the rectangle R2 in the frame image, and the distance between the rectangle R1 and the position of the rectangle R3 in the frame image. This comparison corresponds to the process of step S711 in FIG. In the example of FIG. 28, the distance between the positions of the flow line connecting portion 144 in the frame image between the rectangle R1 and the rectangle R2 (solid arrow) is the distance between the positions in the frame image between the rectangle R1 and the rectangle R3. Closer than (broken arrow). Therefore, the flow line connecting portion 144 compares the person special feature amount in the rectangle R1 with the person feature amount in the rectangle R2. This comparison corresponds to step S712 in FIG. When these rectangular person images have similar person feature quantities, it is determined that the person in the rectangle R1 and the person in the rectangle R2 are the same person, and the flow lines in these people are combined.

According to the third embodiment, the number of flow lines used for the flow line re-detection process can be reduced by combining the flow lines that can be combined before the flow line rediscovery process is performed. As a result, it is possible to keep the probability of recognizing a different person and calculating the total residence time as shown in the flow line line 832a of FIG. 20 low.

The present invention is not limited to the above-described embodiment, and includes various modifications. For example, the above-described embodiment has been described in detail in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to those having all the described configurations. Further, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Further, it is possible to add / delete / replace a part of the configuration of each embodiment with another configuration.

Further, each of the above-mentioned configurations, functions, parts 11, 13, 14, 21, 51, 131, 132, 141, 142, 144, 211 to 214, database 4, etc. includes some or all of them, for example, an integrated circuit. It may be realized by hardware by designing with. Further, as shown in FIG. 5, each configuration, function, etc. described above may be realized by software by interpreting and executing a program in which a processor such as a CPU 72 realizes each function. In addition to storing information such as programs, tables, and files that realize each function in HD (Hard Disk), a memory 71, a recording device such as SSD (Solid State Drive), or an IC (Integrated Circuit) card. It can also be stored in a recording medium such as an SD (Secure Digital) card or a DVD (Digital Versatile Disc).
Further, in each embodiment, the control lines and information lines are shown as necessary for explanation, and not all the control lines and information lines are shown in the product. In practice, you can think of almost all configurations as interconnected.

3 Movie shooting device (imaging unit)
6 Output device (output unit)
11 Video data processing unit (video acquisition unit)
41 Person data (including person images)
75 Input device (input unit)
42b Flow line data (including combined flow line data)
51 Output processing unit 132 Person extraction processing unit (person detection unit)
142 Flow line detection processing unit (judgment unit)
213 Total residence time calculation unit 421 Flow line information 821,821a, 821b Bar graph (information on total residence time, output indicating that the total residence time exceeds the predetermined threshold value)
823 Person image display unit (information on the person whose total residence time is calculated)
837,837a check box (deletes information on the specified flow line)
838 Delete button (Delete information on the specified flow line)
S1 Video analysis process (person detection step)
S2 Flow line detection process (flow line detection process step)
S3 Flow line rediscovery process (judgment step)
S4 Total residence time calculation process (total residence time calculation step)
S5 screen display processing (output processing step)
Z video analysis system (video analysis device)

Claims (7)

1. A video acquisition unit that acquires video from the image pickup unit, and a video acquisition unit
   A person detection unit that detects a person image in each of the acquired images constituting the moving image,
   A flow line detection processing unit that extracts one flow line from the detected person image and detects the flow line information that is information about the extracted flow line.
   A plurality of the once interrupted flow line information detected by the flow line detection processing unit is acquired, and for each of the acquired plurality of the flow line information, whether or not the flow line is the same person is determined. Judgment unit that judges based on the amount of character features in
   A total residence time calculation unit that calculates the total residence time of the person based on the flow line information determined to be the same person, and
   An output processing unit that outputs the calculated total residence time to the output unit, and
   A moving image analysis device characterized by having.
2. The flow line detection processing unit is
   The moving image analysis device according to claim 1, wherein the flow line is detected so that the time indicated by the flow line does not exceed a predetermined time.
3. The output processing unit
   The calculated information on the total residence time and the information on the person for whom the total residence time is to be calculated are displayed in the output unit.
   The moving image analysis apparatus according to claim 1, wherein when the calculated total residence time exceeds a predetermined threshold value, an output indicating that the total residence time exceeds the predetermined threshold value is output.
4. The output processing unit
   Information on the flow line of each person is output to the output unit, and the information is output to the output unit.
   The moving image analysis device according to claim 1, wherein information on a predetermined flow line can be deleted via an input unit.
5. The flow line detection processing unit is
   The flow line is detected based on the person feature amount,
   The moving image analysis device according to claim 1, wherein the threshold value of the person feature amount when extracting the flow line is set to be as small as possible.
6. The flow line detection processing unit is
   When it is determined that different flow lines are the flow lines of the same person based on the person feature amount and the position of the person image in the images before and after, it is characterized by combining these flow lines. The moving image analysis device according to claim 1.
7. A video analysis device that calculates the total residence time of a person
   A person detection step for detecting a person image in each of the images constituting the moving image acquired from the image pickup unit, and
   A flow line detection processing step of extracting one flow line from the detected person image and detecting the flow line information which is information about the extracted flow line, and
   A plurality of the once interrupted flow line information detected by the flow line detection processing step is acquired, and for each of the acquired plurality of the flow line information, whether or not the flow line is the same person is determined. Judgment step to judge based on the person's feature amount in
   A total residence time calculation step for calculating the total residence time of the person based on the flow line information determined to be the same person, and
   An output processing step that outputs the calculated total residence time to the output unit, and
   A video analysis method characterized by executing.

Images