JP2016129306A - Information processing system, information processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make an observer's video confirmation work be efficient, by automatically performing traffic line analysis using imaged video and providing video depending on a result of the analysis.SOLUTION: An information processing system includes: one or more imaging devices for imaging one or more moving objects; and an information processing device. The information processing device includes: traffic line extraction means for extracting traffic lines of the one or more moving objects from video obtained by the imaging devices; range identification means for identifying, on the basis of the traffic lines, at least one of a presentation area and a presentation period to be a presentation range in time and space imaged by the imaging devices; video identification means for identifying, on the basis of the presentation range, presentation video to be presented to a user; and display means for displaying the presentation video.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an information processing system, an information processing method, and a program.

  In recent years, surveillance cameras have been widely used, but there are many environments in which only video is accumulated and real-time video is not utilized. Further, as an analysis technique in the surveillance camera system, an analysis technique using a flow line (individual movement trajectory, or the entire movement trajectory as a human flow or a matrix) has been developed. The results of flow line analysis can be applied to a wide range of customer behavior analysis and abnormality detection in stores, but there are many environments where it is not possible to secure personnel who specialize in flow line analysis. Even in such an environment, in order to improve the service by using the presentation video and flow line analysis results in real time, a mechanism to display the presentation video based on the flow line analysis results on a device independent of the surveillance camera is necessary. It becomes.

  Patent Document 1 discloses a view system that can display a panoramic image along a road. This system helps users understand the surrounding situation by integrating real-time video from a surveillance camera into a panoramic image. Patent Document 2 discloses a technology for providing a traffic jam occurrence area and a cause video of the traffic jam to a vehicle in a traffic jam. This technology detects a traffic jam area based on vehicle position information provided by multiple vehicles, acquires video of the traffic area transmitted by the vehicle, and provides video data of the traffic area as traffic jam presentation information To do.

Special table 2014-502430 gazette JP 2013-20523 A

  However, as described above, in order to utilize the video, the utilization opportunities are limited, for example, it is necessary to secure specialized personnel. With the technique of Patent Document 1, it is possible to view a surveillance camera video according to a predetermined road shape, and it is possible to confirm a person's action with a real-time video. However, in order to grasp the behavior of a person who moves differently each time, it is necessary for the user to predict which road the person has moved and to specify the road on which the video is to be viewed. Further, with the technique of Patent Document 2, it is possible to browse the head video of a traffic jam from a remote position. However, the video cannot be switched to a position other than the head position, which is insufficient for use in grasping the situation of a person's action.

  The present invention has been made in view of such problems, and by automatically performing a flow line analysis utilizing a captured image and providing an image according to the analysis result, the efficiency of the image confirmation work by the observer is improved. The purpose is to make it easier.

  Therefore, the present invention is an information processing system including one or more photographing devices that photograph one or more moving objects, and an information processing device, wherein the information processing device is obtained from a video obtained by the photographing device. A flow line extracting means for extracting a flow line of one or more moving objects, and at least one of a presentation area and a presentation period as a presentation range in a time-space photographed by the photographing apparatus based on the flow line And a display means for displaying the presentation video. The range specification means includes: a video identification means for specifying a presentation video to be presented to a user based on the presentation range; and a display means for displaying the presentation video.

  According to the present invention, the flow line analysis utilizing the captured video is automatically performed, and the video according to the analysis result is provided, so that the efficiency of the video checking operation by the observer can be improved.

1 is an overall view of a monitoring system. It is a hardware block diagram of each apparatus of a monitoring system. It is a software block diagram of each apparatus of the monitoring system. It is explanatory drawing of the utilization form of the monitoring system which concerns on this embodiment. It is a figure which shows the relationship between a person's flow and a presentation image | video. It is a figure which shows the data stored in flow line DB. It is a figure which shows the data of a presentation range. It is a figure which shows an example of a reproduction list. It is a figure which shows an example of setting information. It is a flowchart which shows a flow line extraction process. It is a flowchart which shows a presentation video transmission process. It is a flowchart which shows a presentation range specific process. It is a flowchart which shows a reproduction | regeneration list production | generation process. It is a flowchart which shows a reproduction | regeneration display process. It is a figure which shows an example of the setting information which concerns on 2nd Embodiment. It is explanatory drawing of the utilization form of the monitoring system which concerns on 2nd Embodiment. It is a figure which shows the relationship between a residence time and a presentation image | video. It is a flowchart which shows the presentation range specific process which concerns on 2nd Embodiment. It is a flowchart which shows the play list production | generation process which concerns on 2nd Embodiment. It is a figure which shows an example of the setting information which concerns on 3rd Embodiment. It is explanatory drawing of the utilization form of the monitoring system which concerns on 3rd Embodiment. It is a figure which shows the relationship between an individual flow line and a presentation image | video. It is a flowchart which shows the presentation video transmission process which concerns on 3rd Embodiment. It is a flowchart which shows the play list production | generation process which concerns on 3rd Embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(First embodiment)
FIG. 1 is an overall view of a monitoring system as an example of an information processing system according to the first embodiment. The monitoring system includes a monitoring camera 100, a monitoring server device 110, and a mobile terminal (portable terminal) 120. The monitoring camera 100, the monitoring server device 110, and the mobile terminal 120 can communicate with each other via a network such as the Internet or an intranet. Note that the number of monitoring cameras 100 and mobile terminals 120 is not particularly limited as long as it is one or more.

  The monitoring cameras 100 are respectively installed at predetermined positions in the monitoring space to be monitored. The surveillance camera 100 can record an image of the installed space. Videos taken by a plurality of monitoring cameras 100 are accumulated in the monitoring server device 110. The monitoring server device 110 analyzes and edits the video received from the monitoring camera 100 and provides it to the mobile terminal 120 as appropriate. The mobile terminal 120 is possessed by the user and is a wearable display. As another example, the mobile terminal 120 may be a smartphone, a tablet terminal, or the like.

  FIG. 2 is a hardware configuration diagram of the monitoring camera 100, the monitoring server device 110, and the mobile terminal 120. The CPU 201 controls each device (the monitoring server device 110, the monitoring camera 100, and the mobile terminal 120) by executing a control program stored in the ROM 202. The RAM 203 temporarily stores various data from each component. Further, the program is expanded to make each device executable. The display device 204 displays various information. The imaging device 205 is configured by an optical device, an imaging device, and the like, and performs imaging of a subject in response to an instruction from the CPU 201. The storage device 206 stores various data. The communication device 207 communicates with other devices. The input device 208 accepts various operations by the user. Note that the monitoring server device 110 does not include the imaging device 205.

  FIG. 3 is a software configuration diagram of the monitoring camera 100, the monitoring server device 110, and the mobile terminal 120. The monitoring camera 100 includes an imaging processing unit 300 and a communication processing unit 301. The imaging processing unit 300 captures an image near the installation position. The communication processing unit 301 transmits the video obtained by shooting to the monitoring server device 110.

  The monitoring server device 110 includes a video management unit 310, a video analysis unit 311, a video playback control unit 312, a communication processing unit 313, a video DB 314, a flow line DB 315, and a playback list DB 316. The video management unit 310 records the video received via the communication processing unit 313 in the video DB 314.

  The video analysis unit 311 analyzes the video stored in the video DB 314, detects a moving object, extracts a flow line, and stores information related to the flow line in the flow line DB 315. In the present embodiment, a person will be described as an example of a moving object. The video playback control unit 312 refers to the information stored in the flow line DB 315, specifies the video to be presented to the user of the mobile terminal 120, and further determines the playback order of the specified video. Then, the video playback control unit 312 generates a presentation video and a playback list that defines the playback order, and records this in the playback list DB 316. The reproduction list is transmitted to the mobile terminal 120 by the communication processing unit 313. As a storage device for storing the video DB 314, the flow line DB 315, and the reproduction list DB 316, a flash memory, an HDD, or the like can be used.

  The mobile terminal 120 includes a display processing unit 320, a reception unit 321, an imaging processing unit 322, a position information acquisition unit 323, and a communication processing unit 324. The accepting unit 321 accepts content input by the user to the input device 208. For example, the reception unit 321 receives a video acquisition request. In this case, the communication processing unit 324 transmits a video acquisition request to the monitoring server device 110. The display processing unit 320 displays the video captured by the imaging processing unit 322 and the video received from the monitoring server device 110 on the display device 204 of the own device. The position information acquisition unit 323 acquires position information indicating the position of the own device. The position information is transmitted to the monitoring server device 110 by the communication processing unit 324. The mobile terminal 120 includes a GPS receiver, and the position information acquisition unit 323 acquires position information from the GPS receiver.

  The functions of each device described in FIG. 3 and the processing of each function to be described later are realized by the CPU 201 of each device reading a program stored in the ROM 202 or the like and executing this program. As another example, each function and each process of each device may be realized by hardware such as an electronic circuit.

  FIG. 4 is an explanatory diagram of a usage pattern of the monitoring system according to the present embodiment. FIG. 4A shows a state where the user 400 who is located at the point A and has the mobile terminal 120 is watching the human flow 410 in the monitoring space. The state of the human flow 410 is captured by a plurality of surveillance cameras 100. The monitoring server device 110 enables a wide range of monitoring by collectively managing videos taken by a plurality of monitoring cameras 100 installed at a plurality of different positions in the monitoring space. Here, it is assumed that the user 400 who owns the mobile terminal 120 is a staff of a facility including a monitoring space, and the human flow 410 is a human flow generated in the facility.

  FIG. 4B shows an example of a display screen 420 displayed on the display device 204 of the mobile terminal 120 when the user 400 views the flow of people 410 via the mobile terminal 120 at the point A. It is. At this time, the imaging processing unit 322 of the mobile terminal 120 captures the human flow 410. On the display screen 420, a captured image 421 of the human flow 410 captured by the imaging device 205 of the mobile terminal 120 is displayed. Furthermore, the presentation video 422 received from the monitoring server device 110 is superimposed on the captured video 421. The presentation video 422 is generated as a video to be presented to the user of the mobile terminal 120 based on the captured video obtained by the monitoring server 100 by the monitoring server device 110. In the present embodiment, the presentation video 422 is a monitoring video used for monitoring. The presentation video 422 is a video taken by the monitoring camera 100, and the presentation video 422 shows the state of the head of the human flow 210 that cannot be seen by the user located at the point A. Furthermore, information indicating the distance and direction from the current position of the user 400 to the shooting position may be displayed on the display screen 420 together with the presentation video 422. Although the user 400 cannot directly see the head of the human flow 410 from the point A where the user 400 is located, the user 400 can know the cause of the human flow 410 from the presentation video 422. it can.

  On the photographed video 421, an auxiliary display 423 and a switching display 424 are further superimposed as virtual objects. The auxiliary display 423 is an object indicating the direction and flow rate of the human flow 410. By displaying the auxiliary display 423 and the presentation video 422 together, the user 400 can more intuitively grasp the shooting position of the presentation video 422. The switching display 424 is an object indicating that the presentation video 422 can be switched to another video. The user 400 can switch the presentation video 422 to another presentation video by inputting a switching instruction to the input device 208. Note that a flick operation, a touch operation, a button operation, a gesture operation, or the like can be used to input the switching instruction.

  FIG. 4C is a diagram illustrating an example of the display screen 430 after switching according to the switching instruction. On the display screen 430, a presentation video 432 is superimposed and displayed instead of the presentation video 422. The presentation video 432 is a video of a position moved backward along the flow of people 410 from the shooting position of the presentation video 422. The presentation video 432 and the presentation video 422 are taken by different monitoring cameras 100 and are selected from a group of videos including all the videos shot by all the monitoring cameras 100. The presentation video 422 and the presentation video 432 are selected from the video group along the order of passage of the route so as to cover the passage route of the human flow 410. As described above, the user 400 can confirm the video of the space actually existing in front of the user and the video reproduced along the human flow 410, which is captured by the mobile device 120. it can. Therefore, it is possible to efficiently grasp the cause of the human flow 410 and the overall state. That is, the monitoring system according to the present embodiment can detect a person as a moving object, generate a presentation video based on a flow line generated by the movement of the person, and present it to the user.

  FIG. 5 is a diagram showing the relationship between the human flow 410 and the presented video. The monitoring space 500 is represented by a three-dimensional space-time in which shooting time is added to the shooting position (x, y). The surveillance camera 100 captures this time and space. Note that the video can be uniquely determined by specifying the shooting position and the shooting time. A plurality of individual flow lines 510a, 510b, 510c, 510d, 510e,... Exist in the monitoring space 500 shown in FIG. Here, the individual flow line is a curve indicating the movement trajectory of each person. The mobile terminal 120 obtains a collective flow line 520 by analyzing a plurality of individual flow lines and extracting features of the human flow 410. Then, the mobile terminal 120 identifies the presentation ranges 530a, 530b, 530c... Based on the collective flow line 520, and transmits the video in the presentation range to the mobile terminal 120 as a presentation video.

  Here, the presentation range is information for specifying a video to be used as a presentation video, and is information for specifying a presentation area and a presentation period. Here, the presentation area is a shooting range shown in the presentation video to be presented, and the presentation period is a period on the time axis of the shooting time of the presentation video to be presented. In the present embodiment, a case where the presentation period is determined to be a certain period from the latest time will be described. That is, the monitoring server apparatus 110 according to the present embodiment specifies the presentation area based on the collective flow line, and specifies the presentation range based on the specified presentation area and a predetermined presentation period.

  FIG. 6 is a diagram illustrating data stored in the flow line DB 315. FIG. 6A is a diagram illustrating individual flow line data that is a movement trajectory of the moving object (person) obtained from the video stored in the video DB 314. Each record of the individual flow line includes an individual flow line ID 601, a time 602, a position 603, a direction 604, a speed 605, and an acquisition source 606. The individual flow line ID 601 is information for specifying an individual flow line by the same person. A set of observation points (records) having the same individual flow line ID 601 is treated as one individual flow line. Time 602 is information indicating the observed time. The position 603, the direction 604, and the speed 605 are information indicating the position of the flow line, the moving direction, and the moving speed in the monitoring space at time 602, respectively. The acquisition source 606 is information for specifying the monitoring camera 100 that captured each observation point.

  FIG. 6B is a diagram illustrating collective flow line data generated from individual flow line data. Here, the collective flow line is obtained by extracting the characteristics of the movement pattern by removing the difference in observation time from a plurality of individual flow lines. The collective flow line is generated by the video analysis unit 311 based on the video data. Individual flow lines indicating a common movement pattern are collected into one collective flow line. Each record of the collective flow line includes a collective flow line ID 611, a sequence order 612, a position 613, a direction 614, a speed 615, and an acquisition source 616. The collective flow line ID 611 is information for specifying the collective flow line. A set of observation points (records) having the same collective flow line ID 307 is treated as one collective flow line. The sequence order 612 indicates the observation order of the observation points on the collective flow line. The position 613, the direction 614, and the speed 615 are information indicating the position, moving direction, and moving speed of the collective flow line in the monitoring space, respectively. The acquisition source 616 identifies the monitoring camera 100 that captured each observation point.

  FIG. 6C is a diagram showing data on the correspondence between individual flow lines and collective flow lines. Correspondence relationship data indicates correspondence relationships indicating which individual flow lines are extracted from which individual flow lines. Each record indicates that the collective flow line indicated by the collective flow line ID 621 is extracted from the individual flow line group indicated by the individual flow line ID 622. Although one collective flow line is extracted from one or more individual flow lines, the degree of aggregation is controllable, and one collective flow line may be extracted from one individual flow line. One individual flow line may be used for extracting a plurality of collective flow lines.

  FIG. 7 is a diagram showing the data of the presentation range. The data of the presentation range is created by the video reproduction control unit 312. The data of the presentation range includes a plurality of records indicating the presentation range, and each record includes a time 701, a position 702, and an acquisition source 703. The time 701 and the position 702 are information for determining a time space (presentation period and presentation area) that is a presentation range. The acquisition source 703 indicates the acquisition source (monitoring camera) of the presentation video, and the collection flow line acquisition source 616 is taken over. Note that, when the latest time is set as the presentation range as in the present embodiment, the presentation period is determined in advance, and therefore recording of the time at the time 701 may not be performed.

  FIG. 8 is a diagram showing an example of a playlist stored in the playlist DB 316. As shown in FIG. A playlist is created based on a plurality of records included in the presentation range data. The reproduction list includes a plurality of records. Each record includes a reproduction order 801 of the presented video, a monitoring camera ID 802, a video type 803, a start time 804, an end time 805, a position 806, and a playing flag 807. Is included. The playback order 801 defines the playback order of each presented video when a plurality of presented videos included in the playback list are played back continuously, and the videos can be played back continuously in this order. The surveillance camera ID 802 identifies the surveillance camera 100, and an image captured by the surveillance camera 100 shown here is selected as a presentation image.

  The video type 803 indicates whether the presented video is being shot or recorded. When the video type 803 is being shot, the mobile terminal 120 performs stream playback of the video being shot by the monitoring camera 100, and in the case of recording, the recorded video is played back. A start time 804 and an end time 805 are video start and end times. A position 806 indicates the presence position of the moving object in the video. The in-reproduction flag 807 is used for specifying a video to be reproduced.

  FIG. 9 is a diagram illustrating an example of setting information used by the monitoring system according to the present embodiment. The setting information shown in FIG. 9 is assumed to be recorded in the storage device 206 of the monitoring server device 110. Each item (element) included in the setting information can be individually set according to the usage form. Each process in the flowchart to be described later is executed according to this set value. In “presentation area setting”, “passage range of collective flow line” is set, and in the presentation range data shown in FIG. 6, all observation points (position and time) within the collective flow line are presented. It is recorded as a spatio-temporal. In the present embodiment, “presentation time setting” is set to “latest time”. This indicates that the presentation period is a certain period including the latest time. Corresponding to this, no value is recorded at the time 701 of the data in the presentation range, the start time 804 of the reproduction list, and the end time 805. And the video of the newest time of the presentation area specified from the collective flow line is reproduced as a presentation video as a security video.

  FIG. 10 is a flowchart showing a flow line extraction process by the monitoring server device 110. In the flow line extraction process, the monitoring server device 110 analyzes the video captured by the monitoring camera 100 to extract individual flow lines, and extracts a collective flow line after extracting a predetermined amount of individual flow lines. First, in step S <b> 1001, the video analysis unit 311 acquires one or more video frames in the video DB 314. Next, in step S1002, the video analysis unit 311 detects a moving object from within the acquired video. In step S1003, the video analysis unit 311 records individual flow line data including the position of the moving object. Note that the video analysis unit 311 determines whether a moving object obtained from a different video is the same moving object or a different moving object from the correspondence between the detection position and the detection time. Then, the video analysis unit 311 assigns an individual flow line ID corresponding to each moving object. That is, the data of the individual flow line is updated by this process. A known technique such as a background subtraction method can be used for detecting the moving object.

  In step S1004, the video analysis unit 311 determines whether the number of moving objects detected in step S1002, that is, the number of individual flow lines has reached a predetermined number. If the predetermined number has been reached (Yes in step S1004), the video analysis unit 311 advances the process to step S1005. If the predetermined number has not been reached (No in step S1004), the video analysis unit 311 advances the process to step S1001, and repeats the process from step S1001 again.

  In step S1005, the video analysis unit 311 extracts a collective flow line from an individual flow line group including a plurality of individual flow lines obtained by the processes in steps S1001 to S1004 (flow line extraction process). Then, the video analysis unit 311 updates the collective flow line data illustrated in FIG. 6B and the correspondence data illustrated in FIG. For extracting the collective flow line, a known technique such as a clustering technique for collecting individual flow lines having similar trajectories can be used. Next, in step S1006, the video analysis unit 311 generates an auxiliary image indicating the shape and flow rate of the collective flow line. Furthermore, the video analysis unit 311 initializes the count of the number of individual flow lines, and ends the flow line extraction process.

  It is assumed that the flow line extraction process is repeatedly executed while the monitoring camera 100 is photographing. In this embodiment, the collective flow line is extracted with reference to all the individual flow lines after the predetermined number of individual flow lines reaches a predetermined number. However, the timing for extracting the collective flow line is limited to the embodiment. It is not something. As another example, the video analysis unit 311 may update the collective flow line every time a new observation point is obtained. In this case, the latest individual flow line can always be reflected in the collective flow line.

  FIG. 11 is a flowchart showing the presentation video transmission processing by the monitoring server device 110. In the presentation video transmission process, the monitoring server device 110 specifies a video to be used as the presentation video and transmits this to the mobile terminal 120. First, in step S <b> 1101, the communication processing unit 313 receives a flow line acquisition request from the mobile terminal 120. Here, the flow line acquisition request is information for requesting acquisition of a flow line specified by the monitoring server device 110 as an object to be presented to the display device 204 of the mobile terminal 120. It is assumed that the flow line acquisition request includes the position information of the mobile terminal 120 acquired by the position information acquisition unit 323. Note that the flow line acquisition request is transmitted from the mobile terminal 120 at the timing when the user of the mobile terminal 120 inputs the flow line acquisition request using the input device 208. As another example, the mobile terminal 120 may transmit a flow line acquisition request at a timing when the imaging device 205 of the mobile device 120 detects a flow of a person of a certain amount or more.

  Next, in step S1102, the video analysis unit 311 selects a collective flow line to be presented from the collective flow line group extracted in the flow line extraction process. Specifically, the video analysis unit 311 compares the position of the extracted collective flow line with the position information of the mobile terminal 120 included in the flow line acquisition request, and determines the collective flow line near the mobile terminal 120. Select as the set flow line to be presented. Specifically, the video analysis unit 311 selects a collective video that minimizes the distance from the position of the mobile terminal 120 to the collective flow line. The video analysis unit 311 further includes a collective motion in which at least a part of the collective flow line to be presented is included in a range included in the photographing region of the mobile terminal 120 based on the photographing region by the photographing device 205 of the mobile terminal 120. You may limit to a line. At this time, it is assumed that the imaging region of the mobile terminal 120 is transmitted by being included in the flow line acquisition request.

  Next, in step S1103, the video analysis unit 311 sets a presentation range based on the collective flow line to be presented specified in step S1103. As described above, since the presentation period is predetermined in the presentation range, in step S1103, the video analysis unit 311 first identifies the presentation range based on the collective flow line of the presentation target selected in step S1102. (Presentation range specifying process). Here, the presentation range is a space-time to be presented, that is, a presentation area to be presented and a presentation period. Specifically, the video analysis unit 311 specifies a space in a range including the passage route of the collective flow line as a presentation area. Then, the video analysis unit 311 sets a time space determined based on a predetermined presentation period and a presentation area as a presentation target.

  Next, in step S1104, the video analysis unit 311 identifies a video to be used as a video to be presented to the user based on the presentation range set in step S1103 (presentation video identification processing). Then, the video analysis unit 311 generates a playback list for continuously playing the specified video. Note that the processing in step S1103 and step S1104 will be described in detail later. In step S1105, the video analysis unit 311 sets an initial video from the reproduction list generated in step S1104. The video analysis unit 311 according to the present embodiment is set as the initial video that is initially displayed at the top of the reproduction list, which is the top position of the collective flow line, according to the setting information shown in FIG. That is, the top video of the playback list is set as the playback target video.

  In step S <b> 1106, the communication processing unit 313 transmits the collection target flow line data to be presented, the virtual object such as the auxiliary display 423, and the play list to the mobile terminal 120. In step S <b> 1107, the communication processing unit 313 waits until a video acquisition request is received from the mobile terminal 120. If the video analysis unit 311 receives a video acquisition request (Yes in step S1107), the video analysis unit 311 advances the process to step S1108. In step S1108, the video analysis unit 311 transmits the video related to the video acquisition request, that is, the video shown in the playlist as the presentation video, to the mobile terminal 120 in order according to the playback order 801 via the communication processing unit 313. To do. In step S <b> 1109, the video analysis unit 311 determines whether an end request has been received from the mobile terminal 120. If the video analysis unit 311 receives an end request (Yes in step S1109), the video analysis unit 311 ends the video transmission process. If the video analysis unit 311 does not receive the termination request (No in step S1109), the video analysis unit 311 advances the process to step S1107.

  FIG. 12 is a flowchart showing detailed processing in the presentation range specifying processing (step S1103). In step S <b> 1201, the video analysis unit 311 detects the head position of the set flow line to be presented. Next, in step S1202, the video analysis unit 311 specifies one observation point position in the collective flow line in order of the collective flow line sequence in order from the head position of the collective flow line detected in step S1202. . Next, in step S1203, the video analysis unit 311 identifies a presentation area based on the position of the observation point identified in step S1202. Here, the presentation area is an area in space-time taken by the monitoring camera 100 and is determined by the position. Then, the video analysis unit 311 identifies a presentation range determined by the identified presentation area and the presentation period. Then, the video analysis unit 311 adds the presentation range record to the presentation range data illustrated in FIG. 6.

  Here, the presentation period is a period in time and space, and is a predetermined period determined by the shooting time. In the present embodiment, as shown in FIG. 8, “latest time” is set in the presentation range time setting. This means that the presentation period is a fixed time from the latest time. Here, the latest time is the time at the time of display processing when the video to be presented is displayed on the mobile terminal 120. Thus, in this embodiment, the presentation period is predetermined. Therefore, the video analysis unit 311 does not need to perform processing for specifying the presentation period from the flow line. Note that the processing in step S1203 identifies a presentation area that is a presentation range in the space-time photographed by the monitoring camera 100 based on the flow line, and further, based on the presentation area and a predetermined presentation period. This is an example of a range specifying process for specifying the presentation range.

  Next, in step S1204, the video analysis unit 311 determines whether or not the process of creating a presentation range record corresponding to all observation points included in the set flow line to be presented has ended. If the process has been completed (Yes in step S1204), the video analysis unit 311 ends the presentation range specifying process. If the process has not ended (No in step S1204), the video analysis unit 311 advances the process to step S1102. In this case, in step S1202, the video analysis unit 311 identifies one unprocessed observation point position and repeats the process.

  FIG. 13 is a flowchart showing detailed processing in the playlist generation processing (step S1104). First, in step 1301, the video analysis unit 311 reads one record of presentation range data, that is, one presentation range. Next, in step S1302, the video analysis unit 311 identifies the monitoring camera 100 corresponding to the presentation range. Information of the acquisition source 703 (FIG. 7) can be used for specifying the monitoring camera 100. Since there are overlaps in the space taken by the monitoring camera 100, two or more monitoring cameras 100 may be designated as the acquisition source 703. In this case, the video analysis unit 311 selects the monitoring camera 100 arranged at a position closest to the presentation area that defines the presentation range. Thereby, a video with higher visibility can be obtained.

  Next, in step S1303, the video analysis unit 311 determines whether or not the specified monitoring camera 100 is the same as the monitoring camera 100 specified from the data record of the previous presentation range. If they are the same (Yes in step S1303), the video analysis unit 311 advances the process to step S1306. If they are not the same (No in step S1303), the video analysis unit 311 advances the process to step S1304. In step S1304, the video analysis unit 311 creates a new record in the playback list, and sets the playback order 801, the monitoring camera ID 802, the video type 803, and the like. As described above, when “latest time” is set as the presentation period, “in shooting” is set in the video type 803. In this case, it is not necessary to record the start time 804 and the end time 805. In step S1305, the video analysis unit 311 adds the created record to the reproduction list, and advances the process to step S1307. On the other hand, if the surveillance cameras 100 are the same, the surveillance camera 100 identified in step S1302 has been added to the reproduction list. In step S1306, the video analysis unit 311 adds position information to the position 806 of the record created and added immediately before, and then advances the processing to step S1307.

  In step S1307, the video analysis unit 311 determines whether or not the processing in steps S1301 to S1306 has been completed for all the records (presentation ranges) indicated in the presentation range data. When the process is completed (Yes in step S1307), the video analysis unit 311 ends the playlist generation process. If the process has not been completed (No in step S1307), the video analysis unit 311 advances the process to step S1301. In this case, the video analysis unit 311 reads an unprocessed shooting range in S1301, and repeats the processing after S1302.

  In the present embodiment, the video analysis unit 311 determines the playback order of the video specified as the presentation video according to the movement of the position from the beginning to the end of the collective flow line. However, the playback order is not limited to the embodiment. As another example, the video analysis unit 311 may determine the playback order according to the movement of the position from the tail to the head. Thereby, when the human flow 410 is a long queue, it is possible to quickly grasp where to line up.

  FIG. 14 is a flowchart showing the playback display process by the mobile terminal 120. In step S1401, the communication processing unit 324 of the mobile terminal 120 transmits a flow line acquisition request to the monitoring server device 110. In step S <b> 1402, the communication processing unit 324 receives from the monitoring server device 110 data on the collective flow line to be presented, a virtual object, and a play list. Next, in step S1403, the display processing unit 320 determines whether or not the position of the collective flow line received from the monitoring server device 110 is included in the imaging region of the imaging device 205 of the mobile terminal 120. If the position of the collective flow line is within the imaging region (Yes in step S1403), the display processing unit 320 advances the process to step S1404. If the position of the collective flow line is not within the imaging region (No in step S1403), the display processing unit 320 advances the process to step S1412. In step S1412, the communication processing unit 324 transmits a reproduction end request to the monitoring server device 110. As described above, the mobile terminal 120 ends the reproduction display process. This is because when the position of the flow line is not within the shooting area, there is no flow line within the user's visual recognition area. Note that the process of step S1403 is an example of a determination process for determining whether to reproduce and display the presented video based on the shooting area of the mobile terminal 120.

  In step S1404, the display processing unit 320 superimposes and displays the received virtual object on the captured video imaged by the imaging device 205 of the own device. Here, the virtual object is a spatial object composed of a list of position information. The display processing unit 320 converts the position coordinates of the virtual object into the display coordinates of the display device 204 when the mobile terminal 120 matches the shooting area of the shooting device 205. Next, in step S1405, the display processing unit 320 transmits a video acquisition request specifying the video specified as the presentation video in the reproduction list to the monitoring server device 110. Next, in step S1406, the communication processing unit 324 receives the video related to the video acquisition request from the monitoring server device 110 (reception processing). Then, the display processing unit 320 starts reproducing and displaying the received video (display processing). At this time, the display processing unit 320 may refer to the position 806 of the reproduction list and enlarge and display a specific position in the monitoring video. When the video type of the video related to the video acquisition request is “capturing”, the communication processing unit 324 receives the real-time video captured at the time of the display process, and the display processing unit 320 performs real-time video capture. To display.

  In step S1407, the display processing unit 320 determines whether a video change instruction has been received from the user. If the display processing unit 320 receives a video change instruction (Yes in step S1407), the display processing unit 320 advances the process to step S1408. If the display processing unit 320 has not received the video change instruction (No in step S1407), the display processing unit 320 advances the process to step S1709. In step S1408, the display processing unit 320 moves through the play list by the amount indicated by the video change instruction, changes the video to be played, and advances the process to step S1405.

  In step S1409, the display processing unit 320 determines whether or not the playback of the video to be played has ended at the time of the processing in step S1409. When the reproduction is finished (Yes in step S1409), the display processing unit 320 advances the process to step S1410. If the reproduction has not ended (No in step S1409), the display processing unit 320 advances the process to step S1411. In step S1410, the display processing unit 320 advances the reproduction list by one, sets the next video as a reproduction target, and advances the processing to step S1405. In step S1411, the display processing unit 320 determines whether or not the reception unit 321 has received an end instruction and whether or not playback of all the videos in the playback list has ended. If the display processing unit 320 has not received an end instruction and playback has not ended (No in step S1411), the display processing unit 320 advances the process to step S1407. The display processing unit 320 advances the process to step S1412 in the case where either the end instruction is accepted or the case where the reproduction of all the videos in the reproduction list is completed (Yes in step S1411).

  As described above, the monitoring system according to the present embodiment can continuously reproduce the video corresponding to the collective flow line. Therefore, the user who possesses the user terminal can efficiently grasp the place that cannot be directly visually recognized from the position of the user by checking the presented video. Furthermore, in the monitoring system according to the present embodiment, when the shooting range of the imaging device in the user terminal includes a collective flow line, the presentation video corresponding to the collective flow line is displayed on the user terminal in the user terminal. It is superimposed on the captured video. Therefore, the user can easily grasp the correspondence between the movement of the moving object that can be visually recognized from his / her position and the presented video.

  As a first modification of the monitoring system according to the first embodiment, the transmission timing of the flow line data, the virtual object, and the playlist to be presented by the monitoring server apparatus 110 is not limited to the embodiment. As another example, the monitoring server apparatus 110 may first transmit only the aggregate flow line data to be presented. When the mobile terminal 120 determines that the collective flow line is located within the shooting area, the mobile terminal 120 transmits a virtual object and a play list of the collective flow line located within the shooting range in accordance with a request from the mobile terminal 120. May be. Thereby, the communication amount of the mobile terminal 120 and the monitoring server apparatus 110 can be reduced.

  As a second modification, the setting information of the monitoring server device 110 is not limited to the embodiment. In addition, in step S1101, the monitoring server apparatus 110 may receive setting information from the mobile terminal 120 together with the flow line acquisition request. And the monitoring server apparatus 110 is good also as performing the process according to the setting content of the received setting information in step S1102-step S1105.

  As a third modification, the process of specifying the presentation range by the monitoring server device 110 is not limited to the embodiment. As another example, the monitoring server device 110 may specify the presentation range based on the stay occurrence location, the person density, and the movement direction on the collective flow line. The occurrence of stagnation at a location where a human current is generated, the excess of the person density, and the change in the moving direction may cause a fall or the like. Therefore, by making it possible to continuously play back only the video at these locations, it is possible to take measures such as organizing human flow before a fall occurs.

  As a fourth modification, the preset presentation period in the monitoring server device 110 is not limited to “latest time”.

  As a fifth modification, the monitoring server device 110 may specify at least one of the presentation area and the presentation period based on the flow line. That is, as another example, the monitoring server apparatus 110 may specify only the presentation period based on the flow line. In this case, the presentation area is determined in advance. As another example, the monitoring server device 110 may specify both the presentation area and the presentation period based on the flow line.

(Second Embodiment)
Next, a monitoring system according to the second embodiment will be described. Note that here, the monitoring system according to the second embodiment will be described with respect to parts different from the monitoring system according to the first embodiment. The monitoring system according to the first embodiment sets the spatial position of the presentation target based on the collective flow line, and continuously reproduces videos at a plurality of different positions on the flow line on the display device 204 of the mobile terminal 120. Is. On the other hand, the monitoring system according to the second embodiment continuously reproduces video at different times at the same position on the flow line on the display device 204 of the mobile terminal 120.

  FIG. 15 is a diagram illustrating an example of setting information set in the monitoring server apparatus 110 according to the second embodiment. In the presentation area and the presentation period, “stagnation occurrence location” and “stagnation occurrence time” are set, respectively. This indicates that the stay occurrence point is specified as the presentation area, and the period during which the stay is occurring is specified as the presentation period. That is, the monitoring server apparatus 110 according to the present embodiment identifies the stay occurrence location on the collective flow line as the presentation area, and identifies the period during which the stay is occurring as the presentation period. Then, the monitoring server apparatus 110 records a record of the presentation range corresponding to the stay occurrence position near the position where the mobile terminal 120 exists at the processing time among the stay occurrence locations in the presentation range illustrated in FIG. In addition, in the start time 804 and the end time 805, the start time and end time of occurrence of stay are recorded. In other words, the start time 804 and the end time 805 store time information indicating how long the stay has continued.

  The video analysis unit 311 of the monitoring server device 110 according to the present embodiment extracts a collective flow line by aggregating not only the trajectory but individual flow lines with similar acceleration changes. Further, the video analysis unit 311 detects staying (staying detection processing) based on the collective flow line, and specifies a staying occurrence location. Specifically, the video analysis unit 311 identifies a place where the moving speed of the collective flow line is equal to or less than a threshold value as a stay occurrence place. As another example, the video analysis unit 311 may identify a location where the time during which the speed of the collective flow line is equal to or less than a threshold value is equal to or longer than a certain time as a residence occurrence location. As another example, the video analysis unit 311 may specify a stay occurrence location based on a speed change of each of a plurality of individual flow lines constituting the collective flow line.

  FIG. 16 is an explanatory diagram of a usage pattern of the monitoring system according to the second embodiment. FIG. 16A shows a state where a user 1600 who owns the mobile terminal 120 is looking at a space 1610 in a facility as a monitoring space. Here, the user 1600 is a facility staff or the like. FIG. 16B is a diagram illustrating an example of a display screen 1620 displayed on the display device 204 of the mobile terminal 120 when the user 1600 views the space 1610 in the facility via the mobile terminal 120. On the display screen 1620, a captured video 1621 being captured by the capture processing unit 322 of the mobile terminal 120 is displayed. Further, a presentation video 1622 received from the monitoring server device 110 is superimposed on the captured video 1621. On the photographed video 1621, an auxiliary display 1623, a staying position display 1624, and a switching display 1625 are further displayed as virtual objects. The auxiliary display 1623 and the switching display 1625 are the same as the auxiliary display 423 and the switching display 524 described with reference to FIG. The stay position display 1624 is an object indicating a stay occurrence position (stay occurrence position) of an individual flow line that exists in the monitoring space.

  The staying position display 1624 indicates that a person has stayed in the monitoring space in the past, and it is assumed that there is some problem in the layout of the facility. However, only the information that the stagnation has occurred cannot accurately grasp what happened on the spot, and cannot determine how to improve it. On the other hand, in this embodiment, the mobile terminal 120 can display the presentation video 1622 in accordance with the position of the staying position display 1624. Here, the presentation video 1622 is taken by the monitoring camera 100, and is a video taken at the time when the stagnation has occurred in the monitoring space (stagnation occurrence time). Shooting time information can also be added to the presentation video 1622.

  FIG. 16C is a diagram showing an example of the display screen 1630 after being switched according to the switching instruction from the user. A presentation video 1632 on the display screen 1630 is a video shot at a shooting time that is earlier than the shooting time of the presentation video 1622. As described above, the mobile terminal 120 can continuously reproduce the video of the stay occurrence time while going back in time. Therefore, the user 1600 can efficiently grasp the cause of the stay from the display content of the mobile terminal 120.

  FIG. 17 is a diagram illustrating the relationship between the staying time and the presented video. Similar to FIG. 5, the monitoring space 1700 is expressed in a three-dimensional space in which the photographing time is added to the photographing position (x, y). By specifying the shooting position and the shooting time, a video can be uniquely determined. In the present embodiment, the presentation area is fixed to an area including the stay occurrence position among areas on the collective flow line 1720 formed by the individual flow lines 1710a, 1710b, and 1710c. And the monitoring server apparatus 110 specifies the presentation period 1730 according to the residence occurrence time of the individual flow line in the collective flow line.

  The presentation video transmission process according to the second embodiment is substantially the same as the presentation video transmission process by the monitoring server apparatus 110 according to the first embodiment described with reference to FIG. 11, but in steps S1103 and S1104. Processing is different. FIG. 18 is a flowchart showing a detailed process in the presentation range specifying process by the monitoring server apparatus 110 according to the second embodiment. In step S <b> 1801, the video analysis unit 311 identifies, as a presentation area, an area including a stay occurrence position near the current position of the mobile terminal 120 among the stay occurrence positions of the collection flow line to be presented.

  Next, in step S1802, the video analysis unit 311 acquires one individual flow line included in the collective flow line to be presented selected in step S1102. Next, in step S1803, the video analysis unit 311 specifies a period during which the acquired individual flow line stays in the presentation area specified in step S1801 as a presentation period. In the present embodiment, the video analysis unit 311 determines that stagnation has occurred when a moving speed equal to or lower than a predetermined value continues for a certain period of time. The video analysis unit 311 may assign an index to the individual flow line speed 305 illustrated in FIG. 6 in association with the position where the stay is detected. As a result, the video analysis unit 311 can extract a candidate for a stay occurrence point at high speed. Then, the video analysis unit 311 specifies a presentation range determined by the presentation area and the presentation range specified from the collective flow line, and adds a record of the specified presentation range to the data of the presentation range shown in FIG.

  In step S1804, the video analysis unit 311 determines whether or not the processing in steps S1802 and S1803 has been completed for all individual flow lines included in the collective flow line. When the process is completed (Yes in step S1804), the video analysis unit 311 advances the process to step S1805. If the process has not ended (No in step S1804), the video analysis unit 311 advances the process to step S1802. In this case, the video analysis unit 311 acquires one unprocessed individual flow line in S1802, and repeats the processing from S1803 onward.

  In step S1805, the video analysis unit 311 sorts the presentation range based on the presentation time. Specifically, the video analysis unit 311 sorts the presentation ranges in order from the presentation time to the latest time. This process is performed in order to create a playlist that is in chronological order in the subsequent playlist generation process (step S1104). That is, in the playlist generation process, the video analysis unit 311 determines the playback order 801 in accordance with the order sorted in step S1805.

  FIG. 19 is a flowchart showing detailed processing in the playlist generation processing (step S1104) according to the second embodiment. In step S1901, the video analysis unit 311 identifies the monitoring camera 100 from the presentation range data. In the present embodiment, the target area is fixed. Therefore, one surveillance camera 100 can be defined for all the presentation ranges indicated in the presentation range data. The process in step S1901 is an example of a selection process for selecting the monitoring camera 100 as the imaging unit based on the presentation area.

  Next, in step S1902, the video analysis unit 311 reads one record, that is, one presentation range in order from the top of the presentation range data. Next, in step S1903, the video analysis unit 311 determines whether or not the presentation range presentation period read in step S1902 is continuous with the presentation range presentation period read immediately before. Note that the video analysis unit 311 determines to be continuous when the difference in the boundary position of the shooting period is a predetermined value or less. If the video analysis unit 311 is not continuous (No in step S1903), the video analysis unit 311 advances the process to step S1904. If the video analysis unit 311 continues (Yes in step S1903), the video analysis unit 311 advances the process to step S1907.

  In step 1904, the video analysis unit 311 adds the created record to the reproduction list. Next, in step S1905, the video analysis unit 311 creates a new record. Next, in step S1906, the video analysis unit 311 sets the start time of the presentation period read in step S1902 as the start time of the new record. On the other hand, in step 1907, the video analysis unit 311 does not create a new record, but sets the end time of the presentation period read in step S1902 as the end time of the record in the held playlist. If the end time has already been set, the video analysis unit 311 overwrites (updates) the end time.

  In step S1908, the video analysis unit 311 determines whether or not the processing in steps S1902 to S1907 has been completed for all the records (presentation ranges) indicated in the presentation range data. When the process is completed (Yes in step S1908), the video analysis unit 311 ends the playlist generation process. If the process has not been completed (No in step S1908), the video analysis unit 311 advances the process to S1902. In this case, the video analysis unit 311 reads an unprocessed shooting range in S1902, and repeats the processing after S1903. In step S1909, the video analysis unit 311 adds the created record to the playlist, and ends the playlist generation process. The remaining configuration and processing of the monitoring system according to the second embodiment are the same as the configuration and processing of the monitoring system according to the first embodiment.

  As described above, the monitoring system according to the second embodiment can continuously reproduce only the time at which stagnation occurs on the collective flow line. Therefore, the user can efficiently grasp the facility layout problem, the customer's interest, and the like.

  Note that, as a first modification of the monitoring system according to the second embodiment, the monitoring server device 110 generates a playlist when receiving a flow line acquisition request. It is not limited to the embodiment. That is, the monitoring server apparatus 110 may generate a playlist in advance after receiving the flow line acquisition request after the collective flow line is obtained. In this case, the latest staying video cannot be included in the presentation range, but the time until video playback start can be shortened.

  As a second modification, the monitoring server device 110 may specify the density of persons at a plurality of points at different times on the collective flow line instead of staying (density specifying process). Then, the monitoring server device 110 may specify a presentation range based on an area and a period in which the density of the person has exceeded a predetermined value, and may generate a playlist based on the presentation range. In this case, the display device 204 of the mobile terminal 120 continuously reproduces the video of the area with high human density. Therefore, the user can efficiently grasp the cause of the occurrence of congestion.

  Further, as a third modification example, the monitoring server apparatus 110 replaces with specifying the presentation range corresponding to all the individual flow lines included in the collective flow line to be presented in the presentation range specifying process. Only the presentation range corresponding to the individual flow lines may be specified. For example, the video analysis unit 311 may end the specification of the presentation range when a presentation range corresponding to a predetermined number of individual flow lines is obtained. Thereby, when the number of accumulated individual flow lines is large, the start of video reproduction can be accelerated.

(Third embodiment)
Next, a monitoring system according to the third embodiment will be described. Note that here, the monitoring system according to the third embodiment will be described with respect to parts different from the monitoring system according to other embodiments. The monitoring system according to the third embodiment specifies the presentation range based on the individual flow line specified by the user. That is, the monitoring server apparatus 110 according to the present embodiment does not have to perform the process of extracting the collective flow line.

  FIG. 20 is a diagram illustrating an example of setting information set in the monitoring server device 110 according to the third embodiment. In the presentation range and the presentation period, “passage range of individual flow line” and “existence time of individual flow line” are set, respectively. This indicates that the passing range of the individual flow line is specified as the presentation area, and the period in which the individual flow line exists is specified as the presentation time. Therefore, the time and position of the observation point of the individual flow line are recorded in the data of the presentation range shown in FIG. Then, the video analysis unit 311 integrates the sections in which the individual flow lines are continuously captured by the same monitoring camera 100 from the presentation range data, and generates the reproduction list illustrated in FIG. 8.

  FIG. 21 is an explanatory diagram of a usage pattern of the monitoring system according to the third embodiment. FIG. 21A shows a scene in which a user 2100 holding the mobile terminal 120 is looking at a suspicious person 2101 in the facility. Here, the user 2100 is a facility staff or the like, and the suspicious person 2101 is a customer or the like visiting the facility. FIG. 21B is a diagram illustrating an example of a display screen 2120 displayed on the display device 204 of the mobile terminal 120 when the user 2100 views the monitoring space 2110 including the suspicious person 2101 via the mobile terminal 120. is there. On the display screen 2120, a captured image 2121 being captured by the capturing processing unit 322 of the mobile terminal 120 is displayed. Further, a presentation video 2122 received from the monitoring server device 110 is superimposed on the captured video 2121. The presented video 2122 is a video that captures past actions of the suspicious person 2101. An object for displaying the shooting time and the individual flow line of the suspicious person 2101 can be added to the presentation video 2122. On the captured video 2121, a switching display 2123 as a virtual object is further superimposed. The switching display 2123 is the same as the switching display 424 described with reference to FIG.

  FIG. 21C is a diagram showing an example of the display screen 2130 after being switched according to the switching instruction from the user. A presentation video 2132 on the display screen 2130 is a video shot at a shooting time in the past of the presentation video 2122. That is, the presentation video 2132 is a video with a different shooting location and shooting time from the presentation video 2122. As described above, the monitoring system according to the present embodiment can continuously reproduce and display videos having different positions and times as the suspicious person 2101 moves. Therefore, the user can efficiently confirm whether there has been any action such as shoplifting.

  FIG. 22 is a diagram illustrating the relationship between the individual flow lines and the presented video. Similar to FIG. 5, the monitoring space 2200 is expressed in a three-dimensional space in which the photographing time is added to the photographing position (x, y). By specifying the shooting position and the shooting time, a video can be uniquely determined. In the present embodiment, the presentation range 2210 is switched by moving the spatial position and time of the presentation target according to the individual flow line.

  Next, processing by the monitoring server device 110 according to the third embodiment will be described. In the third embodiment, the monitoring server apparatus 110 extracts individual flow lines in the flow line extraction process, but does not extract collective flow lines. That is, the monitoring server apparatus 110 performs the processes of steps S1001 to S1003 in the flow line extraction process illustrated in FIG. On the other hand, the monitoring server apparatus 110 does not have to perform the processing of steps S1004 to S1006.

  FIG. 23 is a flowchart showing a presentation range specifying process by the monitoring server apparatus 110 according to the third embodiment. In step S <b> 2301, the communication processing unit 313 receives moving object selection information from the mobile terminal 120. In the mobile terminal 120, the user operates the input device 208 of the mobile terminal 120, and a moving object such as a suspicious person is selected. When the moving object is selected, the communication processing unit 324 of the mobile terminal 120 transmits selection information including the time when the moving object is selected and the selected position to the monitoring server device 110.

  Next, in step S2302, the video analysis unit 311 extracts an individual flow line existing at a position close to the position indicated by the selection information (hereinafter referred to as a moving object position) (flow line extraction process). Specifically, the video analysis unit 311 extracts an individual flow line whose distance from the moving object position is equal to or less than a threshold value. Note that, during the flow line extraction process, the video analysis unit 311 may perform a process of discarding individual flow lines whose flow line extraction time is old and listing only the latest individual flow lines. As a result, it is possible to efficiently extract individual flow lines that are likely to be of the selected moving object. Next, in step S2303, the video analysis unit 311 compares the position of the moving object and the observation point of the individual flow line at the time when the moving object is selected. Then, the video analysis unit 311 selects an individual flow line in which the position of the observation point of the individual flow line matches the moving object position as the individual flow line to be presented.

  In step S2304, the video analysis unit 311 sets a presentation range based on the individual flow line to be presented selected in step S2303. Note that the processing from step S2304 to step S2310 is the same as the processing from step S1103 to step S1109 in the presentation video transmission processing described with reference to FIG. 11 in the first embodiment. However, in the present embodiment, the video analysis unit 311 refers to the individual flow line to be presented instead of the collective flow line to be presented for the process of referring to the collection flow line to be presented.

  The presentation range specifying process (step S2304) performs substantially the same process as the presentation range specifying process according to the first embodiment shown in FIG. However, instead of the collective flow line, an individual flow line to be presented is a processing target. Specifically, in step S1201, the video analysis unit 311 detects the head of the individual flow line to be presented. Next, in step S1202, the video analysis unit 311 identifies one observation point in the individual flow line along the time of the individual flow line in order from the head position detected in step S1201. Next, in step S1203, the video analysis unit 311 specifies the presentation range based on the position and time of the observation point specified in step S1202. The video analysis unit 311 performs the above processing for all observation points included in the individual flow lines (step S1204).

  FIG. 24 is a flowchart illustrating detailed processing in the playlist generation processing (step S2305) by the monitoring server device 110 according to the third embodiment. In step S2401, the video analysis unit 311 reads one record, that is, one presentation range in order from the top of the presentation range data. Next, in step S2402, the video analysis unit 311 identifies the monitoring camera 100 corresponding to the presentation range. Next, in step S2403, the video analysis unit 311 determines whether or not the specified monitoring camera 100 is the same as the monitoring camera 100 specified from the data record in the previous presentation range. If they are the same (Yes in step S2403), the video analysis unit 311 advances the process to step S2407. If not identical (No in step S2403), the video analysis unit 311 advances the process to step S2404.

  In step S2404, the video analysis unit 311 adds the created record to the reproduction list. Next, in step S2405, the video analysis unit 311 creates a new record. Next, in step S2406, the video analysis unit 311 sets the start time of the presentation period of the presentation range read in step S2401 as the start time of the new record. On the other hand, in step S2407, the video analysis unit 311 does not create a new record, but sets the end time of the presentation period read in step S2401 as the end time of the record in the held playlist.

  In step S2408, the video analysis unit 311 determines whether or not the processing in steps S2401 to S2407 has been completed for all the records (presentation ranges) indicated in the presentation range data. When the process is completed (Yes in step S2408), the video analysis unit 311 ends the playlist generation process. If the process has not ended (No in step S2408), the video analysis unit 311 advances the process to S2401. In this case, the video analysis unit 311 reads an unprocessed shooting range in S2401, and repeats the processing from step S2402. In step S2409, the video analysis unit 311 adds the created record to the playlist, and ends the playlist generation process. The remaining configuration and processing of the monitoring system according to the third embodiment are the same as the configuration and processing of the monitoring system according to other embodiments.

  As described above, the monitoring system according to the present embodiment can continuously reproduce and display the video image of the person along the individual flow line related to the specific person. Therefore, the user can efficiently grasp the behavior of the person. For example, the user can quickly grasp whether there was any suspicious behavior such as shoplifting by confirming the past behavior of the suspicious person in the facility with a video.

  As a modification example of the monitoring system according to the third embodiment, the process for selecting the individual flow line to be presented is not limited to the embodiment. As another example, the monitoring server device 110 may use a flow line of a person who has lost his / her accompaniment with the user as a presentation target flow line. In this case, it is possible to quickly find a tour customer who has been separated from the lead in a tour or the like and to check the video at the time of separation, which can be used for analysis of the cause of the separation.

  As described above, according to the monitoring system of the present embodiment, the user can browse while moving the video along the flow line using the device independent of the monitoring camera. In addition, the user can view an image in which the position on the flow line and the time are switched based on characteristics on the flow line such as the occurrence of stay. In this way, the user can efficiently view the video taken by the surveillance camera, so that the situation can be efficiently identified.

<Other embodiments>
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  As described above, according to each of the above-described embodiments, the flow line analysis using the captured video is automatically performed, and the video according to the analysis result is provided, thereby improving the efficiency of the video checking operation by the observer. Can do.

  The preferred embodiments of the present invention have been described in detail above, but the present invention is not limited to such specific embodiments, and various modifications can be made within the scope of the gist of the present invention described in the claims.・ Change is possible.

100 surveillance camera 110 surveillance server device 120 mobile terminal 310 video management unit 311 video analysis unit 312 video playback control unit 314 video DB
315 Flow line DB
316 Playback List 320 Display Processing Unit

Claims (20)

An information processing system having one or more photographing devices that photograph one or more moving objects and an information processing device,
The information processing apparatus includes:
A flow line extracting means for extracting a flow line of one or more moving objects from the video obtained by the photographing apparatus;
A range specifying means for specifying at least one of a presentation area and a presentation period as a presentation range in the space-time imaged by the imaging device based on the flow line;
Video specifying means for specifying a presentation video to be presented to the user based on the presentation range;
An information processing system comprising display means for displaying the presented video.   The information according to claim 1, wherein the range specifying unit specifies the presentation area, and further specifies the presentation range based on the presentation area and a predetermined presentation period. Processing system.   The information processing system according to claim 2, wherein the range specifying unit specifies an area including the flow line as the presentation area. It further has a stay detection means for detecting stay of the moving object in the flow line,
The information processing system according to claim 2, wherein the range specifying unit specifies the presentation area based on the staying position. The flow line extraction means extracts one collective flow line determined by a plurality of moving objects,
A density specifying means for specifying a density at a plurality of points on the collective flow line of the moving object;
The range specifying means specifies the presentation area based on the density of each point, and further specifies the presentation range based on the presentation area and a predetermined presentation period. The information processing system according to claim 1. A plurality of the photographing devices;
The range specifying means specifies a certain presentation area,
Further comprising selection means for selecting the imaging device based on the presentation area,
The information processing system according to claim 1, wherein the video specifying unit specifies a presentation video from videos obtained by the photographing apparatus selected by the selection unit.   If the display processing time point at which the presentation video is displayed is set as the presentation period, the display means displays the real time video captured at the display processing time as the presentation video in real time. The display video displayed during the presentation period is displayed when the presentation period past the display processing time point at which the presentation video is displayed is set. Information processing system described in 1.   The information according to claim 1, wherein the range specifying unit specifies the presentation period, and further specifies the presentation range based on the presentation period and a predetermined presentation area. Processing system.   The information processing system according to claim 8, wherein the range specifying unit specifies the presentation period based on a period during which a video corresponding to the flow line is captured. It further has a stay detection means for detecting stay of the moving object in the flow line,
The information processing system according to claim 8, wherein the range specifying unit specifies the presentation period based on a time when the stay is detected. The flow line extraction means extracts one collective flow line determined by a plurality of moving objects,
A density specifying means for specifying a density at a plurality of points on the collective flow line of the moving object;
The range specifying means specifies the presentation period based on densities at different times, and further specifies the presentation range based on the presentation period and a predetermined presentation area. The information processing system according to claim 1.   The range specifying means specifies the presentation area and the presentation period based on the flow line, and further specifies the presentation range based on the presentation area and the presentation period. The information processing system according to claim 1. A flow line extracting means for extracting a flow line of one or more moving objects from the video obtained by the photographing apparatus;
A range specifying means for specifying at least one of a presentation area and a presentation period as a presentation range in the space-time imaged by the imaging device based on the flow line;
An information processing apparatus comprising: video specifying means for specifying a presentation video to be presented to a user based on the presentation range. An information processing system having one or more photographing devices that photograph one or more moving objects, an information processing device, and a mobile terminal,
The information processing apparatus includes:
A flow line extracting means for extracting a flow line of one or more moving objects from the video obtained by the photographing apparatus;
A range specifying means for specifying at least one of a presentation area or a presentation period as a presentation range in a time-space photographed by the photographing device based on the flow line;
Video specifying means for specifying a presentation video to be presented to the user based on the presentation range;
The portable terminal is
Determining means for determining whether to reproduce and display the presented video based on the position of the mobile terminal;
Receiving means for receiving the presented video from the information processing apparatus when it is determined to be reproduced and displayed;
An information processing system comprising display means for displaying the presented video. The portable terminal is
It further has a photographing means,
The information processing system according to claim 14, wherein the determination unit determines whether to reproduce and display the presentation video based on a position of the mobile terminal and a shooting region by the shooting unit.   16. The information processing system according to claim 15, wherein the display unit superimposes and displays the presentation video on the video shot by the shooting unit. An information processing method executed by an information processing system having one or more photographing devices that photograph one or more moving objects and an information processing device,
A flow line extracting step of extracting a flow line of one or more moving objects from the video obtained by the photographing apparatus;
A range specifying step for specifying at least one of a presentation area and a presentation period as a presentation range in the space-time photographed by the photographing device based on the flow line;
A video identifying step for identifying a presentation video to be presented to the user based on the presentation range;
A display step of displaying the presented video. An information processing method executed by an information processing apparatus,
A flow line extracting step of extracting a flow line of one or more moving objects from the video obtained by the imaging device;
A range specifying step for specifying at least one of a presentation area and a presentation period as a presentation range in the space-time photographed by the photographing device based on the flow line;
And a video specifying step of specifying a video to be presented to the user based on the presentation range. An information processing method executed by an information processing system including one or more photographing devices that photograph one or more moving objects, an information processing device, and a mobile terminal,
A flow line extracting step in which the information processing apparatus extracts a flow line of one or more moving objects from the video obtained by the photographing apparatus;
A range specifying step of specifying at least one of a presentation region or a presentation period as a presentation range in the time-space imaged by the imaging device based on the flow line;
A video identifying step for identifying a presentation video to be presented to the user based on the presentation range;
A determination step of determining whether or not the mobile terminal reproduces and displays the presented video based on a position of the mobile terminal;
A reception step of receiving the presentation video from the information processing device when the portable terminal determines to reproduce and display;
The information processing method characterized by including the display step in which the said portable terminal displays the said presentation image | video. Computer
A flow line extracting means for extracting a flow line of one or more moving objects from the video obtained by the photographing apparatus;
A range specifying means for specifying at least one of a presentation area and a presentation period as a presentation range in the space-time imaged by the imaging device based on the flow line;
A program for functioning as video specifying means for specifying a video to be presented to a user based on the presentation range.

Images