JP6738213B2 - Information processing apparatus and information processing method - Google Patents

Description

The present invention relates to an information processing apparatus and an information processing method that allow an operator to efficiently grasp the behavior of a specific person included in moving image data.

2. Description of the Related Art Conventionally, a technique is known in which face image data of a plurality of persons are respectively searched from image data captured by an imaging device such as a camera, and the plurality of searched face image data are displayed on a display unit.

For example, in Patent Document 1, thumbnail image data of a specific person is associated with one section of a moving image in which it is estimated that a specific person appears consecutively, and the thumbnail image data is displayed as a list of indexes in a display unit. There is disclosed a technique in which display control is performed in advance, and when a plurality of thumbnail image data is selected by an operator, new moving image data for continuously reproducing moving image data in a corresponding section is generated. According to Patent Document 1, the appearance position of a specific person included in the moving image data can be presented to the operator in an easy-to-understand manner, and new moving image data including the specific person can be efficiently generated.

Japanese Patent No. 5087867

However, the technique of Patent Document 1 is effective when generating new moving image data in which only a scene in which a specific person is reflected is generated from the moving image data, but is specified from the moving image data captured by the surveillance camera. There is a problem that it is not possible to efficiently deal with the case where the behavior of a person requiring attention is detected.

For example, when it is found that a certain sensitive person A is reflected at a predetermined time point in the moving image data, it is desired to confirm what kind of action this sensitive person A was taking before and after this time point. There is a case. In such a case, according to the above-mentioned Patent Document 1, while the operator finds the thumbnail image data of the person A who needs attention, the operator selects these thumbnail image data and generates new moving image data regarding the person A who needs attention. After that, it is necessary to reproduce this new moving image data.

According to such a series of processes, there is a problem that it is difficult to quickly detect the behavior of the above-mentioned person A requiring attention. In particular, when there is a possibility that the sensitive person A may act together with another person, a new moving image is generated for another person as a monitoring target as well, which complicates the process. In the actual scene, while the operator is playing the video, it is possible to recognize the behaviors of the person requiring attention and other persons who are reflected in the video, and to be able to quickly detect the situation before and after based on the result. Although there is a need, Patent Document 1 described above cannot meet such a need.

For these reasons, it is an important issue to enable the operator to efficiently grasp the behavior of a specific person included in the moving image data.

The present invention has been made in order to solve the above-mentioned problems of the conventional technology, and provides an information processing apparatus and an information processing method that allow an operator to efficiently grasp the behavior of a specific person included in moving image data. The purpose is to do.

In order to solve the above problems, the present invention relates to a storage unit for storing moving image data and a moving image reproduction display area on a display screen of a predetermined display unit for reproducing the moving image data stored in the storage unit. Display control for displaying the thumbnail images showing the characteristics of the person included in the moving image data stored in the storage unit in a time-series manner in the time-series display area on the display screen. The display control unit, when a predetermined alarm report return operation is performed, reproduces the moving image data of the previous timing on the time line when the alarm report is received from the predetermined alarm report device. When the display control is performed in the display area and the predetermined alarm notification sending operation is performed, the moving image data of the timing after the time when the alarm notification is received from the predetermined alarm notification device is displayed in the moving image reproduction display area. and wherein the display control child.

Further, in the present invention, in the above invention, a characteristic region detection unit that detects a characteristic region of a person included in the moving image data stored in the storage unit, and a person whose characteristic region is detected by the characteristic region detection unit, It further comprises a specifying unit that tracks and specifies a section on the time axis of the moving image data in which the person exists, and a thumbnail image generating unit that generates a thumbnail image of the characteristic region detected by the characteristic region detecting unit. The display control unit controls the display of the thumbnail image generated by the thumbnail image generation unit at a position corresponding to the section specified by the specification unit on the timeline indicating the time axis of the moving image data. And

Further, the present invention according to the above-mentioned invention, further comprising a section selecting unit that selects a desired section on the timeline, and the display control unit selects the desired section on the timeline by the section selecting unit. In this case, the moving image data of the selected section is reproduced and displayed on the moving image reproduction display area.

Further, according to the present invention, a storage unit that stores moving image data and a moving image reproduction display area on a display screen of a predetermined display unit perform reproduction processing of the moving image data stored in the storage unit and display control the same. , A display control unit that controls display of thumbnail images showing the characteristics of a person included in the moving image data stored in the storage unit in time series in the time series display area on the display screen, and is registered in advance. A registered characteristic information storage unit that stores registered characteristic information about a specific person, characteristic information included in a characteristic region of a person included in the moving image data stored in the storage unit, and stored in the registered characteristic information storage unit. An authentication processing unit for performing verification processing with the registered registration characteristic information to authenticate whether the characteristic information corresponds to any of the registered characteristic information, and according to an authentication result by the authentication processing unit, characterized that you vary the display mode of the video playback display feature area included in the display being controlled video region and / or the time series display area on the display being controlled thumbnail image.

Also, in the present invention according to the above-mentioned invention, the display control unit corresponds to any one of the registered characteristic information by the authentication processing unit, on condition that a predetermined authentication return operation or a predetermined authentication sending operation is performed. Then, the moving image data of the section before or after the timeline where the authenticated characteristic information is located is controlled to be displayed in the moving image reproduction display area.

Further, in the present invention according to the above-mentioned invention, a feature area selection unit that selects a feature area having feature information that is displayed in the moving image reproduction display area and has been authenticated by the authentication processing unit to correspond to the registered feature information is further provided. The display control unit, when a characteristic region is selected by the characteristic region selection unit, reproduces a plurality of moving image data corresponding to the selected characteristic region to control display on the moving image reproduction display region. It is characterized by

Further, in the present invention according to the above-mentioned invention, a feature area selection unit that selects a feature area having feature information that is displayed in the moving image reproduction display area and has been authenticated by the authentication processing unit to correspond to the registered feature information is further provided. The display control unit, when the characteristic region is selected by the characteristic region selection unit and a predetermined operation is performed, reproduces the moving image data of another section including the characteristic information to perform the moving image. It is characterized in that display control is performed in the reproduction display area.

Further, the present invention is characterized in that, in the above invention, it further comprises an output unit which outputs the moving image data displayed in the moving image reproduction display area when a predetermined moving image output operation is performed.

Further, according to the present invention, in the above invention, when a predetermined still image output operation is performed, a still image output unit that outputs still image data forming a part of the moving image data displayed in the moving image reproduction display area is provided. It is further characterized by being equipped.

Further, the present invention is an information processing method in an information processing device having a storage unit for storing moving image data, wherein the storage unit stores the moving image reproduction display area on a display screen of a predetermined display unit. A first display control step of reproducing and controlling display of moving image data, and a thumbnail image showing a feature of a person included in the moving image data stored in the storage unit with respect to a time-series display area on the display screen. And a second display control step for controlling the display of each of them in time series, wherein the first display control section issues an alarm from a predetermined alarm issuing device when a predetermined alarm issuing return operation is performed. When the predetermined alarm notification sending operation is performed by controlling the display of the moving image data of the previous timing on the timeline that received the alarm notification, the alarm notification is accepted from the predetermined alarm notification device. It is characterized in that the moving image data at a later timing on the timeline is controlled to be displayed in the moving image reproduction display area .
Also, the present invention is an information processing method in an information processing apparatus having a storage unit for storing the video data for video playback display area on the display screen of a predetermined display unit, stored in the storage unit A first display control step of reproducing and controlling display of the moving image data, and a thumbnail showing a feature of a person included in the moving image data stored in the storage unit, with respect to a time-series display area on the display screen. A second display control step of controlling the display of each image in time series, a storage step of storing registration characteristic information about a specific person registered in advance in a registration characteristic information storage section, and a storage step stored in the storage section. The characteristic information included in the characteristic region of the person included in the moving image data is collated with the registered characteristic information stored in the registered characteristic information storage unit, and the characteristic information corresponds to any of the registered characteristic information. An authentication processing step for authenticating whether or not the characteristic area included in the moving image display-controlled in the moving image reproduction display area and/or the time series display area is displayed according to the authentication result by the authentication processing step. It is characterized in that the display mode of the controlled thumbnail image is different.

According to the present invention, it is possible to efficiently grasp the behavior of a particular person included in the moving image to the operator.

FIG. 1 is an explanatory diagram for explaining the concept of the information processing apparatus according to the present embodiment. FIG. 2 is a diagram showing the system configuration of the information processing system according to the present embodiment. FIG. 3 is a functional block diagram showing the configuration of the information processing apparatus shown in FIG. FIG. 4 is a diagram showing an example of a record viewer displayed on the display unit by the display control unit shown in FIG. FIG. 5 is a diagram showing an example of the operation area shown in FIG. FIG. 6 is a flowchart showing a procedure for acquiring moving image data by the information processing apparatus shown in FIG. FIG. 7 is a flowchart showing a display processing procedure of the record viewer by the information processing apparatus shown in FIG. FIG. 8 is an explanatory diagram (1) for explaining an example of an operation on the record viewer shown in FIG. FIG. 9 is an explanatory diagram (2) for explaining an example of an operation on the record viewer shown in FIG. FIG. 10 is an explanatory diagram (3) for explaining an example of an operation on the record viewer shown in FIG.

<Concept of this embodiment>
Embodiments of an information processing apparatus and an information processing method according to this embodiment will be described below with reference to the accompanying drawings. In the present embodiment, a case will be described in which video data is captured by a camera provided in a store such as a convenience store or a game shop, and the video data is played back on an information processing device communicably connected to the camera. To do.

Further, in the following embodiments, for convenience of description, a case where the “face” of the person is set as the “feature” of the person will be described, but the present invention is not limited to this, and the appearance of the person (appearance) ), a color/pattern of clothes, an attribute including at least one of a person's attributes such as glasses, can be defined as a “feature”.

First, the concept of the information processing device 4 according to the present embodiment will be described. FIG. 1 is an explanatory diagram for explaining the concept of the information processing device 4 according to the present embodiment. As shown in the figure, if moving image data is captured by the camera 1a installed in the store, this moving image data is transmitted to the information processing device 4 installed in the backyard of the store, etc. 4 is stored in the storage unit 44.

Here, for example, if a registered suspicious person is included in the video data, the alert monitor 50 is displayed on the display unit 41, and a message such as “A suspicious person has been detected.” is displayed on the alert monitor 50. Is displayed. Further, when a warning signal is received from the EAS device (illegal take-out prevention device) 2, a message such as “Illegal take-out of merchandise has occurred” is displayed on the alert monitor 50. The EAS device 2 will be described later. Further, since the alert monitor 50 is not a characteristic part of this embodiment, its detailed description will be omitted.

In addition, the person in charge of the store activates the record viewer 60 of the information processing device 4 at a predetermined timing such as after the store is closed to determine what kind of behavior (behavior) the suspicious person or the like has performed in the store. You can check. Specifically, the person in charge of the store causes the record viewer 60 to be displayed on the display unit 41 of the information processing device 4, and the person in charge performs various operations on the record viewer 60, whereby the behavior of a suspicious person or the like is displayed. It becomes possible to grasp.

The information processing device 4 according to the present embodiment is characterized by the record viewer 60, and enables the person in charge of the store to efficiently check the behavior of a suspicious person or the like. Specifically, the record viewer 60 is provided with a camera video display area and a timeline display area, and video data whose conditions have been set by a person in charge is displayed in the camera video display area and included in the video data in the timeline display area. The thumbnail images showing the faces of the people to be displayed are controlled in time series. The camera moving image display area corresponds to the moving image reproduction display area of claim 1, and the timeline display area corresponds to the time series display area of claim 1.

With this, the person in charge of the store watches the video at the corresponding timing on the camera video display area while grasping the appearance timing of the person by referring to the thumbnail image displayed in the timeline display area, and the behavior of the person. It is possible to check efficiently.

Although detailed description will be given later, the record viewer 60 according to the present embodiment can reproduce a moving image by skipping a section where no person exists. This is because the time for reproducing the moving image data in the section where no person exists is wasted.

In addition, the face area of a person included in the moving image data captured by the camera 1a is detected, the person whose face area is detected is tracked, and the section on the time axis of the moving image data in which the person exists is specified. A thumbnail image is displayed at a position corresponding to the specified section on the line. When there are a plurality of thumbnail images of different persons to be displayed at the same position, the plurality of thumbnail images are displayed and controlled in multiple stages.

In addition, the registered face data of the suspicious person registered in advance is stored, the face information included in the face area detected from the video data and the registered face data are collated, and the face information is registered face data. It is authenticated whether or not any one of them corresponds, and according to the authentication result, thumbnail image data display-controlled in the face area and/or the timeline display area included in the video data display-controlled in the camera video display area. The display mode of can be different. In the present embodiment, if any of the registered face data is applicable, a rectangle enclosing the thumbnail image data is displayed by a broken line, and if none of the registered face data is applied, a rectangle enclosing the thumbnail image is displayed by a solid line. I am going to do it. In addition, for example, when it corresponds to the registered face data, it can be highlighted by using a rectangular frame such as red or a double line.

When a section on the timeline or thumbnail image data is selected, display of the moving image data on the time axis corresponding to this section is controlled in the camera moving image display area. In addition, the face area of the person displayed in the camera moving image display area is selected, and a predetermined operation (press operation of the “next face>>” button or the “<previous face” button described later) is performed. If so, the display control is performed on the video data before and after including this person in the camera video display area. In addition, the thumbnail image data displayed in the timeline display area is selected, and the above-described predetermined operation (operation of pressing the "next face>>" button or "<<previous face" button described later) is performed. Even when the person is broken, the display of the moving image data before and after including this person is controlled in the camera moving image display area.

In addition, when a predetermined operation (operation of pressing the “previous authentication>>” button or the “next authentication” button described later) is performed, a face that matches any of the registered faces registered in advance is displayed. Moving to a previous or later time, moving image data is displayed in the camera moving image display area. Further, when a predetermined operation (pressing the “previous EAS notification>>” button or the “next EAS notification” button described later) is performed, the time is shifted to the time before or after the EAS notification and the camera movie is displayed. Video data is displayed in the display area.

As described above, the information processing apparatus 4 according to the present embodiment displays the camera moving image display area displaying the moving image data on the record viewer 60 and the thumbnail image data of the face of the person who appears on the time axis of the moving image data. A timeline display area is provided, and these displays are configured to interlock with each other, and the moving image data can be efficiently viewed.

<System configuration>
Next, the system configuration of the information processing system according to the present embodiment will be described. FIG. 2 is a diagram showing the system configuration of the information processing system according to the present embodiment. As shown in the figure, in this information processing system, a plurality of cameras 1a to 1b (hereinafter collectively referred to as "camera 1"), an EAS device 2 and an information processing device 4 are connected to a network 3. ing.

The camera 1 is an imaging device that captures moving image data installed in a store at a predetermined angle of view, and is connected to the network 3. The moving image data captured by the camera 1 is transmitted to the information processing device 4 via the network 3.

The EAS device 2 is an unauthorized carry-out prevention (shoplifting prevention) device that forms a part of an electronic article surveillance system (Electric Article Surveillance). The EAS device 2 is provided as a gate at the entrance of a store and has a function as a reader/writer of an RFID tag. When the EAS device 2 detects an RFID tag attached to a product, an alarm is given by a buzzer or voice, and the information processing device 4 is notified of the identification information of the RFID tag or the like via the network 3.

The network 3 is a communication means for communicatively connecting the camera 1, the EAS device 2, and the information processing device 4, and here, a case where a wired LAN (Local Area Network) is used is shown. However, the network 3 is not limited to a wired LAN, and can be constructed by a wireless LAN or the like. Further, instead of using the network 3, the information processing device 4 and the camera 1 and the information processing device 4 and the EAS device 2 may be directly connected by a cable.

The information processing device 4 is a computer used when reproducing and analyzing moving image data captured by the camera 1. By displaying the record viewer 60 on the display unit 41 of the information processing apparatus 4, the behavior of a specific person can be efficiently confirmed on the record viewer 60.

<Configuration of information processing device 4>
Next, the configuration of the information processing device 4 shown in FIG. 2 will be described. FIG. 3 is a functional block diagram showing the configuration of the information processing device 4 shown in FIG. As shown in the figure, the information processing device 4 includes a display unit 41, an operation unit 42, a communication I/F unit 43, a storage unit 44, and a control unit 45.

The display unit 41 is a display device such as a liquid crystal panel, and various display screens such as the alert monitor 50 and the record viewer 60 are displayed on the display unit 41. The operation unit 42 is an input device such as a mouse and a keyboard. The communication I/F unit 43 is a communication interface unit for communicating with the camera 1 and the EAS device 2 via the network 3.

The storage unit 44 is a storage device such as a hard disk device or a non-volatile memory, and stores moving image data 44a, registered face data 44b, thumbnail image data 44c, and EAS data 44d.

The moving image data 44a is moving image data captured by the camera 1, and also includes attribute information such as the shooting date and time and the camera number of the camera that shot the image. This moving image data 44a is H.264. It may be encoded and compressed in a compression format such as H.264 or MPEG (Moving Picture Experts Group). The information processing device 4 sequentially receives the moving image data captured by the camera 1, and stores it in the storage unit 44 as a file together with attribute information such as the image capturing date and time and the camera number.

The registered face data 44b is face image data registered in advance as a suspicious person, and when a face area displayed in the camera moving image area is designated and a predetermined operation is performed, the still image data of this face area is changed to moving image data. And is stored in the storage unit 44 as registered face data 44b. In the present embodiment, for convenience of description, a case of storing registered face data of a suspicious person will be described, but registered face data of an important person such as VIP can also be stored.

The thumbnail image data 44c is face image data displayed in the timeline display area, and is thumbnail image data generated from the face area including the person's face included in the moving image data 44a. The thumbnail image data 44c is provided with identification information that uniquely identifies the thumbnail image data as attribute information.

The EAS data 44d is history data received from the EAS device 2, and includes the time when the EAS device 2 issued the EAS, and the like.

The control unit 45 is a control unit that performs overall control of the information processing device 4, and includes a moving image data search processing unit 45a, a face area detection unit 45b, a section identification unit 45c, a thumbnail image generation unit 45d, an authentication processing unit 45e, and display control. The unit 45f and the data output unit 45g are included. Specifically, these functional units are realized by the CPU of the computer loading the programs corresponding to these units onto the internal memory and executing them.

The moving image data search processing unit 45 a is a processing unit that searches for the specified moving image data 44 a from the plurality of moving image data 44 a stored in the storage unit 44. Specifically, when the condition designation buttons 61 to 65 on the record viewer 60 shown in FIG. 4 are operated to input a condition, the moving image data 44a corresponding to the condition is searched using the attribute information of the moving image data 44a as a key. To be done.

The face area detection unit 45b is a processing unit that searches for a face area of a person included in the video data 44a searched by the video data search processing unit 45a. Specifically, when the moving image data 44a is compression-encoded, the moving image data 44a is sequentially expanded on a predetermined image memory, and template matching using a template corresponding to a person's face area is performed. Thus, the face area including the face of the person is detected together with the time on the timeline and the coordinates on the moving image data. By performing such processing, it is possible to specify at which position on the timeline of the moving image data 44a the face area of the person exists.

The section identifying unit 45c is a processing unit that performs processing for identifying a section on the timeline in which a person is included in the moving image data 44a. Specifically, when the face area is searched by the face area searching unit 45b, the person of the face included in the face area is tracked in time series using the image processing technique, and the section in which the person appears continuously Is specified as a person existing section. A well-known image processing technique can be used as the tracking processing technique.

For example, when the moving image data 44a is not compressed and is composed of a plurality of still image data that continues at regular intervals, the still image data at the time when the face area is searched and the next still image data A person is tracked by obtaining a vector of the area, and when both still image data include the same person, they are specified as continuous sections. In addition, when a person who is reflected in the moving image data 44a disappears in the middle and is then reflected again in the moving image data 44a, it is considered as two sections instead of one section, but obviously the same. In the case of a person, even if the person disappears for a period of time, it can be considered as one section.

The thumbnail image generation unit 45d is a processing unit that generates thumbnail image data to be displayed in the timeline display area of the record viewer 60. Specifically, the image data of the face area searched by the face area search unit 45b is scaled (normalized) to a predetermined size (for example, 120 pixels vertically×90 pixels horizontally) to generate thumbnail image data. It is also possible to remove the background portion other than the face portion of the person to generate thumbnail image data showing the face of the person. Further, the time and coordinate position on the timeline of the moving image data 44a and identification information for uniquely identifying the thumbnail image data are added to the thumbnail image data as attribute information and stored in the storage unit 44. Further, when there are a plurality of thumbnail image data (for example, the cross-correlation value is a predetermined value or more) that are similar to each other, these thumbnail image data can be shared (the same one is used).

The authentication processing unit 45e is a processing unit that, when acquiring the moving image data 44a captured by the camera 1, performs an authentication process on whether or not the person of the registered face data 44b is reflected in the moving image data 44a. Specifically, the moving image data 44a is sequentially expanded in the image memory, and it is determined by image processing whether or not the expanded image data has a face corresponding to the registered face data 44b. For example, template matching is performed using the registered face data 44b as a template, and when the cross-correlation coefficient is greater than or equal to a predetermined value, it is determined that the person corresponding to the registered face data 44b is reflected. If the face corresponding to the registered face data 44b is present as a result of this authentication processing, the time and position information on the timeline of the moving image data 44a is temporarily stored in the storage unit 44.

The display control unit 45f is a control unit that performs display control on the display unit 41, and controls the display of the alert monitor 50, the record viewer 60, and the like on the display unit 41. Specifically, the display control unit 45f causes the display unit 41 to display a record viewer 60 having a camera moving image display area and a timeline display area. The camera moving image display area reproduces the condition-set moving image data 44a, and the timeline display area displays thumbnail image data representing a person's face at a corresponding temporal position on the timeline. As a result, the person in charge of the store watches the video at the corresponding timing in the camera video display area while grasping the appearance timing of the person by referring to the thumbnail image displayed in the timeline display area to see the behavior of the person. It is possible to check efficiently. An example of the record viewer 60 displayed on the display unit 41 by the display control unit 45f will be described later.

The data output unit 45g is a processing unit that specifies the face of the person displayed in the camera moving image display area and captures and outputs the still image data at this point when a predetermined output operation is performed. .. Such output may be storage in an external memory such as a flash memory, data transfer to another device, or print output.

<Example of the record viewer 60>
Next, an example of the record viewer 60 displayed on the display unit 41 by the display control unit 45f shown in FIG. 3 will be described. FIG. 4 is a diagram showing an example of the record viewer 60 displayed on the display unit 41 by the display control unit 45f shown in FIG. As shown in the figure, in this record viewer 60, a camera designation button 61, a calendar 62, a date setting button 63, an update button 64, a condition display area 65, a still image output button 66, a moving image output button 67, a shooting date/time display. An area 68, a camera moving image display area 70, an operation area 80, and a timeline display area 90 are included.

The camera designation button 61 is a button for designating the type of camera, and designates which of the cameras 1a to 1b the moving image data captured is to be the target. The calendar 62 is for designating the date when the moving image data was captured by the camera 1. When the date on the calendar 62 is designated, the corresponding date is specified. The “today” button 63 is a button for simple input that specifies today's date. The “update” button 64 is a button for updating the display content of the condition display area 65. When the "update" button 64 is operated, the display content of the condition display area 65 becomes the latest status.

The condition display area 65 is a list of moving image data 44a that matches the specified condition among the moving image data 44a stored in the storage unit 44, and here, 07:00, 07:10, 07:20. Shows the situation in which nine video data, each of which has started shooting at 0, 07:30, 07:40, 07:50, 08:00, 08:10, and 08:20, are displayed. There is.

Here, if there is a check in the "Face" column, it means that a person's face was found in that time zone, and if there is a check in the "Authentication" column, the person who matches the registrant in that time zone Indicates that it was found. Further, if there is a check in the "EAS notification" column, it indicates that a signal has been received from the EAS device during that time period. That is, by referring to the condition display area 65, it is possible to know whether or not each event of "face", "authentication", and "EAS operation" has occurred.

Although the case where the information on the occurrence of the event is displayed using the condition display area 65 is shown here, for example, when the “authentication” column in the condition display area 65 is checked, the corresponding moving image data is displayed. It is also possible to perform the authentication process with the registrant and not perform the authentication process when the “authentication” column is not checked.

The still image output button 66 is an operation button used when outputting still image data, and when the still image output button 66 is operated in a situation where moving image data is displayed in the camera moving image display area 70, Still image data at the time of operation is captured and stored in the storage unit 44 or a flash memory (not shown). By operating the still image output button 66, a still image can be printed or the still image data can be data-communicated with another device.

The moving image output button 67 is an operation button used when outputting moving image data, and when the moving image output button 67 is operated in a situation where moving image data is displayed in the camera moving image display area 70, a camera moving image display is displayed. The moving image data displayed in the area 70 is stored in the storage unit 44, a flash memory (not shown), or the like.

In the imaging date/time display area 68, information regarding the imaging date/time of the moving image data displayed in the camera moving image display area 70 is displayed. As such information, the information added as the attribute information of the moving image data 44a is acquired and displayed. It should be noted that as the reproduction of the moving image progresses, the imaging time advances in association with the reproduction situation.

The camera moving image display area 70 is a display area for reproducing and displaying the moving image data 44a that matches the input conditions. The face area included in the moving image data displayed in the camera moving image display area 70 is marked and displayed regardless of whether it is the same as any of the registered face data. In this embodiment, the face area determined to correspond to any of the registered face data is displayed by a broken line, and the face area determined not to correspond to any of the registered face data is displayed in a solid line. However, the face area determined to correspond to any of the registered face data can be highlighted in a different manner from the normal one by coloring such as red.

The operation area 80 is an operation unit for performing operations such as a feed operation and a return operation of the moving image data displayed in the camera moving image display area 70. Specifically, as shown in FIG. 5, when the "previous" button 80a is operated, the moving image data 44a immediately before is moved, and when the "next" button 80b is operated, The moving image data 44a, which is one after the time, is entered. The bar 80c shows the timeline, and the time tab 80d shows the time on the timeline. When the operator slides the bar 80c while clicking the time tab 80d, the reproduction start time of the video data displayed in the camera video display area 70 jumps to the time corresponding to the position of the time tab 80d. ..

When the "play" button 80e is operated, the moving image data is played from the jumped portion. Further, when the “<” button (return button) 80f is operated, the moving image data displayed in the camera moving image display area 70 is returned for a predetermined time, and when the “>” button (forward button) 80g is operated, the camera moving image is displayed. The moving image data displayed in the display area 70 advances for a predetermined time. Also, each time the "play" button 80e is pressed, the double speed reproduction of the moving image data can be performed (for example, no double speed→2 times→4 times→8 times→no double speed...).

Further, when the "<<previous face" button 80h or the "next face>>" button 80i is operated, the section in which no face exists is skipped and the moving image data 44a is jumped to the position before and after the face exists. This is because the time to play a video without people is wasted. Since the face area detection unit 45b detects the face area together with the time in advance, such processing can be performed.

If the face area of the video data displayed in the camera video display area 70 is selected and operated, and the “<<previous face” button 80h or the “next face>>” button 80i is operated, the same face area is displayed. Skips the section where does not exist, and jumps to the positions before and after the same face area exists.

Further, when the “<<previous authentication” button 80j or the “next authentication>>” button 80k is operated, a jump is made to the time at which the face determined to correspond to the registered face data appears, and the camera moving image display area is displayed. Video data is displayed on 70.

When the "<<Previous EAS notification" button 80l or the "Next EAS notification>>" button 80m is operated, the time shifts to the time before and after the EAS notification and the video is displayed in the camera video display area 70. The data is displayed.

Returning to the description of FIG. 4, thumbnail image data of the person detected by the face area detection unit 45b is displayed in the timeline display area 90 at a position corresponding to the corresponding time on the bar 80d indicating the timeline. When a plurality of persons are reflected in the moving image data, thumbnail image data of each person is displayed in multiple stages. When one of the thumbnail image data is selected and operated, the moving image data of the portion corresponding to the selected and operated thumbnail image data on the camera moving image display area 70 is reproduced and displayed.

<Procedure for obtaining video data>
Next, a procedure for acquiring moving image data by the information processing device 4 shown in FIG. 3 will be described. FIG. 6 is a flowchart showing a procedure for acquiring moving image data by the information processing device 4 shown in FIG.

When the information processing device 4 and the camera 1 are turned on, they are activated respectively, the camera 1 starts imaging, and the information processing device 4 enters a standby state for receiving moving image data (step S101; No). Here, when the information processing device 4 receives the transmission notification of the moving image data from the camera 1 (step S101; Yes), the moving image data is received (step S102), the process of detecting a human face is executed, and the information is detected. A matching process is performed to determine whether or not the face matches the registered face. The processing result and the moving image data are stored as a file in the storage unit 44 (step S103). The shooting condition data such as the shooting date and time is stored as the attribute information of the file.

By performing the series of processes described above, the moving image data captured by the camera 1 and the execution results of face detection and face authentication are sequentially accumulated in the storage unit 44 only by turning on the power of the information processing device 4, and the record viewer is recorded. The state can be confirmed at 60.

<Display processing of the record viewer 60>
Next, a display processing procedure (No. 2) of the record viewer 60 by the information processing apparatus 4 shown in FIG. 3 will be described. FIG. 7 is a flowchart showing a display processing procedure of the record viewer 60 by the information processing device 4 shown in FIG.

As shown in the figure, when the operator accepts the setting condition in the record viewer 60 (step S201), the moving image data search processing unit 45a searches the storage unit 44 for the moving image data 44a that matches the setting condition (step S201). S202). Specifically, with reference to the attribute information of the moving image data, the moving image data 44a that matches the setting condition is searched.

After that, the face area detection unit 45b performs processing for detecting a face area from the moving image data 44a (step S203), and the authentication processing unit 45e performs authentication processing (step S204). Specifically, it is determined whether or not the face included in the face area detected by the face area detecting unit 45b corresponds to any of the registered face data 44b registered in advance.

After that, the section specifying unit 45c performs processing for specifying the section in which the face area exists (step S205), and the thumbnail image generating unit 45d generates a thumbnail image (step S206). The series of processes is performed on all the moving image data 44a searched by the moving image data search processing unit 45a (step S207; No).

When all the moving image data 44a have been processed (step S207; Yes), the thumbnail image data 44c is displayed at the corresponding position in the timeline display area 90 (step S208), and the moving image data 44a is displayed by the camera. It is displayed in the moving image display area 70 (step S209), and the face area included in the moving image data 44a displayed in the camera moving image display area 70 is marked and displayed (step S210). At this time, if any of the registered face data 44b is found as a result of the authentication processing, a rectangle surrounding the thumbnail image data is displayed by a broken line.

<Example of operation of the record viewer 60>
Next, an example of an operation on the record viewer 60 shown in FIG. 4 will be described. 8 to 10 are explanatory views for explaining an example of an operation on the record viewer 60 shown in FIG. When the “<<previous face” button 80h is operated on the record viewer 60 shown in FIG. 4, as shown in FIG. 8, the process moves to the time when the face area is detected before that, and the camera moving image display area is displayed. From this point, moving image data is reproduced and displayed on 70.

By temporarily storing in the storage unit 44 the detection result in which the face area is previously detected from the moving image data 44a by the face area detection unit 45b, the time on the timeline, and the coordinate information, the "<<previous face" button 80h When is operated, it is possible to jump to the time when the face area was detected before that.

Further, when the “<<previous face” button 80h is further operated on the record viewer 60 shown in FIG. 8, as shown in FIG. From this point, moving image data is reproduced and displayed in the moving image display area 70.

On the other hand, when the face area of a specific person in the camera moving image display area 70 of the record viewer 60 shown in FIG. 4 is selected and the “<<previous face” button 80h is operated, as shown in FIG. Moving to the time when this person exists, the video data is reproduced and displayed in the camera video display area 70 from this time.

By obtaining the correlation value of the face area detected by the face area detection unit 45b and determining in advance whether or not it is the face of the same person, it is possible to jump to the time when the same person exists. Becomes

As described above, the information processing apparatus 4 according to the present embodiment includes the camera movie display area 70 and the timeline display area 90 in the record viewer 60, and displays the movie data for which conditions are set in the camera movie display area 70. In addition, since the thumbnail image data showing the face of the person included in the moving image data is displayed and controlled in time series in the timeline display area 90, the person included in the moving image data while referring to the timeline display area 90 is displayed. It is possible to efficiently grasp the behavior of a person by grasping

Further, in the record viewer 60 according to the present embodiment, by skipping the section in which no person exists, the moving image is reproduced, so that it is possible to eliminate waste associated with reproducing a moving image of a scene in which no person exists.

In addition, the face area of a person included in the moving image data captured by the camera 1 is detected, the person in which the face area is detected is tracked, and the section on the time axis of the moving image data in which the person exists is specified. Since the thumbnail image is displayed at the position corresponding to the specified section on the line, the thumbnail image data of the person included in the moving image data can be efficiently displayed at an appropriate position.

Further, when there are a plurality of thumbnail images of different persons to be displayed at the same position, the display control of the plurality of thumbnail images is performed in multiple stages, so that it is possible that a plurality of persons exist at the same time. However, it is possible to efficiently grasp the behavior of each person. Also, even if multiple thumbnail images are displayed in multiple stages at the same position, it cannot be concluded that these persons are involved with each other, but it is possible to suspect fraudulent acts in a group of multiple people. Is possible. In particular, when thumbnail images of multiple people are displayed in multiple stages at many timings, it is more likely that the action is as a group, and by checking the video playback, it is possible to confirm the relationship between people and the group. It is possible to understand that it is a behavior. It is also expected that the detection rate of a group can be increased by automatically incorporating a scene in which a plurality of the same persons are photographed and displaying the result.

In addition, the registered face data 44b of the suspicious person is stored in the storage unit 44 in advance, and the face information included in the face area detected from the moving image data 44a is collated with the registered face data 44b to obtain the face information. Whether or not it corresponds to any of the registered face data 44b is verified, and the face area and/or the timeline display area 90 included in the video data display-controlled in the camera video display area 70 is displayed in accordance with the authentication result. Since the display mode of the display-controlled thumbnail image data is different, it is possible to efficiently deal with a suspicious person or other person requiring attention. In particular, when any of the registered face data 44b is applicable, the thumbnail image data is highlighted with a broken line or the like, so that it is possible to easily recognize the person requiring attention.

Further, when the section on the timeline or the thumbnail image data is selected, the moving image data on the time axis corresponding to this section is controlled to be displayed in the camera moving image display area 70, so that a desired moving image can be quickly displayed. It becomes possible to reproduce. If the face area of the person displayed in the camera moving image display area 70 is selected and a predetermined operation (operation of the forward button, etc.) is performed, the next moving image data including this person is displayed in the camera moving image display area. Since the display control is performed on 70, the behavior of the desired person can be efficiently grasped.

Further, when the section on the timeline or the thumbnail image data is selected, the moving image data on the time axis corresponding to this section is configured to be displayed and controlled in the camera moving image display area 70. It is possible to efficiently grasp the behavior of a desired person by using.

Note that, in the present embodiment, the case where the face area that has been subjected to the authentication processing when it corresponds to the registered face data 44b is displayed by the broken line and the other face areas are displayed by the solid line has been shown, but the present invention is not limited to this. Instead, it is also possible to highlight in red or the like.

INDUSTRIAL APPLICABILITY The information processing apparatus and the information processing method of the present invention are useful when the operator can efficiently grasp the behavior of a specific person included in moving image data.

1, 1a, 1b Camera 2 EAS device 3 Network 4 Information processing device 41 Display unit 42 Operation unit 43 Communication I/F unit 44 Storage unit 44a Video data 44b Registered face data 44c Thumbnail image data 44d EAS data 45 Control unit 45a Video data Search processing unit 45b Face area detecting unit 45c Section specifying unit 45d Thumbnail image generating unit 45e Authentication processing unit 45f Display control unit 45g Data output unit 50 Alert monitor 60 Record viewer 61 Camera designation button 62 Calendar 63 Today button 64 Update button 65 Condition display Area 66 Still image output button 67 Video output button 68 Imaging date/time display area 70 Camera video display area 80 Operation area 90 Timeline display area

Claims (11)

A storage unit for storing moving image data,
In a moving image reproduction display area on a display screen of a predetermined display unit, the moving image data stored in the storage unit is reproduced and display-controlled, and the time series display area on the display screen is And a display control unit for controlling display of thumbnail images showing the features of the person included in the moving image data stored in the storage unit in time series,
The display control unit,
When a predetermined alarm reporting return operation is performed, display control of the moving image data of the previous timing on the timeline that has received the alarm reporting from the predetermined alarm reporting device in the moving image playback display area,
When a predetermined alarm notification sending operation is performed, display control is performed on the moving image reproduction display area of moving image data at a later timing on the time line when the alarm notification is received from the predetermined alarm issuing device. Information processing device. A characteristic region detection unit that detects a characteristic region of a person included in the moving image data stored in the storage unit;
An identifying unit that tracks a person whose characteristic region is detected by the characteristic region detecting unit and identifies a section on the time axis of the moving image data in which the person exists,
A thumbnail image generation unit that generates a thumbnail image of the characteristic region detected by the characteristic region detection unit,
The display control unit,
The thumbnail image generated by the thumbnail image generation unit is controlled to be displayed at a position corresponding to the section specified by the specification unit on the timeline indicating the time axis of the moving image data. Information processing equipment. Further comprising a section selection unit for selecting a desired section on the timeline,
The display control unit,
When the desired interval on the time line is selected by the section selecting unit, to claim 2 and playback processing video data of the selected section, characterized in that the display control on the video play display area The information processing device described. A storage unit for storing moving image data,
In a moving image reproduction display area on a display screen of a predetermined display unit, the moving image data stored in the storage unit is reproduced and display-controlled, and the time series display area on the display screen is A display control unit that controls the display of thumbnail images showing the characteristics of the person included in the moving image data stored in the storage unit in time series,
A registered characteristic information storage unit that stores registered characteristic information about a specific person registered in advance;
The characteristic information included in the characteristic region of the person included in the moving image data stored in the storage unit is collated with the registered characteristic information stored in the registered characteristic information storage unit, and the characteristic information is the registered characteristic. An authentication processing unit that authenticates whether or not it corresponds to any of the information,
According to the authentication result by the authentication processing unit, the display mode of the characteristic region included in the moving image display-controlled in the moving image reproduction display region and/or the display mode of the thumbnail image display-controlled in the time-series display region may be changed. A characteristic information processing device. The display control unit,
On condition that a predetermined authentication return operation or a predetermined authentication sending operation has been performed, the front or rear of the timeline where the characteristic information authenticated as corresponding to any of the registered characteristic information by the authentication processing unit is located. The information processing apparatus according to claim 4 , wherein display control is performed on the moving image data of the section in the moving image reproduction display area. Further comprising a characteristic region selection unit that is displayed in the moving image reproduction display region and that selects a characteristic region having the characteristic information that has been authenticated by the authentication processing unit to correspond to the registered characteristic information,
The display control unit,
If the feature region by the feature area selection portion is selected, according to claim 4, characterized in that playing processing a plurality of video data corresponding to the selected feature region display control on the video play display area Alternatively , the information processing device according to item 5 . Further comprising a characteristic region selection unit that is displayed in the moving image reproduction display region and that selects a characteristic region having the characteristic information that has been authenticated by the authentication processing unit to correspond to the registered characteristic information,
The display control unit,
When a characteristic region is selected by the characteristic region selection unit and a predetermined operation is performed, the moving image data of the other section including the characteristic information is reproduced to control the display in the moving image reproduction display region. The information processing apparatus according to claim 4 or 5 , characterized in that: When a predetermined video output operation is performed, information of any one of claims 1-7, characterized in that the output unit further comprising outputting the video data displayed on the moving image reproduction display area Processing equipment. The still image output unit is further provided for outputting still image data forming a part of the moving image data displayed in the moving image reproduction display area when a predetermined still image output operation is performed. The information processing apparatus according to any one of items 1 to 8 . An information processing method in an information processing device having a storage unit for storing moving image data,
A first display control step of performing display processing by performing reproduction processing of the moving image data stored in the storage unit in a moving image reproduction display area on a display screen of a predetermined display unit;
A second display control step of time-sequentially controlling display of thumbnail images showing the characteristics of the person included in the moving image data stored in the storage unit in the time-series display area on the display screen. ,
In the first display control step,
When a predetermined alarm reporting return operation is performed, display control of the moving image data of the previous timing on the timeline that has received the alarm reporting from the predetermined alarm reporting device in the moving image playback display area,
When a predetermined alarm notification sending operation is performed, display control is performed on the moving image reproduction display area of moving image data at a later timing on the time line when the alarm notification is received from the predetermined alarm issuing device. Information processing method. An information processing method in an information processing device having a storage unit for storing moving image data,
A first display control step of performing a reproduction process of the moving image data stored in the storage unit to display control the moving image reproduction display area on the display screen of the predetermined display unit;
A second display control step of performing time-series display control of the thumbnail images showing the characteristics of the person included in the moving image data stored in the storage unit in the time-series display area on the display screen;
A storage step of storing registration characteristic information relating to a specific person registered in advance in the registration characteristic information storage section;
Characteristic information included in the characteristic area of the person included in the moving image data stored in the storage unit,
An authentication processing step of performing a verification process with the registered characteristic information stored in the registered characteristic information storage unit to authenticate whether the characteristic information corresponds to any of the registered characteristic information;
According to the authentication result of the authentication processing step, the display mode of the characteristic region included in the moving image display-controlled in the moving image reproduction display region and/or the display mode of the thumbnail image display-controlled in the time-series display region may be changed. A characteristic information processing method.

Images