JP2020064684A - Control device, control method, and program - Google Patents

Abstract

To appropriately present whether there is a single person or are a plurality of persons in an image.SOLUTION: Persons are detected from a moving image. On the basis of at least one of positions of the detected persons, orientations of faces of the persons, and distance between the persons, the persons are associated with each other. At least one of information enabling identification that a person who cannot be associated with the other person is included and information enabling identification that a person who cannot be associated with the other person is not included is displayed on a display device.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a display control device, a display control method, and a program for detecting a person from a moving image and displaying it on a display device.

  Conventionally, a method of detecting a person from an image is known. Patent Document 1 discloses a method of detecting a person shielded from an image.

JP, 2014-199506, A

The technique disclosed in Patent Document 1 described above can notify the position of a person in an image. For example, in the above technique, a person existing in the image can be circled and the position of the person can be presented to the user. However, only by indicating the position of a person based on the above technique, does the user determine whether the detected person is isolated from the group and working alone, or collaborating with another person at a close position? There was a problem that I could not judge at a glance. For example, at a site that handles data that should prevent leakage, rules that prohibit working alone may be imposed, and a work monitor may monitor workers via an imaging device, There are times when you want to call attention to a single worker who is working independently from the group. However, only by looking at the captured image, the work monitor cannot determine at a glance whether each of the workers in the captured image is working alone or a plurality of workers. Further, it is a heavy burden for the work supervisor to constantly monitor the worker.
An object of the present invention is to appropriately present whether a person in an image is a single person or a plurality of persons.

  Therefore, the display control device of the present invention includes at least one of a detection unit that detects a person from a moving image, a position of the person detected by the detection unit, a face direction of the person, and a distance between the persons. Based on, the association means for associating the persons, information identifiable to include a person who is not associated with another person by the association means, and a person who is not associated with another person by the association means are included. Display control means for displaying on the display device at least one of the information that can be identified as not being displayed.

  According to the present invention, it is an object to appropriately present whether a person in an image is a single person or a plurality of persons.

It is a figure showing an example of a system configuration and hardware configuration of a system. It is a figure showing an example of functional composition of an imaging device and a client device. It is a flow chart which shows an example of processing of notice control. It is a figure which shows an example of a captured image. It is a figure which shows an example of the image which shows the result of having performed person detection. It is a figure which shows an example of a horizontal plane image. It is a figure which shows an example of a work area frame. It is a figure which shows an example of a display screen. It is a figure which shows an example of a display screen. It is a figure which shows an example of a display screen. It is a figure which shows an example linked with the entry / exit management system. It is a flow chart which shows an example of processing of notice control. It is a figure explaining an example of judgment.

  Embodiments of the present invention will be described below with reference to the drawings.

<Embodiment 1>
In the present embodiment, in the field where data that should prevent leakage is obligatory for two or more workers to work, a work supervisor is an independent worker (a person who is not associated with another person). ) Or an image including a collaborator (a person associated with another person) will be described. Such a situation is an example of a situation in which the present embodiment can be used, and can be used in various situations in which it is desired to determine whether a person is a single person or a plurality of persons. For example, the present invention can also be used in a situation where it is used for marketing, solving social problems, etc. by determining whether a passerby or a visitor is a single person or a group.
FIG. 1 shows an example of the system configuration and hardware configuration of the system in this embodiment. In FIG. 1, the imaging device 110 performs imaging. The client device 120 drives the imaging device 110 and displays a captured image captured by the imaging device 110. The input device 130 includes a mouse, a keyboard, and the like, and inputs a user input to the client device 120. The display device 140 includes a display and the like, and displays the image output by the client device 120. Although the client device 120 and the display device 140 are shown as independent devices in FIG. 1, the client device 120 and the display device 140 may be integrated. Further, the input device 130 and the display device 140 may be integrated, or the client device 120, the input device 130, and the display device 140 may be integrated. The network 150 connects the imaging device 110 and the client device 120. The network 150 is composed of, for example, a plurality of routers, switches, cables, etc. that satisfy a communication standard such as a local network. In the present embodiment, any communication standard, scale, and configuration can be used as long as communication can be performed between the imaging device and the client. For example, the network 150 may be configured by the Internet, a wired LAN (Local Area Network), a wireless LAN (Wireless LAN), a WAN (Wide Area Network), or the like. Further, the number of image pickup devices connected to the client device 120 is not limited to one, and may be plural.

The hardware configuration of the imaging device 110 will be described. The imaging device 110 includes an imaging unit 11, a CPU 12, a memory 13, and a communication I / F 14 as a hardware configuration. The image pickup unit 11 has an image pickup element and an engineering system of a subject on the image pickup element, and under the control of the CPU 12, picks up an image on the image pickup element with the intersection of the optical axis of the optical system and the image pickup element as the image pickup center. The image pickup device is an image pickup device such as a CMOS (Complementary Metal-Oxide Semiconductor) or a CCD (Charged Coupled Device). The CPU 12 controls the entire imaging device 110. The memory 13 stores a program, an image captured by the image capturing unit 11, data used when the CPU 12 executes a process, and the like. The communication I / F 14 controls communication with the client device 120 via the network 150 under the control of the CPU 12. The functions and the like of the image pickup apparatus 110 are realized by the CPU 12 executing processing based on the programs stored in the memory 13. Note that a processor other than the CPU may be used.
The hardware configuration of the client device 120 will be described. The client device 120 includes, as a hardware configuration, a memory 21, a CPU 22, a communication I / F 23, an input I / F 24, and a display I / F 25. The CPU 22 controls the entire client device 120. The memory 21 stores a program, a captured image transmitted from the imaging device 110, data used when the CPU 22 executes processing, and the like. The communication I / F 23 controls communication with the imaging device 110 via the network 150 under the control of the CPU 22. The input I / F 24 connects the client device 120 and the input device 130, and controls the input of information from the input device 130. The display I / F 25 connects the client device 120 and the display device 140, and controls the output of information to the display device 140. When the CPU 22 executes processing based on the program stored in the memory 21, the function of the client device 120 and the processing of the flowcharts of FIGS. 3 and 12 described later are realized.

FIG. 2 is a diagram illustrating an example of functional configurations of the image capturing apparatus 110 and the client apparatus 120. The imaging device 110 includes a control unit 111, a signal processing unit 112, a drive control unit 113, and a communication control unit 114 as a functional configuration. The control unit 111 controls the imaging device 110. The signal processing unit 112 processes the image signal captured by the image capturing unit 11. For example, the signal processing unit 112 encodes the image captured by the image capturing unit 11. The signal processing unit 112 uses, for example, JPEG (Joint Photographic) as an encoding method.
Experts Group) can be used. Alternatively, the signal processing unit 112 uses H.264 as an encoding method. H.264 / MPEG-4 AVC (hereinafter H.264) can be used. Alternatively, the signal processing unit 112 can use HEVC (High Efficiency Video Coding method) as an encoding method. However, the encoding method is not limited to these. Further, the signal processing unit 112 may select a coding method from a plurality of coding methods and perform coding.
The drive control unit 113 controls to change the imaging direction of the imaging unit 11 and the angle of view. In the present embodiment, the case where the imaging unit 11 can change the imaging direction such as the pan direction and the tilt direction and the imaging angle of view can be changed, but the present invention is not limited to this. The imaging device 110 may not have the function of changing the imaging direction in the pan direction and the tilt direction, or may not have the function of changing the angle of view. The communication control unit 114 transmits the captured image processed by the signal processing unit 112 to the client device 120. Furthermore, the communication control unit 114 receives a control command for the imaging device 110 from the client device 120.
The client device 120 includes a control unit 121, a communication control unit 123, an information acquisition unit 124, and a display control unit 125 as a functional configuration. The control unit 121 controls the client device 120. The communication control unit 123 receives the captured image from the client device 120. The information acquisition unit 124 receives a user input from the input device 130 and acquires input information from the input device 130. The display control unit 125 outputs the video to the display device 140, and causes the display device 140 to execute a display process described below.

Next, the flow of processing in this embodiment will be described with reference to FIG. FIG. 3 is a flowchart showing an example of display control processing in the client device 120. The client device 120 repeatedly executes a process of repeatedly acquiring from the imaging device 110 a captured image, that is, a moving image, and presenting to the display device 140 in a distinguishable manner whether or not a single worker is included in the acquired captured image. Let
The control unit 121 acquires a captured image from the imaging device 110 (S200). FIG. 4 is a diagram showing an example of a captured image 300 acquired by the client device 120 from the imaging device 110. In FIG. 4, workers 301 and 302 and workers 303 and 304 represent collaborative workers who work in pairs. On the other hand, the worker 305 represents an independent worker who is working alone. Further, the workbench 306 represents a workbench located at a position far from the imaging device 110. The workbench 307 represents a workbench located near the imaging device 110.
At this time, it is preferable that the position, the mounting angle, and the angle of view of the image capturing unit 11 of the image capturing apparatus 110 are adjusted in advance so that the operator does not overlap the captured images captured by the client apparatus 120 as much as possible. In the present embodiment, the client device 120 processes the captured image acquired from the imaging device 110. As a result, real-time live video can be processed. However, it is not limited to this. For example, the client device 120 may acquire a moving image stored in the image capturing device 110 and perform processing on each frame, or process each frame of the moving image stored in the client device 120. You may go. Alternatively, the client device 120 may access the recording server or the like and perform processing on each frame of the moving image stored in the recording server. Further, in the present embodiment, the group of two co-workers is used, but the number is not limited to this. The client device 120 may have three or more co-workers.

  Next, the control unit 121 performs person detection processing on the captured image 300 acquired in S200 (S201). Since the control unit 121 performs the person detection process, in the present embodiment, first, the captured image 300 is scaled with various sizes. By performing scaling, people of various sizes can be detected. Next, the control unit 121 performs raster scanning on the captured image of each scale using a detection window of a specific size. At this time, the control unit 121 previously calculates the characteristic amount of the person using the learning data, and the difference between the characteristic amount calculated in the detection window at the time of scanning and the characteristic amount based on the learning data is a threshold value. If it is smaller than, it is determined to be a person. For the person detection processing, various methods such as the pattern matching processing can be used. FIG. 5 is a diagram showing an example of an image 400 showing a result of performing person detection on the captured image 300. In FIG. 5A, ellipses 401 to 405 represent positions where the workers 301 to 305 are detected, respectively.

Next, the control unit 121 performs person detection processing on the captured image 300 acquired in S200 (S201). Since the control unit 121 performs the person detection process, in the present embodiment, first, the captured image 300 is scaled with various sizes. By performing scaling, people of various sizes can be detected. Next, the control unit 121 performs raster scanning on the captured image of each scale using a detection window of a specific size. At this time, the control unit 121 previously calculates the characteristic amount of the person using the learning data, and the difference between the characteristic amount calculated in the detection window at the time of scanning and the characteristic amount based on the learning data is a threshold value. If it is smaller than, it is determined to be a person.
FIG. 5 is a diagram showing an example of an image 400 showing a result of performing person detection on the captured image 300. In FIG. 5A, ellipses 401 to 405 represent positions where the workers 301 to 305 are detected, respectively. Further, arrows 406 to 408 are half lines for expressing the direction of the face of each person, with the end side of the arrow as a base point and the head side extending infinitely. Further, as described above, in addition to the detection method using the characteristic amount of the person, as shown in FIG. 5B, at least one of a predetermined uniform, bibs, and hat is set as a rule of the worker. You may decide to wear. That is, the feature amount of the object worn by the person to be detected as the worker may be stored in advance in the memory 21 of the client device 120 as a rule setting file. With this configuration, the control unit 121 determines the uniforms and the like having the feature amount matching the rule based on the rule of the setting file and the feature amount such as the color and shape of the object worn by the person who is the processing target of the captured image. It is also possible to detect a worker who wears it.

ここで、式（２）中のｐｘ１、ｐｘ２、．．．、ｐｘ４、ｐｙ４、及びｑｘ１、ｑｘ２、．．．、ｑｘ４、ｑｙ４は、ユーザーによって指定された撮像画像上の四点のｘ、ｙ座標、及び水平面画像上の四点のｘ、ｙ座標をそれぞれ表す。
次に制御部１２１は、撮像画像上のｘ座標ｐｘ、Ｙ座標ｐｙとホモグラフィ行列Ｈとに対し、以下の式（３）を適用することにより、ｐｘ、ｐｙに対応する水平面画像上のＸ座標ｑｘと、Ｙ座標ｑｙとを計算することができる。

Next, the control unit 121 converts the position of the person detected in S101 into coordinates on the horizontal plane image obtained by projecting the shooting range of the captured image from directly above (S202). In this embodiment, it is assumed that each point on the captured image corresponds to which point on the horizontal plane image. For example, the user operates the input device 130 or the like to specify the four points on the captured image displayed on the display device 140 or the like and the four points on the horizontal plane image. The control unit 121 calculates the homography matrix H in advance based on the four points on the captured image input by the user, the four points on the horizontal plane image, and the equations (1) and (2) shown below. .

Here, px1, px2 ,. ． ． , Px4, py4, and qx1, qx2 ,. ． ． , Qx4, qy4 represent the x and y coordinates of four points on the captured image specified by the user and the x and y coordinates of the four points on the horizontal plane image, respectively.
Next, the control unit 121 applies the following equation (3) to the x-coordinate px, the Y-coordinate py and the homography matrix H on the captured image, and thereby X on the horizontal plane image corresponding to px and py. The coordinate qx and the Y coordinate qy can be calculated.

  FIG. 6 is a diagram showing an example of a horizontal plane image 500 of the work site. In FIG. 6, coordinates 501 to 505 represent results obtained by converting the representative points of the ellipses 401 to 405, which are person detection positions, into coordinates on the horizontal plane image. In the present embodiment, the representative point is, for example, the center of the person detection position. The representative point may be the center of the area corresponding to the head of the person. Areas 506 and 507 represent areas when the work benches 306 and 307 are represented on a horizontal plane image, respectively. In the present embodiment, a sketch of a work site created by CAD or the like is used as the horizontal plane image, but the present invention is not limited to this. For example, an image obtained by transforming the captured image may be used, or a handwritten image may be used. Further, it is not necessary to have an image, and the program may only have the above-mentioned coordinate information. That is, the horizontal plane image 500 does not have to be generated, and only the coordinate position on the horizontal plane image 500 may be calculated. If the image is taken with a composition close to the horizontal plane image 500 (a bird's-eye view or the like), the process of S203 does not necessarily have to be performed. In this case, the distance on the captured image can be used to determine whether the worker is a single worker or a collaborative worker. Further, as the method of generating the horizontal plane image 500, various other methods may be used.

Next, the control unit 121 updates the list of detected persons (S203). If it is the beginning of the repetitive processing, the control unit 121 lists all the detected persons. At this time, the control unit 121 stores the coordinates of the representative point on the captured image where the person is detected and the coordinates of the representative point on the horizontal plane image in each node in the list. If it is the second and subsequent times of the repeated processing, the control unit 121 performs the following processing for all the detected persons.
(Processing 1). The control unit 121 calculates the distance between the coordinates on the captured image of the person stored in each node in the list and the coordinates on the captured image of the person. Replace with data. At this time, the control unit 121 saves the history of coordinates previously saved in the node.
(Processing 2). The control unit 121 adds its own data to the end of the list if there is no node satisfying the conditions in (Process 1).
In addition, the control unit 121 performs the following processing after finishing the above processing for all the detected persons.
(Processing 3). The control unit 121 deletes the node that has not been updated.

Next, the control unit 121 determines whether or not a single worker exists in the captured image (S204 to S206). In the present embodiment, as shown in FIG. 7A, the control unit 121 sets the work area frames (501 to 506) in the captured image display based on a user operation or the like. The processing flow is as follows.
(Processing 4). The control unit 121 confirms whether the coordinates on the captured image of the person to be processed are included in the work area frame in order from the head of the list, and if the coordinates are included in the work area frame, the work frame is added to the list. Stores a flag indicating that the entry is in. At this time, when there are a plurality of work areas, the control unit 121 stores a unique ID number assigned to each work area frame in the list (S204).
(Processing 5). The control unit 121 searches the list and confirms, for each work area frame, how many workers belong to the area (S205). When the number of workers in the work area frame is 1, the control unit 121 determines that the single work is performed in the area, and otherwise determines that the single work is not performed (S206). . That is, when the number of workers in the work area frame is 1, the control unit 121 determines that the worker in the area is an independent worker who cannot be associated with another person. When the number of workers in the work area frame is two or more, the control unit 121 associates the workers in the area with each other and determines that they are joint independent workers. In this way, the control unit 121 has a function as an association unit.
In the present embodiment, the control unit 121 first determines whether or not each person is within the work area frame based on each work area frame, although it is determined based on the detected list of persons. It is also possible to check and make a decision based only on the number of people in the work area.
Further, although the description has been given assuming that the work area frame is set in the captured image display, the control unit 121 sets the work area frame in the horizontal plane image based on the user operation as shown in FIG. 7B. May be.

Here, the process of determining whether the worker is a single worker or a collaborative worker is not limited to the above-described process. For example, the control unit 121 associates the persons with each other based on at least one of the position of the person detected from the captured image, the orientation of the face of the person, and the distance between the persons, and determines that the person is a collaborator. It may be determined. For example, the control unit 121 may associate people who are located within a predetermined range from a predetermined object in the captured image, such as a work table, or may estimate the face direction (or line of sight) of the person, Persons whose face directions (or eyes) intersect may be associated with each other. In addition, for example, the control unit 121 may associate persons whose distances between persons are within a predetermined distance. Then, the control unit 121 may determine persons who are associated with other persons as collaborators and persons who are not associated with other persons as independent workers.
Further, the control unit 121 may further associate the persons with each other based on the color or shape of the person's uniform, bibs, or hat. For example, the control unit 121 associates persons having the same uniform shape among persons whose distance between persons is within a predetermined distance, and even if the distance between persons is within a predetermined distance, the shape of the uniform People with different names may not be associated with each other. In this way, the control unit 121 may associate the person based on the feature amount of the object worn by the person.
In addition, even if there are a plurality of workers in the work area frame, the control unit 121 may change the face orientation of the plurality of workers in the work area frame or the distance between the workers. Alternatively, it may be determined that the operator is an independent worker.

Next, the control unit 121 determines whether or not a single work is being performed, and if the conditions are satisfied, records the log as a single work and notifies it (S207 to S209). The processing flow is as follows.
(Process 6). When there is a work area frame in which independent work is performed in (Process 5), the control unit 121 increments the cumulative number of independent work frames, that is, increases the value of the variable by 1, and at the same time, A person belonging to the work area is extracted from the list (S207). If the independent work is not performed, the control unit 121 sets the cumulative number of independent work frames to 0 (S208).
(Process 7). When the cumulative number of independent work frames exceeds the threshold value, the control unit 121 determines that the individual work is in progress (S209). By using this determination condition, it is possible to prevent erroneous notification of being an independent worker due to a momentary failure to detect a person even though the worker is a collaborative worker.
When the control unit 121 determines that the single work is performed, the display unit 140 displays the occurrence time and the duration of the work, the imaging device 110 in which the work is occurring, and the place where the single work is performed in the captured image, for example. Is displayed on the screen (S210). The screen of FIG. 8A is an example of a screen showing a list of moving images in the time zone in which the independent work is detected. An example of information in which the text information “individual work detection list” in FIG. 8A, time, duration, and location information can be identified as a moving image including a person who is not associated with another person Is. It should be noted that, depending on the use, moving images of time bodies that do not include persons who are not associated with other persons may be displayed as a list. In that case, for example, the text information “Collaboration detection list” is displayed.
Next, the control unit 121 determines whether or not to repeat the processing from acquisition of the captured image (S200) to determination of whether or not to notify the user of the presence or absence of the sole supplier (S210, S211) (S212). The control unit 121 determines to end the iterative processing when acquisition of a captured image from the imaging device becomes impossible or when there is an instruction from the user to suspend the iterative processing.

In the display in S210, the control unit 121 displays the information (list) about the independent work together with the thumbnail image as shown in FIG. 8A, but may display only the list format (list). Further, the control unit 121 may display the recorded video together with the timeline display as shown in FIG. 8B, and may display the time zone of the independent work on the timeline. The timeline portion (black-painted or shaded portion) shown as the time zone of the independent work in FIG. 8B is an example of information that can identify that a person who is not associated with another person is included. . In addition, for example, a black-painted portion is a time zone in which there is one independent worker, and a shaded portion is a time zone in which there are a plurality of independent workers. Further, "A-1" is information indicating an ID or the like for identifying the image pickup device. In the example of FIG. 8B, the time lines in the two imaging devices “A-1” and “B-1” are shown.
On the other hand, it is possible to identify that the portion of the timeline (white portion) represented as a time zone other than the time zone of the independent work in FIG. 8B does not include a person who is not associated with another person. It is an example of information. Further, the control unit 121 may display the time zone of the independent work detected by a plurality of cameras on one timeline. As described above, at least information that can be identified on the timeline as including a person who is not associated with another person and information that can be identified as not including a person that is not associated with another person It suffices to display at least one of the above. It suffices that the time period in which a person not associated with another person is included in the moving image and the time period in which a person not associated with another person is not included can be identified.
Further, for example, as shown in FIG. 9A, the user specifies the display of each individual work with a mouse, a finger, or the like. Then, the control unit 121 reproduces the recorded image of the corresponding imaging device on the display device 140 via the display control unit 125 and displays the captured image at that time, as shown in FIG. 9B. You may be able to confirm. By doing so, user convenience is enhanced. At this time, the control unit 121 may reproduce only the recorded video when the independent work occurs, but the independent work may be reproduced from a video which is a predetermined time (for example, several tens of seconds) before the occurrence time, or the independent work is continued. Playback for longer than the time is more preferable because the situation before and after that can be easily confirmed.
Further, the control unit 121 may connect the recorded images in the time zones detected as the independent work and reproduce them as a digest image. That is, the moving images in the list shown in FIG. 9A may be continuously reproduced. At that time, the moving image in the time zone of the independent work of about several seconds may not be included in the digest video.
In order to prevent erroneous detection and the like, the notification of the independent work issues an alarm when the independent work continues for a predetermined time, but the duration of the independent work is the time from the detected first frame. Therefore, the control unit 121 may set the section displayed as an independent work on the timeline or the start of recording or reproduction not to the time when the alarm is issued but to the time when the independent work is started to be detected.

In addition, the control unit 121 may display the timeline in the following manner, not only in the single work time zone when the cumulative number of frames in which the single worker exists exceeds the threshold and an alarm is issued. That is, the control unit 121 confirms both of them by displaying them in a color, shape, or pattern different from that of the independent work time zone when the alarm is issued even when it is determined that the work is an independent work for a short period of time when the alarm is not issued. You may make it possible. In this case, since there is a possibility that one of the collaborators may be temporarily hidden by the obstacle in the independent work detected only for a moment, two thresholds are used for the determination of the independent work. Should be provided. At this time, the control unit 121 issues an alarm based on the threshold having the larger value of the two thresholds. For example, the first threshold value is used in the information that can identify that a person who is not associated with another person is included. Then, a second threshold value larger than the first threshold value is used to determine whether to give an alarm.
In addition, the control unit 121 may save a list of image pickup devices and areas in which independent work is detected and time information as a CSV file, and paste a list with thumbnail images, a screenshot of recorded video, etc. as a report. It may be possible to print.
In addition to the captured image for detecting the independent work, the control unit 121 also performs the following if there is an imaging device that similarly captures the area from another angle or captures the peripheral area. You may make it process. That is, the control unit 121 may display a video image that has been captured and recorded by the image capturing apparatus in which the association setting has been performed in advance.
Further, as shown in FIG. 10 (A), in the case of a system that detects an independent work from a live image and, if it is judged as the independent work, issues a real-time alarm, a work supervisor checks the current situation. , It is necessary to check the past situation. Therefore, the control unit 121 may perform control so as to display both the live video and the recorded video. In FIG. 10 (A), in the image of the room A, by changing the index in the upper right, information that can identify that a person who is not associated with another person is included is displayed.
As shown in FIG. 10B, the control unit 121 may delete the selected image or the like from the series of displays when the work monitor determines that there is no false alarm or problem.

In the present embodiment, the control unit 121 has mainly been described using an example of setting the work area frame. When detecting the independent work, the control unit 121 may superimpose the work area frame at the time of detecting the independent work on the image and display the work on the display device 140. In addition, the control unit 121 displays all work area frames by superimposing them on the image, and when the independent work is performed, the work area frame in which the independent work is performed has a different color from other work area frames. It may be displayed explicitly or emphasized by changing to, blinking, or the like. By doing so, the work supervisor can easily recognize whether the person in the video is single or plural.
Further, in the present embodiment, an example in which a person or the like is detected and displayed by one client device has been described, but the processing may be performed by a plurality of client devices. For example, a first client device for detecting and notifying an independent work or the like when a video is input, and a second client device for outputting an alarm in response to the notification, displaying a list, or displaying on a timeline. You may make it process with a client apparatus of. An example of the first client device is a VCA (Video Content Analysis) device. An example of the second client device is a VMS (Video Management System) device. The hardware configuration of the first client device and the second client device is similar to the hardware configuration of the client device 120 of FIG. The function of each client device is realized by the CPU of each client device executing processing based on the program stored in the memory of each client device.

  FIG. 11A shows an example of the case where the system of this embodiment and the entrance / exit management system 160 cooperate with each other. In the entry / exit management system, the time when the worker enters and leaves the work room is recorded. Therefore, as shown in FIG. 11B, the control unit 121 acquires information on the time of entry and exit for each worker from the room entry / exit management system 160, and the work entering the work room based on the acquired information. Extract the person. Then, the control unit 121 may record the extracted list of the worker names as a part of the information of the occurrence list of the independent work or display it on the screen. However, the present invention is not limited to this, and the control unit 121 may send an email to the user. This recording and display may be realized by the system of the present embodiment, or may be incorporated in the entry / exit management system or the like. The list of extracted worker names is an example of entry / exit information.

<Embodiment 2>
In the present embodiment, the configuration is the same as that of the first embodiment, and a description will be given using an example in which the determination as to whether or not there is an independent worker is performed by distance instead of detection by the work area frame. The process flow of this embodiment will be described with reference to FIG.
The process from the control unit 121 acquiring the captured image from the image capturing apparatus 110 (S1101) to updating the detected person list (S1103) is the same as the process described in the first embodiment.
Next, the control unit 121 determines whether or not a single worker exists in the captured image (S1104 to S1106). The processing flow is as follows.
(Processing 1). The control unit 121 sets the first person in the list as a processing target (S1104).
(Processing 2). The control unit 121 determines whether the person to be processed is an independent worker (S1105). In the present embodiment, the control unit 121 confirms the following items with respect to the person who is the processing target in order to determine whether or not the worker is an independent worker.

(Item 1). The control unit 121 determines whether or not there is a detected person within a radius R centered on the coordinates on the horizontal plane image of the processing target person.
This radius R may be set appropriately according to the situation to which this embodiment is applied. For example, when monitoring indoor work, the radius R can be a distance corresponding to 2 m. However, at this time, the control unit 121 refers to the coordinate history stored in the node of the processing target person, determines that the processing target person is moving, and determines that the processing target person is not moving. The value of R, which is a threshold value, is increased. At this time, the control unit 121 may use an ellipse obtained by contracting a circle having a radius R in the y-axis direction if the collaborators work side by side in the x-axis direction on the horizontal plane image. Similarly, the control unit 121 may use an ellipse obtained by contracting a circle having a radius R in the x-axis direction if the collaborators work side by side in the y-axis direction on the horizontal plane image.
(Item 2). When there is a person who satisfies (Item 1), the control unit 121 determines whether the person exists in the detection area (work area frame).
In the present embodiment, the control unit 121 operates the input device 130 or the like to set an area designated by the user as a detection area in which the worker exists. This can prevent erroneous detection of a non-worker moving on the other side of a glass wall and a person shown on a poster or the like placed on a desk as a co-worker. However, the setting of the detection area is not limited to the input by the user via the input device 130 or the like. The control unit 121 may automatically set the area surrounded by the markers arranged on the captured image as the detection area. As a method of detecting using a marker, for example, a two-dimensional marker installed in the four corners of the detection area is registered in the client device 120 in advance by raster-scanning a captured image, and There is a method of pattern matching with information.
When all of the above (Item 1) and (Item 2) are satisfied, the control unit 121 determines that the worker is a collaborative worker, and proceeds to (Process 3-1). That is, the control unit 121 associates the workers who satisfy all of the above (Item 1) and (Item 2) with each other, and determines that they are joint independent workers. When at least one is not satisfied, the control unit 121 determines that the person who is the processing target is an independent worker, and proceeds to (Processing 3-2).

  Here, the process of determining whether the worker is a single worker or a collaborative worker is not limited to the above-described process. For example, the control unit 121 may associate the persons with each other based on at least one of the position of the person detected from the captured image, the orientation of the face of the person, and the distance between the persons. For example, the control unit 121 may associate people who are located in a predetermined range from a predetermined object in the captured image, such as a workbench, or may estimate the face direction (or line of sight) of the person, Persons whose face directions (or eyes) intersect may be associated with each other. In addition, for example, the control unit 121 may associate persons whose distances between persons are within a predetermined distance. Then, the control unit 121 may determine persons who are associated with other persons as collaborators and persons who are not associated with other persons as independent workers. In addition, the control unit 121 may further associate the persons with each other based on the color or shape of the person's uniform, bibs (vest-like clothes), or hat. For example, the control unit 121 associates persons having the same uniform shape among persons whose distance between persons is within a predetermined distance, and even if the distance between persons is within a predetermined distance, the shape of the uniform Different persons may not be associated with each other, or a combination within a predetermined distance and a different color of bibs may be determined as a collaborative worker, and other combinations may be determined as a single worker.

式（４）中のｑｘ、ｑｙはそれぞれ、処理対象としている人物の、水平面画像上でのｘ座標、及びｙ座標を表す。図１３では、ｑｘ、ｑｙはそれぞれ楕円１２０２のｘ座標、及びｙ座標を表す。式（４）中のａ、ｂは楕円の変換パラメータである。例えば、ａ＝１、ｂ＝１とすると、楕円は半径Ｒ、中心（ｑｘ、ｑｙ）の円となる。また、ａ＝１、ｂ＝１／２とすると、ｘ軸方向の長さがＲ、ｙ軸方向の長さがＲ／２となる楕円となる。本実施形態では共同作業者同士が水平面画像上のｘ軸方向に並んで作業を行うため、ａ＞ｂとなる値を用いることとする。 The determination in (Item 1) and (Item 2) will be described with reference to FIG. In FIG. 13, ellipses 1201 to 1205 represent results obtained by converting the centers of the ellipses 401 to 405, which are human detection positions, into coordinates on the horizontal plane image. A region 1206 represents a detection region designated by the user. Further, an ellipse 1200 represents an ellipse composed of x and y that satisfies the following expression (4).

Qx and qy in the equation (4) respectively represent the x coordinate and the y coordinate on the horizontal plane image of the person to be processed. In FIG. 13, qx and qy represent the x coordinate and the y coordinate of the ellipse 1202, respectively. In the equation (4), a and b are elliptic conversion parameters. For example, if a = 1 and b = 1, the ellipse is a circle with radius R and center (qx, qy). Further, when a = 1 and b = 1/2, an ellipse having a length R in the x-axis direction and a length R / 2 in the y-axis direction is obtained. In the present embodiment, since the co-workers perform work side by side in the x-axis direction on the horizontal plane image, a value that satisfies a> b is used.

For example, it is assumed that the control unit 121 sets the person on the ellipse 402 at the person detection position as the processing target during scanning of the list. At this time, the control unit 121 makes the above (item 1) to (item 2) determination regarding the coordinates of the ellipse 1202 on the horizontal plane image corresponding to the ellipse 402 at the person detection position. First, in (item 1), the control unit 121 determines whether or not the detected person exists inside the ellipse 1200. In the situation shown in FIG. 13, the coordinates of the ellipse 1201 on the horizontal plane image corresponding to the ellipse 401 at the person detection position are included in the ellipse, so the control unit 121 determines that the determination is satisfied. Next, in (Item 2), the control unit 121 determines whether or not the coordinates 601 on the horizontal plane image corresponding to the ellipse 401 of the detection position of the person who satisfies the condition of (Item 1) are included in the region 1206. . Under the situation described in the example of the present embodiment, the coordinates of the ellipse 1201 are included in the area 1206, so the control unit 121 determines that the determination is satisfied. As described above, since (Item 1) and (Item 2) are all satisfied, the control unit 121 determines that the person with the ellipse 402 is a collaborator.

(Process 3-1). The control unit 121 determines whether the person at the end of the list is the processing target (S1106). If it is not at the end of the list, the control unit 121 sets the next person to be processed, and returns to (Process 2) (S1107). If it is the end of the list, the control unit 121 ends scanning in the list and sets the cumulative number of frames in which an independent worker exists to 0 (S1108).
(Processing 3-2). The control unit 121 increments the cumulative number of frames in which an independent worker exists (S1109).
Next, the control unit 121 determines whether or not to notify the user that there is an independent worker, and if the conditions are satisfied, the controller 121 notifies the user (S1110, S1111).
The control unit 121 performs the same processing as in the first embodiment regarding the display processing and the like at the time of notification.
Further, in the present embodiment, the control unit 121 displays a UI indicating that an independent worker exists on the display device 140 as a method of notifying the user that the independent worker exists. For example, the control unit 121 may superimpose a double circle or a red circle on the position of an independent operator on the captured image 300 or the horizontal plane image 500. However, the present invention is not limited to this, and the control unit 121 may send an email to the user, output to an external output, and issue an alert from a speaker.

<Other embodiments>
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium. Then, it can be realized by a process in which one or more processors in the computer of the system or apparatus read and execute the program. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Although the preferred embodiments of the present invention have been described in detail above, the present invention is not limited to the specific embodiments. The functional configuration of the image capturing apparatus 110 may be implemented in the image capturing apparatus as a software configuration realized by the CPU 12 executing processing based on a program stored in the memory 13. Further, part or all of the functional configuration of the image capturing apparatus 110 may be implemented in the image capturing apparatus 110 as a hardware configuration. Similarly, the functional configuration of the client device 120 may be implemented in the client device 120 as a software configuration realized by the CPU 22 executing processing based on a program stored in the memory 21. Further, a part or all of the functional configuration of the client device 120 may be implemented in the client device 120 as a hardware configuration.
In addition to superimposing a double circle and a red circle on the position of an independent worker, the character string “Independent worker” may be superimposed on the independent worker, and a warning color such as red may be displayed. You may make it superimpose on a single operator on a captured image. Also, these may be combined arbitrarily and displayed in a superimposed manner.
The hardware configurations of the client device and the imaging device described above are examples, and for example, a plurality of CPUs, memories, communication I / Fs, and the like may be included.

As described above, according to the processes of the above-described embodiments, it is possible to appropriately present whether the person in the image is a single person or a plurality of persons. By receiving the notification that there is a single worker in the work site and easily confirming the video, the work supervisor can find the worker who violates the rule and pay attention to the worker. It becomes possible to encourage.

21 memory 22 CPU
120 client device 140 display device

The present invention relates to a control device , a control method, and a program for detecting a person from a moving image and displaying it on a display device.

Therefore, the control device of the present invention, a detection means for detecting a person in the area where the person works ,
The specifying means for specifying whether the person detected by the detecting means is a person who is working independently, and the person detected by the detecting means in the area by the specifying means, is working independently. Controlled so that the user is notified that there is a person working alone in the area where multiple people must work, provided that it is specified that the person is working. And a notification control means for performing.

Claims (11)

Detection means for detecting a person from a moving image,
Based on at least one of the position of the person detected by the detection means, the orientation of the face of the person, and the distance between the persons, association means for associating the persons with each other,
At least information that can be identified by the associating unit to include a person who is not associated with another person, and information that can be identified by the associating unit not to include a person that is not associated with another person A display control device for displaying any one on a display device.   The information capable of identifying that a person who is not associated with another person is included is information that indicates a time zone in which a person who is not associated with another person on the timeline is included. The display control device according to 1.   The information capable of identifying that a person who is not associated with another person is not included is information that indicates a time period when a person who is not associated with another person on the timeline is not included. Item 3. The display control device according to item 1 or 2.   The information that can be identified as including a person who is not associated with the other person is at least any one of time, duration, and location information of the person including the moving image including the person. The display control device according to any one of claims 1 to 3. Further comprising an acquisition unit for acquiring a moving image from the imaging device,
The display control device according to any one of claims 1 to 4, wherein the detection unit detects a person from the moving image acquired by the acquisition unit. Further comprising an acquisition means for acquiring the entry and exit information of the person from the entry and exit management system,
The display control device according to any one of claims 1 to 5, wherein the display control unit further displays the entry / exit information on the display device.   7. The display control device according to claim 1, wherein the detection unit detects a person from the moving image based on a characteristic amount of the person.   8. The display control device according to claim 1, wherein the associating unit associates the persons with each other based on the feature amount of the object worn by the person.   The display control device according to claim 8, wherein the person wears at least one of a uniform, bibs, and a hat. A display control method executed by a display control device, comprising:
A detection step of detecting a person from a moving image,
An associating step of associating the persons based on at least one of the position of the person detected by the detection step, the orientation of the face of the person, and the distance between the persons,
At least information that can be identified as including a person who is not associated with another person by the associating step and information that can be identified as not including a person that is not associated with another person by the associating step A display control step of displaying any of them on a display device.   A program for causing a computer to function as each unit of the display control device according to claim 1.

Images