JP6987902B2 - Controls, control methods and programs - Google Patents

Description

The present invention relates to a control device , a control method, and a program for detecting a person from a moving image and displaying it on a display device.

Conventionally, a method of detecting a person from an image is known. Patent Document 1 discloses a method of detecting a person shielded from an image.

Japanese Unexamined Patent Publication No. 2014-199506

In the technique disclosed in Patent Document 1 described above, it is possible to inform the position of a person in an image. For example, in the above technique, a person existing in an image can be circled and the position of the person can be presented to the user. However, by simply indicating the position of a person based on the above-mentioned technique, the user is whether the detected person is isolated from the group and working alone, or is the user collaborating with another person in a close position. There was a problem that could not be judged at a glance. For example, in the field where data that should be prevented from leaking is handled, there may be a rule that prohibits working alone, and a work observer monitors the workers via an image pickup device. You may want to call attention to a single worker who is isolated from the group and is working alone. However, by looking at the captured image alone, the work observer cannot determine at a glance whether each of the workers in the captured image is working alone or in multiples. In addition, it is a heavy burden for the work observer to constantly monitor the worker.
An object of the present invention is to appropriately present whether a person in an image is single or plural.

Therefore, the control device of the present invention comprises a detection means for detecting a person in an area where a person works.
A specific means for identifying whether the person detected by the detection means is a person who is working alone, and a person detected by the detection means in the area by the specific means perform the work independently. Control to notify the user of recognizable information that there is a person working alone in the area where multiple people must work, provided that the person is identified as a person. It is characterized by having a notification control means for the user.

According to the present invention, it is an object of the present invention to appropriately present whether a person in an image is a single person or a plurality of people.

It is a figure which shows an example of the system configuration and the hardware configuration of the system. It is a figure which shows an example of the functional structure of an image pickup apparatus and a client apparatus. It is a flowchart which shows an example of the processing of a notification control. It is a figure which shows an example of the captured image. It is a figure which shows an example of the image which shows the result of performing a person detection. It is a figure which shows an example of a horizontal plane image. It is a figure which shows an example of a work area frame. It is a figure which shows an example of the display screen. It is a figure which shows an example of the display screen. It is a figure which shows an example of the display screen. It is a figure which shows an example which cooperated with the entrance / exit management system. It is a flowchart which shows an example of the processing of a notification control. It is a figure explaining an example of determination.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<Embodiment 1>
In the present embodiment, a single worker (a person who cannot be associated with another person) is used as a work observer at a site where data for which leakage should be prevented, which is obligatory for a worker to work with two or more people, is handled. ) Is included, or an image that includes a collaborator (a person associated with another person) is described. Such a situation is an example of a situation in which the present embodiment can be used, and can be used in various situations in which it is desired to determine whether a person is a single person or a plurality of people. For example, the present invention can also be used in situations where it is determined whether a passerby, a visitor, etc. is a single person or a group, and the present invention is used for marketing, solving social problems, and the like.
FIG. 1 shows an example of the system configuration and the hardware configuration of the system in this embodiment. In FIG. 1, the image pickup apparatus 110 performs imaging. The client device 120 drives the image pickup device 110 and displays the captured image captured by the image pickup device 110. The input device 130 is composed of a mouse, a keyboard, and the like, and performs user input to the client device 120. The display device 140 is composed of a display or the like, and displays an image output by the client device 120. Although the client device 120 and the display device 140 are shown as independent devices in FIG. 1, the client device 120 and the display device 140 may be integrated and configured. Further, the input device 130 and the display device 140 may be integrated, or the client device 120, the input device 130, and the display device 140 may be integrated. The network 150 connects the image pickup device 110 and the client device 120. The network 150 is composed of a plurality of routers, switches, cables, etc. that meet communication standards such as a local network. In the present embodiment, the communication standard, scale, and configuration are not limited as long as the image pickup device and the client can communicate with each other. For example, the network 150 may be configured by the Internet, a wired LAN (Local Area Network), a wireless LAN (Wireless LAN), a WAN (Wide Area Network), or the like. Further, the number of image pickup devices connected to the client device 120 is not limited to one, and may be a plurality of.

The hardware configuration of the image pickup apparatus 110 will be described. The image pickup apparatus 110 includes an image pickup unit 11, a CPU 12, a memory 13, and a communication I / F 14 as a hardware configuration. The image pickup unit 11 has an image pickup element and an engineering system for a subject on the image pickup element, and under the control of the CPU 12, an image pickup is performed on the image pickup element with the intersection of the optical axis of the optical system and the image pickup element as the image pickup center. The image pickup device is an image pickup device such as a CMOS (Complementary Metal-Oxide Semiconductor) or a CCD (Charged Coupled Device). The CPU 12 controls the entire image pickup apparatus 110. The memory 13 stores a program, an image captured by the image pickup unit 11, data used when the CPU 12 executes processing, and the like. The communication I / F 14 controls communication with the client device 120 via the network 150 based on the control of the CPU 12. The function of the image pickup apparatus 110 and the like are realized by the CPU 12 executing the process based on the program stored in the memory 13. A processor other than the CPU may be used.
The hardware configuration of the client device 120 will be described. The client device 120 includes a memory 21, a CPU 22, a communication I / F23, an input I / F24, and a display I / F25 as hardware configurations. The CPU 22 controls the entire client device 120. The memory 21 stores a program, a captured image transmitted from the imaging device 110, data used when the CPU 22 executes processing, and the like. The communication I / F 23 controls communication with the image pickup apparatus 110 via the network 150 based on the control of the CPU 22. The input I / F 24 connects the client device 120 and the input device 130, and controls the input of information from the input device 130. The display I / F 25 connects the client device 120 and the display device 140, and controls the output of information to the display device 140. By executing the process based on the program stored in the memory 21, the CPU 22 realizes the function of the client device 120 and the process of the flowcharts of FIGS. 3 and 12 described later.

FIG. 2 is a diagram showing an example of the functional configuration of the image pickup apparatus 110 and the client apparatus 120. The image pickup apparatus 110 includes a control unit 111, a signal processing unit 112, a drive control unit 113, and a communication control unit 114 as functional configurations. The control unit 111 controls the image pickup apparatus 110. The signal processing unit 112 processes the image signal captured by the image pickup unit 11. For example, the signal processing unit 112 encodes the image captured by the image pickup unit 11. The signal processing unit 112 has, as a coding method, for example, JPEG (Joint Photographic).
Experts Group) can be used. Alternatively, the signal processing unit 112 uses the coding method of H.I. 264 / MPEG-4 AVC (hereinafter referred to as H.264) can be used. Alternatively, the signal processing unit 112 can use HEVC (High Efficiency Video Coding coding method) as the coding method. However, the coding method is not limited to these. Further, the signal processing unit 112 may select a coding method from a plurality of coding methods to perform coding.
The drive control unit 113 controls to change the image pickup direction and the angle of view of the image pickup unit 11. In the present embodiment, the case where the image pickup unit 11 can change the image pickup direction in the pan direction and the tilt direction and change the image pickup angle of view will be described, but the present invention is not limited to this. The image pickup apparatus 110 may not have a function of changing the image pickup direction in the pan direction and the tilt direction, and may not have a function of changing the angle of view. The communication control unit 114 transmits the captured image processed by the signal processing unit 112 to the client device 120. Further, the communication control unit 114 receives a control command from the client device 120 to the image pickup device 110.
The client device 120 includes a control unit 121, a communication control unit 123, an information acquisition unit 124, and a display control unit 125 as functional configurations. The control unit 121 controls the client device 120. The communication control unit 123 receives the captured image from the client device 120. The information acquisition unit 124 accepts user input by the input device 130 and acquires input information from the input device 130. The display control unit 125 outputs an image to the display device 140, and causes the display device 140 to execute a display process described later.

Next, the flow of processing in this embodiment will be described with reference to FIG. FIG. 3 is a flowchart showing an example of display control processing in the client device 120. The client device 120 repeatedly acquires an image taken from the image pickup device 110, that is, a moving image, and executes a process of presenting to the display device 140 identifiable whether or not a single operator is included in the acquired captured image. Let me.
The control unit 121 acquires an captured image from the imaging device 110 (S200). FIG. 4 is a diagram showing an example of an image captured image 300 acquired by the client device 120 from the image pickup device 110. In FIG. 4, the workers 301 and 302 and the workers 303 and 304 represent collaborative workers who are working in pairs. On the other hand, the worker 305 represents a single worker who is working alone. Further, the workbench 306 represents a workbench located at a position far from the image pickup apparatus 110. The workbench 307 represents a workbench located near the image pickup apparatus 110.
At this time, it is preferable to adjust the position, attachment angle, and angle of view of the image pickup unit 11 of the image pickup device 110 in advance so that the captured images acquired by the client device 120 do not overlap as much as possible. In the present embodiment, the client device 120 processes the captured image acquired from the image pickup device 110. This makes it possible to process real-time live video. However, it is not limited to this. For example, the client device 120 may acquire a moving image stored in the image pickup device 110 and perform processing on each frame, or may process each frame of the moving image stored in the client device 120. May be done. Alternatively, the client device 120 may access the recording server or the like and perform processing on each frame of the moving image stored in the recording server. Further, in the present embodiment, the collaborators are set as a pair, but the present invention is not limited to this. The client device 120 may have three or more collaborators.

Next, the control unit 121 performs a person detection process on the captured image 300 acquired in S200 (S201). In order to perform the person detection process, the control unit 121 first scales the captured image 300 in various sizes in the present embodiment. By scaling, people of various sizes can be detected. Next, the control unit 121 raster scans the captured image of each scale using a detection window of a specific size. At this time, the control unit 121 calculates the feature amount of the person using the learning data in advance, and the error between the feature amount calculated in the detection window at the time of scanning and the feature amount based on the above-mentioned learning data is a threshold. If it is smaller than, it is judged to be a person. For the person detection process, various methods such as such pattern matching process can be used. FIG. 5 is a diagram showing an example of an image 400 showing the result of performing person detection on the captured image 300. In (A) of FIG. 5, the ellipses 401 to 405 represent the positions where the workers 301 to 305 are detected, respectively.

Next, the control unit 121 performs a person detection process on the captured image 300 acquired in S200 (S201). In order to perform the person detection process, the control unit 121 first scales the captured image 300 in various sizes in the present embodiment. By scaling, people of various sizes can be detected. Next, the control unit 121 raster scans the captured image of each scale using a detection window of a specific size. At this time, the control unit 121 calculates the feature amount of the person using the learning data in advance, and the error between the feature amount calculated in the detection window at the time of scanning and the feature amount based on the above-mentioned learning data is a threshold. If it is smaller than, it is judged to be a person.
FIG. 5 is a diagram showing an example of an image 400 showing the result of performing person detection on the captured image 300. In (A) of FIG. 5, the ellipses 401 to 405 represent the positions where the workers 301 to 305 are detected, respectively. Further, the arrows 406 to 408 are half-lines for expressing the direction of each person's face, with the tail end side of the arrow as the base point and the head side extending infinitely. Further, as described above, in addition to the detection method using the feature amount of the person, as shown in FIG. 5 (B), at least one of the predetermined uniforms, bibs, and hats is set as the worker's rule. You may decide to wear a hat. That is, the feature amount of the object worn by the person to be detected as a worker may be stored in the memory 21 or the like of the client device 120 as a setting file in advance with rules. In this way, the control unit 121 can obtain a uniform or the like having a feature amount that matches the rule based on the rule of the setting file and the feature amount such as the color and shape of the object worn by the person to be processed of the captured image. It is also possible to detect the worn worker.

ここで、式（２）中のｐｘ１、ｐｘ２、．．．、ｐｘ４、ｐｙ４、及びｑｘ１、ｑｘ２、．．．、ｑｘ４、ｑｙ４は、ユーザーによって指定された撮像画像上の四点のｘ、ｙ座標、及び水平面画像上の四点のｘ、ｙ座標をそれぞれ表す。
次に制御部１２１は、撮像画像上のｘ座標ｐｘ、Ｙ座標ｐｙとホモグラフィ行列Ｈとに対し、以下の式（３）を適用することにより、ｐｘ、ｐｙに対応する水平面画像上のＸ座標ｑｘと、Ｙ座標ｑｙとを計算することができる。

Next, the control unit 121 converts the position of the person detected in S101 into the coordinates on the horizontal plane image projected from directly above the shooting range of the captured image (S202). In the present embodiment, it is assumed that each point on the captured image corresponds to which point on the horizontal plane image in advance. For example, the user operates the input device 130 or the like to specify the four points on the captured image displayed on the display device 140 or the like and the four points on the horizontal plane image. The control unit 121 calculates the homography matrix H in advance based on the four points on the captured image input by the user, the four points on the horizontal plane image, and the following equations (1) and (2). ..

Here, px1, px2 ,. .. .. , Px4, py4, and qx1, qx2 ,. .. .. , Qx4, qy4 represent the x, y coordinates of the four points on the captured image specified by the user, and the x, y coordinates of the four points on the horizontal plane image, respectively.
Next, the control unit 121 applies the following equation (3) to the x-coordinate px, Y-coordinate py and the homography matrix H on the captured image, so that the X on the horizontal plane image corresponding to px and py The coordinate qx and the Y coordinate qy can be calculated.

FIG. 6 is a diagram showing an example of a horizontal plane image 500 of a work site. In FIG. 6, the coordinates 501 to 505 represent the results of converting the representative points of the ellipses 401 to 405, which are the person detection positions, into the coordinates on the horizontal plane image. In this embodiment, the representative point is the center of the person detection position as an example. The representative point may be the center of the area corresponding to the head of the person. Further, the areas 506 and 507 represent areas when the workbench 306 and 307 are represented on a horizontal plane image, respectively. In the present embodiment, it is assumed that a sketch of the work site created by CAD or the like is used as a horizontal plane image, but this is not the case. For example, a deformed image of the captured image may be used, or a handwritten image may be used. Further, it is not necessary to have an image, and it is sufficient to have the above-mentioned coordinate information inside the program. That is, the horizontal plane image 500 does not have to be generated, and only the coordinate position on the horizontal plane image 500 may be calculated. Further, if the image is taken with a composition (overhead view or the like) close to the horizontal plane image 500, the processing of S203 does not necessarily have to be performed. In this case, it is possible to determine whether the worker is a single worker or a collaborative worker by using the distance on the captured image. Further, various other methods may be used for generating the horizontal plane image 500.

Next, the control unit 121 updates the list of detected persons (S203). At the beginning of the iterative process, the control unit 121 lists all the detected persons. At this time, the control unit 121 stores the coordinates of the representative point on the captured image in which the person is detected and the coordinates of the representative point on the horizontal plane image in each node in the list. If it is the second and subsequent times of the iterative process, the control unit 121 performs the following process for all the detected persons.
(Process 1). The control unit 121 calculates the distance between the coordinates on the captured image of the person stored in each node in the list and the coordinates on its own captured image, and if it is equal to or less than the threshold value, the target node is set to its own. Replace with data. At this time, the control unit 121 saves the history of the coordinates saved in the node so far.
(Process 2). If there is no node satisfying the condition in (Process 1), the control unit 121 adds its own data to the end of the list.
Further, the control unit 121 performs the following processing on all the detected persons after completing the above processing.
(Process 3). The control unit 121 deletes the node that has not been updated.

Next, the control unit 121 determines whether or not there is a single operator in the captured image (S204 to S206). In the present embodiment, as shown in FIG. 7A, the control unit 121 sets the work area frame (501 to 506) in the captured image display based on the user operation or the like. The processing flow is as follows.
(Process 4). The control unit 121 confirms whether the coordinates on the captured image of the person to be processed in order from the top of the list are within the work area frame, and if it is within the work area frame, the work frame is added to the list. Stores a flag indicating that it is in. At this time, when there are a plurality of work areas, the control unit 121 stores the unique ID numbers assigned to each work area frame in the list (S204).
(Process 5). The control unit 121 searches the list and confirms how many workers belong to the area for each work area frame (S205). When the number of workers in the work area frame is 1, the control unit 121 determines that independent work is being performed in that area, and determines that independent work is not being performed in other cases (S206). .. That is, when the number of workers in the work area frame is 1, the control unit 121 determines that the worker in that area is a single worker who cannot be associated with another person. When there are two or more workers in the work area frame, the control unit 121 associates the workers in the area with each other and determines that they are joint independent workers. As described above, the control unit 121 has a function as an association unit.
In the present embodiment, the control unit 121 first determines whether or not the person is within the work area frame based on the list of detected persons, but whether or not each person is within the work area frame based on each work area frame. You may check and judge only by the number of people in the work area.
Further, although the description has been made assuming that the work area frame is set in the captured image display, the control unit 121 sets the work area frame in the horizontal plane image based on the user operation as shown in FIG. 7B. You may.

Here, the process of determining whether the worker is a single worker or a collaborative worker is not limited to the above-mentioned process. For example, the control unit 121 is a collaborative worker in association with each other based on at least one of the position of the person detected from the captured image, the orientation of the face of the person, and the distance between the people. It may be determined. For example, the control unit 121 may associate people in a predetermined range from a predetermined object in the captured image, for example, a workbench, or estimate the direction (or line of sight) of the person's face. People whose face orientations (or eyes) intersect may be associated with each other. Further, for example, the control unit 121 may associate people whose distances between people are within a predetermined distance. Then, the control unit 121 may determine a person associated with another person as a collaborator and a person not associated with the other person as a single worker.
Further, the control unit 121 may further associate the persons with each other based on the color or shape of the uniform, bibs, or hat of the person. For example, the control unit 121 associates people with the same uniform shape among those who are within a predetermined distance, and even if the distance between the people is within a predetermined distance, the shape of the uniform. People with different values may not be associated with each other. In this way, the control unit 121 may associate the person based on the feature amount of the object worn by the person.
Further, even if a plurality of workers are present in the work area frame, the control unit 121 may determine the orientation of the faces of the plurality of workers in the work area frame or the distance between the workers. , You may decide that you are a single worker.

Next, the control unit 121 determines whether or not the single work is in progress, and if the condition is satisfied, records the log as the single work and notifies the log (S207 to S209). The processing flow is as follows.
(Process 6). When there is a work area frame in which the independent work is performed in (Process 5), the control unit 121 increments the cumulative number of frames for the independent work, that is, increases the value of the variable by 1, and at the same time, the control unit 121 increases the value of the variable by 1. The person belonging to the work area is extracted from the list (S207). When the independent work is not performed, the control unit 121 sets the cumulative number of frames for the independent work to 0 (S208).
(Processing 7). When the cumulative number of frames for single work exceeds the threshold value, the control unit 121 determines that single work is in progress (S209). By using this determination condition, for example, it is possible to suppress erroneous notification that a person is a single worker due to a momentary failure to detect a person even though he / she is a collaborative worker.
When the control unit 121 determines that the work is independent, the control unit 121 displays the display device 140 so that the time and duration of the operation, the image pickup device 110 in which the operation is occurring, or, for example, the place where the independent work is performed among the captured images can be known. Is displayed on the screen of (S210). The screen of FIG. 8A is an example of a screen showing a list of moving images in a time zone in which a single operation is detected. The text information "list of independent work detection" in FIG. 8A and the time, duration, and location information are examples of information that can be identified as a moving image including a person who is not associated with another person. Is. Depending on the intended use, a moving image of a time body that does not include a person who is not associated with another person may be displayed as a list. In that case, for example, the text information "collaboration work detection list" is displayed.
Next, the control unit 121 determines whether to repeat the process from the acquisition of the captured image (S200) to the determination (S210, S211) of whether or not to notify the user of the presence or absence of a single vendor (S212). The control unit 121 determines that the repetitive processing is terminated when it becomes impossible to acquire the captured image from the image pickup apparatus or when the user gives an instruction to interrupt the repetitive processing.

In the display in S210, the control unit 121 is supposed to display information (list) related to the independent work together with the thumbnail image as shown in FIG. 8A, but it may be displayed only in the list format (list). Further, the control unit 121 may display the recorded video and the timeline as shown in FIG. 8B, and display the time zone of the independent work on the timeline. The part of the timeline (blackened or shaded part) represented as the time zone of the single work in FIG. 8B is an example of information that can identify a person who is not associated with another person. .. For example, the black-painted portion is a time zone in which one single worker is present, and the shaded portion is a time zone in which there are a plurality of single workers. Further, "A-1" is information indicating an ID or the like for specifying the image pickup apparatus. In the example of FIG. 8B, the timelines of the two image pickup devices "A-1" and "B-1" are shown.
On the other hand, it is identifiable that the part of the timeline (white part) represented as a time zone other than the time zone of the single work in FIG. 8B does not include a person who is not associated with another person. It is an example of information. Further, the control unit 121 may display the time zone of the independent work detected by a plurality of cameras on one timeline. As described above, at least, on the timeline, information that can be identified to include a person that cannot be associated with another person, and information that can be identified that a person that cannot be associated with another person is not included. At least one of the above may be displayed. It suffices if it is possible to distinguish between a time zone in which a person who is not associated with another person is included in a moving image and a time zone in which a person who is not associated with another person is not included.
Further, for example, as shown in FIG. 9A, the user specifies the display of each independent work with a mouse, a finger, or the like. Then, the control unit 121 reproduces the recorded image of the corresponding image pickup device on the display device 140 via the display control unit 125 as shown in FIG. 9B, and displays the captured image at that time. You may be able to confirm it. By doing so, the convenience of the user is enhanced. At this time, the control unit 121 may reproduce only the recorded video when the independent work occurs, but may reproduce the video from the video when the independent work occurs a predetermined time (for example, several tens of seconds) before the occurrence time, or continue the independent work. Playing for longer than the time is more preferable because the situation before and after that can be easily confirmed.
Further, the control unit 121 may connect the recorded video in the time zone detected as a single operation and reproduce it as a digest video. That is, the moving images in the list shown in FIG. 9A may be continuously reproduced. At that time, the moving image in the time zone of the independent work of about several seconds may not be included in the digest video.
In order to prevent erroneous detection, the notification of the single work issues an alarm when the single work continues for a predetermined time, but the duration of the single work is the time from the first detected frame. Therefore, the control unit 121 may set the section displayed as a single work on the timeline or the start of recording or reproduction to the time when the single work is started to be detected, not the time when the alarm is issued.

Further, the control unit 121 may display the timeline not only in the single work time zone when the cumulative number of frames in which the single worker exists exceeds the threshold value and an alarm is issued, but also as follows. That is, even if the control unit 121 determines that the work is independent work for a short period of time without issuing an alarm, the control unit 121 confirms both by displaying the color, shape, and pattern different from the single work time zone in which the alarm is issued. It may be possible. In this case, the single work detected only for a moment may actually be a temporary temporary hiding of one of the collaborators in the shield, so there are two thresholds for determining the single work. It is good to provide. At this time, the control unit 121 shall issue an alarm based on the threshold value having the larger value among the two threshold values. For example, the first threshold is used in information that can be identified as including a person that is not associated with another person. Then, in order to determine whether to issue an alarm, a second threshold value larger than the first threshold value is used.
Further, the control unit 121 may save a list of image pickup devices, areas, and time information in which independent work is detected as a CSV file, and paste a list with thumbnail images, a screenshot of a recorded video, and the like as a report. , May be printable.
Further, when the control unit 121 has an image pickup device that similarly captures an area at a different angle or captures a peripheral area in addition to the captured image for detecting a single work, the following The processing may be performed. That is, the control unit 121 may display a video image taken and recorded by an image pickup device for which association settings have been made in advance.
Further, as shown in FIG. 10A, in the case of a system that detects a single work in a live image and issues an alarm in real time when it is determined that the single work is performed, a work observer confirms the current situation. , It is necessary to check the past situation. Therefore, the control unit 121 may control to display both the live image and the recorded image. In FIG. 10A, in the image of room A, information that can be identified as including a person who is not associated with another person is displayed by changing the index on the upper right.
As shown in FIG. 10B, the control unit 121 may delete the selected image or the like from the series of displays after the work observer determines that there is no false alarm or problem.

In the present embodiment, the control unit 121 has mainly described using an example of setting a work area frame. When the control unit 121 detects a single work, the control unit 121 may superimpose the work area frame when the single work is detected on the image and display it on the display device 140. Further, the control unit 121 superimposes and displays all the work area frames on the image, and when the independent work is performed, the work area frame on which the independent work is performed has a different color from the other work area frames. It may be displayed explicitly or highlighted, such as by changing to, or by making it blink. By doing so, the work observer has good visibility and can grasp whether the person in the image is single or plural.
Further, in the present embodiment, an example in which a person or the like is detected and displayed by one client device is shown, but processing may be performed by a plurality of client devices. For example, a first client device for inputting a video, detecting and notifying an independent work, and a second client device for receiving the notification, outputting an alarm, displaying a list, or displaying on a timeline. The processing may be performed with the client device of. An example of the first client device is a VCA (Video Content Analysis) device. An example of the second client device is a VMS (Video Management System) device. The hardware configuration of the first client device and the second client device is the same as the hardware configuration of the client device 120 of FIG. The function of each client device is realized by the CPU of each client device executing processing based on the program stored in the memory of each client device.

FIG. 11A shows an example of the case where the system of the present embodiment and the entry / exit management system 160 are linked. The entry / exit management system records the times when workers enter and leave the work room. Therefore, as shown in FIG. 11B, the control unit 121 acquires information on the time of entry / exit and exit time for each worker from the entry / exit management system 160, and from the acquired information, the work of entering the work room. Extract the person. Then, the control unit 121 may record the extracted list of worker names as a part of the information of the occurrence list of the independent work, or display it on the screen. However, this is not the case, and the control unit 121 may send an e-mail to the user. This recording and display may be realized by the system of this embodiment, or may be incorporated into an entry / exit management system or the like. The list of extracted worker names is an example of room entry / exit information.

<Embodiment 2>
In the present embodiment, the configuration is the same as that of the first embodiment, and the determination of whether or not a single worker exists will be described by using an example in which the determination by the distance is performed instead of the detection by the work area frame. The processing flow of this embodiment will be described with reference to FIG.
The process from the control unit 121 acquiring the captured image from the image pickup device 110 (S1101) to updating the list of detected persons (S1103) is the same as the process described in the first embodiment.
Next, the control unit 121 determines whether or not there is a single operator in the captured image (S1104 to S1106). The processing flow is as follows.
(Process 1). The control unit 121 targets the first person in the list for processing (S1104).
(Process 2). The control unit 121 determines whether or not the person to be processed is a single worker (S1105). In the present embodiment, the control unit 121 confirms the following items with respect to the person to be processed in order to determine whether or not it is a single worker.

(Item 1). The control unit 121 determines whether or not there is a person detected within the radius R with the coordinates on the horizontal plane image of the person to be processed as the center.
This radius R may be appropriately set according to the situation to which the present embodiment is applied. For example, when monitoring work indoors, the radius R can be a distance corresponding to 2 m. However, at this time, when the control unit 121 refers to the coordinate history stored in the node of the person to be processed and determines that the person to be processed is moving, when it is determined that the person is not moving. The value of R, which is a threshold value, is made larger than that. At this time, if the collaborators work side by side in the x-axis direction on the horizontal plane image, the control unit 121 may use an ellipse in which a circle having a radius R is contracted in the y-axis direction. Similarly, if the collaborators work side by side in the y-axis direction on the horizontal plane image, the control unit 121 may use an ellipse in which a circle having a radius R is contracted in the x-axis direction.
(Item 2). When a person satisfying (item 1) exists, the control unit 121 determines whether or not the person exists in the detection area (work area frame).
In the present embodiment, the control unit 121 operates the input device 130 and the like to set the area designated by the user as the detection area in which the operator exists. This makes it possible to prevent, for example, a non-worker moving across the glass wall and a person reflected in a poster or the like placed on a desk from being erroneously detected as a collaborator. However, the setting of the detection area is not limited to the input by the user via the input device 130 or the like. The control unit 121 may automatically set a region surrounded by a marker arranged on the captured image as a detection region. As a method of detecting using a marker, for example, the two-dimensional marker installed at the four corners of the detection area is registered in the client device 120 in advance by raster scanning the captured image of the above-mentioned two-dimensional marker. There is a way to pattern match with information.
If all of the above (item 1) and (item 2) are satisfied, the control unit 121 determines that the worker is a collaborator, and proceeds to (process 3-1). That is, the control unit 121 associates the workers satisfying all of the above (item 1) and (item 2) with each other, and determines that they are joint independent workers. If the control unit 121 does not satisfy at least one, the control unit 121 determines that the person to be processed is a single worker, and proceeds to (Process 3-2).

Here, the process of determining whether the worker is a single worker or a collaborative worker is not limited to the above-mentioned process. For example, the control unit 121 may associate the persons with each other based on at least one of the position of the person detected from the captured image, the orientation of the face of the person, and the distance between the persons. For example, the control unit 121 may associate people in a predetermined range from a predetermined object in the captured image, for example, a workbench, or estimate the direction (or line of sight) of the person's face. People whose face orientations (or eyes) intersect may be associated with each other. Further, for example, the control unit 121 may associate people whose distances between people are within a predetermined distance. Then, the control unit 121 may determine a person associated with another person as a collaborator and a person not associated with the other person as a single worker. Further, the control unit 121 may further associate the persons with each other based on the color or shape of the person's uniform, bibs (vest-shaped clothes), or hat. For example, the control unit 121 associates people with the same uniform shape among those who are within a predetermined distance, and even if the distance between the people is within a predetermined distance, the shape of the uniform. People with different values may not be associated with each other, or combinations with different bibs colors within a predetermined distance may be determined by a plurality of conditions such as a collaborative worker and the others as a single worker.

式（４）中のｑｘ、ｑｙはそれぞれ、処理対象としている人物の、水平面画像上でのｘ座標、及びｙ座標を表す。図１３では、ｑｘ、ｑｙはそれぞれ楕円１２０２のｘ座標、及びｙ座標を表す。式（４）中のａ、ｂは楕円の変換パラメータである。例えば、ａ＝１、ｂ＝１とすると、楕円は半径Ｒ、中心（ｑｘ、ｑｙ）の円となる。また、ａ＝１、ｂ＝１／２とすると、ｘ軸方向の長さがＲ、ｙ軸方向の長さがＲ／２となる楕円となる。本実施形態では共同作業者同士が水平面画像上のｘ軸方向に並んで作業を行うため、ａ＞ｂとなる値を用いることとする。 The determination in the above (item 1) and (item 2) will be described with reference to FIG. In FIG. 13, the ellipses 1201 to 1205 represent the result of converting the center of the ellipses 401 to 405, which are the person detection positions, into the coordinates on the horizontal plane image. Further, the area 1206 represents a detection area designated by the user. Further, the ellipse 1200 represents an ellipse consisting of x and y satisfying the following equation (4).

Qx and qy in the equation (4) represent the x-coordinate and the y-coordinate of the person to be processed on the horizontal plane image, respectively. In FIG. 13, qx and qy represent the x-coordinate and the y-coordinate of the ellipse 1202, respectively. A and b in the equation (4) are conversion parameters of the ellipse. For example, if a = 1 and b = 1, the ellipse is a circle having a radius R and a center (qx, qy). Further, when a = 1 and b = 1/2, the ellipse has a length in the x-axis direction of R and a length in the y-axis direction of R / 2. In the present embodiment, since the collaborators work side by side in the x-axis direction on the horizontal plane image, a value such that a> b is used.

For example, it is assumed that the control unit 121 targets the person in the ellipse 402 at the person detection position during the scanning of the list. At this time, the control unit 121 determines the above (item 1) to (item 2) with respect to the coordinates of the ellipse 1202 on the horizontal plane image corresponding to the ellipse 402 at the person detection position. First, in (item 1), the control unit 121 determines whether or not the detected person exists inside the ellipse 1200. In the situation shown in FIG. 13, since the coordinates of the ellipse 1201 on the horizontal plane image corresponding to the ellipse 401 of the person detection position are included in the ellipse, the control unit 121 determines that the determination is satisfied. Next, in (item 2), the control unit 121 determines whether the coordinates 601 on the horizontal plane image corresponding to the ellipse 401 of the detection position of the person satisfying the condition of (item 1) are included in the region 1206. .. Under the circumstances described in the example of the present embodiment, since the coordinates of the ellipse 1201 are included in the region 1206, the control unit 121 determines that the determination is satisfied. Since the above (item 1) and (item 2) are all satisfied, the control unit 121 determines that the person of the ellipse 402 is a collaborator.

(Process 3-1). The control unit 121 determines whether or not the person at the end of the list is the processing target (S1106). If it is not at the end of the list, the control unit 121 targets the next person for processing, and returns to (processing 2) (S1107). If it is the end of the list, the control unit 121 ends the scan in the list and sets the cumulative number of frames in which a single worker exists to 0 (S1108).
(Process 3-2). The control unit 121 increments the cumulative number of frames in which a single worker exists (S1109).
Next, the control unit 121 determines whether or not to notify the user that a single worker exists, and if the condition is satisfied, notifies the user (S1110, S1111).
The control unit 121 performs the same processing as in the first embodiment for the display processing and the like at the time of notification.
Further, in the present embodiment, the control unit 121 displays a UI indicating that a single worker exists on the display device 140 as a method of notifying the user that a single worker exists. For example, the control unit 121 superimposes a double circle on the position of a single operator on the captured image 300 or the horizontal plane image 500, superimposes a red circle, and the like. However, this is not limited to this, and the control unit 121 may send an e-mail to the user, output to an external output, and issue an alert from the speaker.

<Other embodiments>
The present invention supplies a system or apparatus via a network or storage medium a program that realizes one or more of the functions of the above-described embodiment. It can also be realized by a process in which one or more processors in the computer of the system or apparatus reads and executes a program. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Although the preferred embodiment of the present invention has been described in detail above, the present invention is not limited to the specific embodiment. The functional configuration of the image pickup apparatus 110 may be implemented in the image pickup apparatus as a software configuration realized by the CPU 12 executing a process based on a program stored in the memory 13. Further, a part or all of the functional configurations of the image pickup apparatus 110 may be mounted on the image pickup apparatus 110 as a hardware configuration. Similarly, the functional configuration of the client device 120 may be implemented in the client device 120 as a software configuration realized by the CPU 22 executing a process based on a program stored in the memory 21. Further, a part or all of the functional configuration of the client device 120 may be mounted on the client device 120 as a hardware configuration.
In addition to superimposing the double circle on the position of the single worker and superimposing the red circle, the single worker may be superposed with the character string "single worker", or a warning color such as red may be displayed. It may be superimposed on a single operator on the captured image. Further, these may be arbitrarily combined and displayed in an superimposed manner.
Further, the hardware configuration of the client device and the image pickup device described above is an example, and for example, a plurality of CPUs, memories, communication I / Fs, and the like may be provided.

As described above, according to the processing of each of the above-described embodiments, it is possible to appropriately present whether the person in the image is a single person or a plurality of people. By receiving a notification that there is a single worker in the work site and easily checking the video, the work observer can find the worker who violates the rules and pay attention to the worker. It will be possible to encourage.

21 memory 22 CPU
120 Client device 140 Display device

Claims (15)

Detection means to detect people in the area where people work,
A specific means for identifying whether the person detected by the detection means is a person who is working alone, and
Alone in the area where a plurality of people must work on condition that the person detected by the detection means in the area is identified as a person who is working alone by the specific means. A notification control means that controls to notify the user of recognizable information that there is a person working in
A control device characterized by having. The control device according to claim 1, wherein the detection means detects a person from a moving image of an area in which the person works. Further having a setting means for setting the area on the moving image,
The control device according to claim 2, wherein the specific means identifies whether the person detected by the detection means is a person who is working alone in the area set by the setting means. .. The claim is characterized in that, when there is only one person detected by the detection means in the area set by the setting means, the specific means identifies the person as a person who is working alone. 3. The control device according to 3. The specific means is characterized in that when a plurality of persons are detected by the detection means and the distance between the plurality of persons is more than a predetermined distance, the person is identified as a person who is working alone. The control device according to any one of claims 1 to 3. The specific means is characterized in that a plurality of persons are detected by the detection means, and when the face orientations or lines of sight of the plurality of persons do not intersect, the person is identified as a person who is working alone. The control device according to claim 2 or 3. The notification control means notifies by displaying on the display unit the information that the user can recognize that there is a person who is working alone in the area where the work must be performed by a plurality of people. The control device according to claim 3, further comprising control. The notification control means is controlled so as not to notify the information on condition that the specific means identifies that the person detected by the detection means in the area is not a person who is working alone. The control device according to any one of claims 1 to 7. A detection means that detects a person from a moving image,
A setting means for setting an area on the moving image for identifying whether the person detected by the detection means is a person who is working alone, and a setting means.
In the area set by the setting means, a specific means for identifying whether the person detected by the detection means is a person who is working alone, and a specific means.
A display control means for controlling the display of the timeline related to the moving image, and
Have,
The specific means further specifies whether or not the duration of the work independently by the person detected by the detection means in the area exceeds a predetermined time.
The display control means includes a time zone of the duration specified by the specific means to exceed the predetermined time, and a time zone of the duration specified by the specific means not to exceed the predetermined time. A control device for controlling so as to display identifiable information on the timeline. A detection means that detects a person from a moving image,
A setting means for setting an area on the moving image for identifying whether the person detected by the detection means is a person who is working alone, and a setting means.
In the area set by the setting means, and identifying means for identifying whether a person the person detected is working solely by the detecting means,
It has a display control means for controlling the display of the timeline related to the moving image.
The setting means can set a plurality of the areas on the moving image.
The display control means includes a time zone in which a plurality of areas specified by the specific means exist as a person who is independently working on the person detected by the detection means, and a person detected by the detection means. It is characterized in that the time zone in which a single area specified by the specific means exists as a person who is working alone is controlled so as to display identifiable information on the timeline. Control device. The specific means further specifies whether or not the duration of the work independently by the person detected by the detection means in the area exceeds a predetermined time.
When the display control means is an area specified by the specific means that the person detected by the detection means is a person who is working alone, and the duration exceeds the predetermined time. A time zone in which a plurality of specified areas exist, an area specified by the specific means that the person detected by the detection means is a person who is working alone, and the duration. The control device according to claim 10, wherein the control device controls to display information that can identify a single area specified as exceeding a predetermined time on the timeline. It is a control method executed by the control device.
A detection step person to detect a person area for work,
A specific step for identifying whether the person detected in the detection step is a person who is working alone, and
In the specific step, on condition that the person detected in the detection step in the area is identified as a person who is working alone, in the area where a plurality of people must work alone. A control method comprising: a notification control step that controls to notify a user recognizable information that a person working in is present. A detection step that detects a person from a moving image,
A setting step for setting an area on the moving image for identifying whether the person detected in the detection step is a person who is working alone, and a setting step.
In the area set in the setting step, it is controlled to display a specific step for specifying whether the person detected in the detection step is a person who is working alone, and a timeline related to the moving image. Including display control steps,
In the specific step, it is further specified whether or not the duration of the person detected by the detection step in the area is a person who is working alone and exceeds a predetermined time.
In the display control step, the time zone of the duration specified in the specific step to exceed the predetermined time, and the time zone of the duration specified not to exceed the predetermined time in the specific step. A control method, characterized in that information that can be identified is controlled so as to be displayed on the timeline. A detection step that detects a person from a moving image,
A setting step for setting an area on the moving image for identifying whether the person detected by the detection step is a person who is working alone, and a setting step.
In the area set by the setting step, a specific step for identifying whether the person detected by the detection step is a person who is working alone, and a specific step.
Including a display control step for controlling to display the timeline related to the moving image.
In the setting step, a plurality of the areas can be set on the moving image.
In the display control step, a time zone in which there are a plurality of areas specified in the specific step that the person detected in the detection step is a person who is working alone, and a person detected in the detection step. It is characterized in that information that can identify the time zone in which a single area specified in the specific step exists as a person who is working alone is displayed on the timeline. Control method. A program for making a computer function as each means of the control device according to any one of claims 1 to 11.

Images