JP6648923B2 - Control device, control method, and program - Google Patents

Description

The present invention, control apparatus, control relating to control method and a program.

  Conventionally, a method of detecting a person from an image has been known. Patent Literature 1 discloses a method of detecting a shielded person from an image.

JP 2014-199506 A

According to the technique disclosed in Patent Document 1, it is possible to notify the position of a person in an image. For example, in the above technique, a person present in an image can be circled and the position of the person can be presented to the user. However, by merely indicating the position of a person based on the above-described technology, the user is either isolated from the group and working alone, or is the person working together at a position close to another person? There was a problem that could not be determined at a glance. For example, sites that handle data that should be protected from leaks may have rules that prohibit them from working alone, and work watchers monitor workers via imaging devices, There is a case where it is desired to call attention to a single worker who is working alone in isolation from a group. However, only by looking at the captured image, the work monitor cannot determine at a glance whether each of the workers in the captured image is working alone or working with a plurality of workers. Further, it is burdensome for the work observer to constantly monitor the worker.
An object of the present invention is to appropriately present whether a person in an image is single or plural.

Therefore, the control apparatus of the present invention includes a detection means for detecting a person from a moving image, an area for people detected by the detection means to identify whether the one set on the moving image set Means, specifying means for specifying whether the number of persons detected by the detecting means is one in the area set by the setting means, and display for controlling to display a timeline related to the moving image Control means, and the specifying means further specifies whether or not the duration of which the number of persons detected by the detection means in the area is one exceeds a predetermined time, and the display control means Identifying the time period of the duration specified by the specifying unit as exceeding the predetermined time and the time period of the duration specified not to exceed the predetermined time by the specifying unit. And controlling to display the ability information on the timeline.

  According to the present invention, an object is to appropriately present whether a person in an image is a single person or a plurality of persons.

FIG. 2 is a diagram illustrating an example of a system configuration and a hardware configuration of the system. FIG. 3 is a diagram illustrating an example of a functional configuration and the like of an imaging device and a client device. It is a flow chart which shows an example of processing of notice control. It is a figure showing an example of a picked-up image. It is a figure showing an example of an image showing the result of having performed person detection. It is a figure showing an example of a horizontal plane image. It is a figure showing an example of a work area frame. It is a figure showing an example of a display screen. It is a figure showing an example of a display screen. It is a figure showing an example of a display screen. It is a figure showing an example which cooperated with an entrance / exit management system. It is a flow chart which shows an example of processing of notice control. FIG. 9 is a diagram illustrating an example of determination.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<First embodiment>
In the present embodiment, in a site where data is to be prevented from being leaked, which is obligatory for two or more workers to perform work, a work monitor is assigned to a single worker (a person not associated with another person). ) Or an image including a collaborator (a person associated with another person) will be described. Such a situation is an example of a situation in which the present embodiment can be used, and can be used in various situations where it is desired to determine whether a person is a single person or a plurality of persons. For example, the present invention can be used in situations where it is determined whether passers-by or visitors are alone or in a group and used for marketing, solving social problems, and the like.
FIG. 1 shows an example of a system configuration and a hardware configuration of the system according to the present embodiment. In FIG. 1, an imaging device 110 performs imaging. The client device 120 drives the imaging device 110 and displays a captured image captured by the imaging device 110. The input device 130 includes a mouse, a keyboard, and the like, and performs user input to the client device 120. The display device 140 includes a display or the like, and displays an image output by the client device 120. Although the client device 120 and the display device 140 are shown as independent devices in FIG. 1, the client device 120 and the display device 140 may be integrated. Further, the input device 130 and the display device 140 may be integrated, or the client device 120, the input device 130, and the display device 140 may be integrated. The network 150 connects the imaging device 110 and the client device 120. The network 150 includes, for example, a plurality of routers, switches, cables, and the like that satisfy communication standards such as a local network. In the present embodiment, any communication standard, scale, and configuration can be used as long as communication between the imaging device and the client can be performed. For example, the network 150 may be configured by the Internet, a wired LAN (Local Area Network), a wireless LAN (Wireless LAN), a WAN (Wide Area Network), or the like. Further, the number of imaging devices connected to the client device 120 is not limited to one, but may be a plurality.

The hardware configuration of the imaging device 110 will be described. The imaging device 110 includes an imaging unit 11, a CPU 12, a memory 13, and a communication I / F 14 as a hardware configuration. The imaging unit 11 has an image sensor and an engineering system of a subject on the image sensor, and performs imaging on the image sensor based on the intersection of the optical axis of the optical system and the image sensor under the control of the CPU 12. The imaging device is an imaging device such as a CMOS (Complementary Metal-Oxide Semiconductor) or a CCD (Charged Coupled Device). The CPU 12 controls the entire imaging device 110. The memory 13 stores programs, images captured by the imaging unit 11, data used when the CPU 12 executes processing, and the like. The communication I / F 14 controls communication with the client device 120 via the network 150 based on the control of the CPU 12. The functions and the like of the imaging device 110 are realized by the CPU 12 executing the processing based on the program stored in the memory 13. Note that a processor other than the CPU may be used.
The hardware configuration of the client device 120 will be described. The client device 120 includes a memory 21, a CPU 22, a communication I / F 23, an input I / F 24, and a display I / F 25 as a hardware configuration. The CPU 22 controls the entire client device 120. The memory 21 stores programs, captured images transmitted from the imaging device 110, data used when the CPU 22 executes processing, and the like. The communication I / F 23 controls communication with the imaging device 110 via the network 150 under the control of the CPU 22. The input I / F 24 connects the client device 120 and the input device 130, and controls input of information from the input device 130. The display I / F 25 connects the client device 120 and the display device 140, and controls output of information to the display device 140. When the CPU 22 executes the processing based on the program stored in the memory 21, the function of the client device 120 and the processing of the flowcharts in FIGS. 3 and 12 described below are realized.

FIG. 2 is a diagram illustrating an example of a functional configuration of the imaging device 110 and the client device 120. The imaging device 110 includes a control unit 111, a signal processing unit 112, a drive control unit 113, and a communication control unit 114 as a functional configuration. The control unit 111 controls the imaging device 110. The signal processing unit 112 performs processing on an image signal captured by the imaging unit 11. For example, the signal processing unit 112 encodes an image captured by the imaging unit 11. The signal processing unit 112 can use, for example, JPEG (Joint Photographic Experts Group) as an encoding method. Alternatively, the signal processing unit 112 uses H.264 as the encoding method. H.264 / MPEG-4 AVC (hereinafter, H.264) can be used. Alternatively, the signal processing unit 112 can use HEVC (High Efficiency Video Coding) as an encoding method. However, the encoding method is not limited to these. Further, the signal processing unit 112 may perform encoding by selecting an encoding method from a plurality of encoding methods.
The drive control unit 113 performs control to change the imaging direction and the angle of view of the imaging unit 11. In the present embodiment, a case will be described in which the imaging unit 11 can change the imaging direction in the pan direction and the tilt direction, and can change the imaging angle of view, but is not limited thereto. The imaging device 110 may not have a function of changing the imaging direction in the pan direction and the tilt direction, and may not have a function of changing the angle of view. The communication control unit 114 transmits the captured image processed by the signal processing unit 112 to the client device 120. Further, the communication control unit 114 receives a control command for the imaging device 110 from the client device 120.
The client device 120 includes a control unit 121, a communication control unit 123, an information acquisition unit 124, and a display control unit 125 as functional configurations. The control unit 121 controls the client device 120. The communication control unit 123 receives the captured image from the client device 120. The information obtaining unit 124 receives a user input from the input device 130 and obtains input information from the input device 130. The display control unit 125 outputs the video to the display device 140, and causes the display device 140 to execute a display process described later.

Next, a processing flow in the present embodiment will be described with reference to FIG. FIG. 3 is a flowchart illustrating an example of display control processing in the client device 120. The client device 120 repeatedly executes the process of repeatedly acquiring the captured image, that is, the moving image from the imaging device 110, and presenting the display device 140 in an identifiable manner whether or not the acquired captured image includes a single worker. Let it.
The control unit 121 acquires a captured image from the imaging device 110 (S200). FIG. 4 is a diagram illustrating an example of a captured image 300 acquired by the client device 120 from the imaging device 110. In FIG. 4, workers 301 and 302 and workers 303 and 304 represent co-workers who are working in pairs. On the other hand, the worker 305 represents a single worker working alone. The work table 306 represents a work table located far from the imaging device 110. The work table 307 represents a work table existing at a position near the imaging device 110.
At this time, it is preferable to adjust the position, the mounting angle, and the angle of view of the imaging unit 11 of the imaging device 110 in advance so that the captured images acquired by the client device 120 do not overlap as much as possible by the operator. In the present embodiment, the client device 120 performs processing on a captured image acquired from the imaging device 110. As a result, processing can be performed on real-time live video. However, it is not limited to this. For example, the client device 120 may acquire a moving image stored in the imaging device 110 and perform processing on each frame, or may perform processing on each frame of the moving image stored in the client device 120. May be performed. Alternatively, the client device 120 may access a recording server or the like and perform processing on each frame of the moving image stored in the recording server. Further, in the present embodiment, the co-workers are set in pairs, but the present invention is not limited to this. The client device 120 may have three or more co-workers.

  Next, the control unit 121 performs a person detection process on the captured image 300 acquired in S200 (S201). In the present embodiment, the control unit 121 first performs scaling on the captured image 300 in various sizes to perform the person detection process. By performing scaling, people of various sizes can be detected. Next, the control unit 121 performs a raster scan on the captured image of each scale using a detection window of a specific size. At this time, the control unit 121 calculates in advance the feature amount of the person using the learning data, and sets an error between the feature amount calculated in the detection window at the time of scanning and the feature amount based on the learning data as a threshold. If it is smaller, it is determined that the person is a person. Various methods such as such a pattern matching process can be used for the person detection process. FIG. 5 is a diagram illustrating an example of an image 400 indicating a result of performing a person detection on the captured image 300. In FIG. 5A, ellipses 401 to 405 represent positions where the workers 301 to 305 are detected, respectively.

Next, the control unit 121 performs a person detection process on the captured image 300 acquired in S200 (S201). In the present embodiment, the control unit 121 first performs scaling on the captured image 300 in various sizes to perform the person detection process. By performing scaling, people of various sizes can be detected. Next, the control unit 121 performs a raster scan on the captured image of each scale using a detection window of a specific size. At this time, the control unit 121 calculates in advance the feature amount of the person using the learning data, and sets an error between the feature amount calculated in the detection window at the time of scanning and the feature amount based on the learning data as a threshold. If it is smaller, it is determined that the person is a person.
FIG. 5 is a diagram illustrating an example of an image 400 indicating a result of performing a person detection on the captured image 300. In FIG. 5A, ellipses 401 to 405 represent positions where the workers 301 to 305 are detected, respectively. Arrows 406 to 408 are semi-straight lines for expressing the direction of the face of each person, with the end side of the arrow as a base point and the head side extending infinitely. In addition, as described above, in addition to the detection method using the feature amount of the person, as shown in FIG. 5B, at least one of a uniform, a bibs, and a hat determined in advance as a rule of the worker. May be decided to be worn. That is, the feature amount of an object worn by a person to be detected as an operator may be stored in advance in the memory 21 of the client device 120 as a rule as a setting file. In this way, the control unit 121 determines uniforms and the like of the feature amount that matches the rule based on the rule of the setting file and the feature amount such as the color and shape of the thing worn by the person to be processed of the captured image. It can also detect worn workers.

ここで、式（２）中のｐｘ１、ｐｘ２、．．．、ｐｘ４、ｐｙ４、及びｑｘ１、ｑｘ２、．．．、ｑｘ４、ｑｙ４は、ユーザーによって指定された撮像画像上の四点のｘ、ｙ座標、及び水平面画像上の四点のｘ、ｙ座標をそれぞれ表す。
次に制御部１２１は、撮像画像上のｘ座標ｐｘ、Ｙ座標ｐｙとホモグラフィ行列Ｈとに対し、以下の式（３）を適用することにより、ｐｘ、ｐｙに対応する水平面画像上のＸ座標ｑｘと、Ｙ座標ｑｙとを計算することができる。

Next, the control unit 121 converts the position of the person detected in S101 into coordinates on a horizontal plane image obtained by projecting a shooting range of a captured image from right above on a flat screen (S202). In the present embodiment, it is assumed that which point on the captured image corresponds to which point on the horizontal image is determined in advance. For example, the user specifies four points on the captured image displayed on the display device 140 or the like and four points on the horizontal image by operating the input device 130 or the like. The control unit 121 calculates the homography matrix H in advance based on the four points on the captured image input by the user, the four points on the horizontal plane image, and the following equations (1) and (2). .

Here, px1, px2,. . . , Px4, py4, and qx1, qx2,. . . , Qx4, and qy4 represent the x and y coordinates of four points on the captured image specified by the user and the x and y coordinates of four points on the horizontal plane image, respectively.
Next, the control unit 121 applies the following expression (3) to the x coordinate px, the Y coordinate py and the homography matrix H on the captured image, thereby obtaining the X coordinate on the horizontal image corresponding to the px and py. The coordinates qx and the Y coordinates qy can be calculated.

  FIG. 6 is a diagram illustrating an example of a horizontal plane image 500 of the work site. In FIG. 6, coordinates 501 to 505 represent the results of converting representative points of the ellipses 401 to 405, which are human detection positions, into coordinates on a horizontal plane image. In the present embodiment, the representative point is, for example, the center of the person detection position. The representative point may be the center of the area corresponding to the head of the person. Areas 506 and 507 represent areas when the work tables 306 and 307 are represented on a horizontal plane image, respectively. In the present embodiment, a sketch of the work site created by CAD or the like is used as the horizontal plane image, but the present invention is not limited to this. For example, an image obtained by transforming a captured image may be used, or a handwritten image may be used. Further, the image may not be provided, and only the above-mentioned coordinate information may be stored in the program. That is, the horizontal plane image 500 does not need to be generated, and only the calculation of the coordinate position on the horizontal plane image 500 may be performed. Further, if the image is captured with a composition close to the horizontal plane image 500 (overhead view or the like), the processing of S203 does not necessarily need to be performed. In this case, it is possible to determine whether the worker is a single worker or a co-worker using the distance on the captured image. Also, the horizontal plane image 500 may be generated by any of various other methods.

Next, the control unit 121 updates the list of the detected persons (S203). If it is the beginning of the repetition processing, the control unit 121 lists all the detected persons. At this time, the control unit 121 stores, in each node in the list, the coordinates of the representative point on the captured image where the person is detected and the coordinates of the representative point on the horizontal plane image. If it is the second time or more of the repetition processing, the control unit 121 performs the following processing for all the detected persons.
(Process 1). The control unit 121 calculates the distance between the coordinates on the captured image of the person stored in each node in the list and the coordinates on the captured image of the own device. Replace with data. At this time, the control unit 121 stores the history of the coordinates stored in the node up to now.
(Process 2). If there is no node that satisfies the condition in (Process 1), the control unit 121 adds its own data to the end of the list.
In addition, the control unit 121 performs the following processing after finishing the above processing for all the detected persons.
(Process 3). The control unit 121 deletes the node that has not been updated.

Next, the control unit 121 determines whether or not a single worker exists in the captured image (S204 to S206). In the present embodiment, as illustrated in FIG. 7A, the control unit 121 sets the work area frames (501 to 506) in the captured image display based on a user operation or the like. The processing flow is as follows.
(Process 4). The control unit 121 checks whether or not the coordinates on the captured image of the person to be processed are included in the work area frame in order from the top of the list. Stores a flag indicating that it is in At this time, when there are a plurality of work areas, the control unit 121 stores in the list a unique ID number assigned to each work area frame (S204).
(Process 5). The control unit 121 searches the list and confirms, for each work area frame, how many workers belong to the area (S205). When the worker is 1 in the work area frame, the control unit 121 determines that the single work is performed in that area, and otherwise determines that the single work is not performed (S206). . That is, when the worker is 1 in the work area frame, the control unit 121 determines that the worker in that area is a single worker who is not associated with another person. When there are two or more workers in the work area frame, the control unit 121 associates the workers in the area with each other and determines that the worker is a joint German worker. As described above, the control unit 121 has a function as an association unit.
In the present embodiment, the control unit 121 first determines whether or not each person is within the work area frame based on the list of detected persons, but determines whether each person is within the work area frame based on each work area frame. After confirmation, the determination may be made only by the number of persons in the work area.
Also, the description has been made assuming that the work area frame is set in the captured image display. However, as illustrated in FIG. 7B, the control unit 121 sets the work area frame in the horizontal plane image based on a user operation. You may.

Here, the processing of determining whether a worker is a single worker or a joint worker is not limited to the above-described processing. For example, the control unit 121 associates the persons based on at least one of the position of the person detected from the captured image, the direction of the face of the person, and the distance between the persons, and determines that the person is a co-worker. The determination may be made. For example, the control unit 121 may associate persons located within a predetermined range from a predetermined object of the captured image, for example, a workbench, or may estimate the face direction (or eyes) of the person, The persons whose face directions (or eyes) intersect may be associated with each other. Further, for example, the control unit 121 may associate the persons whose distance between the persons is within a predetermined distance. Then, the control unit 121 may determine a person associated with another person as a co-worker and a person not associated with another person as a single worker.
Further, the control unit 121 may further associate the persons based on the color or the shape of the uniform, the bibs, or the hat of the persons. For example, the control unit 121 associates persons with the same uniform shape among the persons whose distance between the persons is within a predetermined distance. May not be associated with each other. As described above, the control unit 121 may associate the person based on the feature amount of the thing worn by the person.
Further, even when there are a plurality of workers in the work area frame, the control unit 121 may change the face directions of the plurality of workers in the work area frame or the distance between the workers. Alternatively, it may be determined that the operator is a single worker.

Next, the control unit 121 determines whether or not a single operation is being performed, and if the condition is satisfied, records the log as the single operation and notifies the user (S207 to S209). The processing flow is as follows.
(Process 6). When there is a work area frame in which the independent work is being performed in (Process 5), the control unit 121 increments the number of cumulative frames of the single work, that is, increases the value of the variable by one, and at the same time, The person belonging to the work area is extracted from the list (S207). When the independent operation is not performed, the control unit 121 sets the accumulated number of independent operation frames to 0 (S208).
(Process 7). When the number of cumulative frames of the single work exceeds the threshold, the control unit 121 determines that the single work is being performed (S209). By using this determination condition, it is possible to suppress, for example, an erroneous notification of being a sole worker due to a momentary failure in detecting a person despite being a joint worker.
When the control unit 121 determines that the single work is performed, the control unit 121 displays the display device 140 so that the occurrence time and duration of the work, the imaging device 110 in which the work is occurring, and a location in the captured image where the single work is performed are known. (S210). The screen in FIG. 8A is an example of a screen showing a list of moving images in a time zone in which an independent operation is detected. The text information “single work detection list” and the time, duration, and location information in FIG. 8A are examples of information that can be identified as a moving image including a person that is not associated with another person. It is. In some applications, a list of time-series moving images that do not include a person that is not associated with another person may be displayed. In that case, for example, text information of "cooperative work detection list" is displayed.
Next, the control unit 121 determines whether or not to repeat the processing from the acquisition of the captured image (S200) to the determination of whether or not to notify the user of the presence / absence of a sole trader (S210, S211) (S212). The control unit 121 determines that the repetition processing is to be ended when it becomes impossible to obtain a captured image from the imaging device or when there is an instruction from the user to interrupt the repetition processing.

In the display in S210, the control unit 121 displays the information (list) on the independent work together with the thumbnail image as shown in FIG. 8A, but may display it only in the list format (list). Also, the control unit 121 may display a timeline together with the display of the recorded video as shown in FIG. 8B, and may display on the timeline the time zone of the independent work. The part of the time line (black or hatched part) shown as the time slot of the independent work in FIG. 8B is an example of information that can identify that a person who is not associated with another person is included. . In addition, for example, a black portion is a time zone in which there is one single worker, and a hatched portion is a time zone in which there are a plurality of single workers. “A-1” is information indicating an ID or the like for specifying the imaging device. In the example of FIG. 8B, the time lines of the two imaging devices “A-1” and “B-1” are shown.
On the other hand, in the timeline portion (white portion) represented as a time zone other than the time zone of the independent work in FIG. 8B, it can be identified that a person who is not associated with another person is not included. It is an example of information. Further, the control unit 121 may display the time zone of the single work detected by a plurality of cameras on one timeline. As described above, at least, on the timeline, information that can be identified as including a person that is not associated with another person, and information that can be identified as not including a person that is not associated with another person, May be displayed. It suffices if the time zone in which the moving image includes a person who is not associated with another person and the time zone in which the person not associated with another person is not included.
Further, for example, as shown in FIG. 9A, the user designates the display of each individual work with a mouse, a finger, or the like. Then, the control unit 121 reproduces the recorded video of the corresponding imaging device on the display device 140 via the display control unit 125 as shown in FIG. 9B, and displays the captured image at that time. You may make it possible to confirm. By doing so, user convenience is enhanced. At this time, the control unit 121 may play back only the recorded video at the time of occurrence of the independent work, but may start playback of the video from the time before the occurrence of the independent work by a predetermined time (for example, several tens of seconds) or continue the independent work. It is more preferable that the reproduction is performed for a time longer than the time because the situation before and after the reproduction can be easily confirmed.
In addition, the control unit 121 may connect the recorded images in the time zone detected as the independent work and reproduce the digest video. That is, the moving images in the list shown in FIG. 9A may be continuously reproduced. At this time, the moving image in the time zone of the independent work of about several seconds may not be included in the digest video.
In order to prevent erroneous detection or the like, a notification of the independent work will give an alarm if the independent work continues for a predetermined time. However, the duration of the independent work is the time from the first detected frame. Therefore, the control unit 121 may set the section displayed as a single task on the timeline or the start of recording or playback to be the time at which the single task is started to be detected, instead of the time at which an alarm is issued.

Further, the control unit 121 may display the timeline not only in the single work time zone when the cumulative number of frames in which the single worker exists exceeds the threshold value and an alarm is issued, but also in the following manner. That is, even when the control unit 121 determines that the individual work is performed for a short period of time in which no warning is issued, the control unit 121 confirms both by displaying the color, shape, and pattern different from the single work time zone in which the warning was issued. You may make it possible. In this case, the single work detected only for a moment may actually be any one of the co-workers temporarily hidden by the obstacle, so two thresholds are used for the determination of the single work. Should be provided. At this time, the control unit 121 issues an alert based on the larger one of the two thresholds. For example, the first threshold is used for information that can be identified as including a person not associated with another person. Then, to determine whether to issue an alarm, a second threshold larger than the first threshold is used.
In addition, the control unit 121 may save a list of imaging devices, areas, and time information in which the independent work is detected as a CSV file, and paste a list with thumbnail images, a screen shot of a recorded video, or the like as a report. It may be possible to print.
In addition, in addition to the captured image for detecting the independent work, the control unit 121 similarly includes the following when there is an imaging device that captures an area at a different angle or captures a surrounding area. Processing may be performed. That is, the control unit 121 may display a video captured and recorded by an imaging device in which association has been set in advance.
Further, as shown in FIG. 10A, in the case of a system that detects a single operation in a live video and determines that the single operation is the single operation, in the case of a system that issues an alarm in real time, the operation observer checks the current situation. Need to check the past situation. Therefore, the control unit 121 may perform control to display both the live video and the recorded video. In FIG. 10A, in the video of the room A, by changing the index at the upper right, information that allows identification of the presence of a person who is not associated with another person is displayed.
As illustrated in FIG. 10B, the control unit 121 may delete the selected image or the like from the series of displays when the work monitor determines that there is no false report or problem.

In the present embodiment, the control unit 121 has mainly been described using an example of setting a work area frame. When detecting the independent work, the control unit 121 may display the work area frame at the time of detecting the independent work on the display device 140 so as to be superimposed on the video. Further, the control unit 121 displays all the work area frames superimposed on the video, and when the single work is performed, the work area frame in which the single work was performed is different in color from the other work area frames. , Or may be displayed explicitly or in an emphasized manner, for example, by blinking. By doing so, the work monitor can recognize whether the person in the video is single or plural with good visibility.
Further, in the present embodiment, an example has been described in which a single client device detects and displays a person or the like, but a plurality of client devices may perform processing. For example, a first client device for inputting an image and detecting and notifying an independent work or the like, and a second client device for receiving the notification and outputting an alarm, or displaying a list or displaying on a timeline. The processing may be performed by the client device. An example of the first client device is a VCA (Video Content Analysis) device. An example of the second client device is a VMS (Video Management System) device. The hardware configuration of the first client device and the second client device is the same as the hardware configuration of the client device 120 in FIG. The function of each client device is realized by the CPU of each client device executing processing based on a program stored in the memory of each client device.

  FIG. 11A shows an example in which the system of the present embodiment and the entry / exit management system 160 cooperate. In the entry / exit management system, times when an operator enters and exits the work room are recorded. Therefore, as shown in FIG. 11B, the control unit 121 acquires information on the time of entry and exit of each worker from the entry / exit management system 160 and, based on the acquired information, the work entering the work room. People. Then, the control unit 121 may record the extracted list of worker names as a part of the information on the list of occurrences of the single work, or may display the list on the screen. However, the present invention is not limited to this, and the control unit 121 may send a mail to the user. This recording and display may be realized by the system of the present embodiment, or may be incorporated in an entry / exit management system or the like. The extracted list of worker names is an example of the entry / exit information.

<Embodiment 2>
In the present embodiment, the configuration is the same as that of the first embodiment, and an explanation will be given using an example in which the determination as to whether or not a single worker is present is performed based on a distance instead of a detection based on a work area frame. The flow of the process according to the present embodiment will be described with reference to FIG.
The processing from when the control unit 121 acquires the captured image from the imaging device 110 (S1101) to when the detected person list is updated (S1103) is the same as the processing described in the first embodiment.
Next, the control unit 121 determines whether or not a single worker exists in the captured image (S1104 to S1106). The processing flow is as follows.
(Process 1). The control unit 121 sets the first person in the list as a processing target (S1104).
(Process 2). The control unit 121 determines whether the person to be processed is a single worker (S1105). In the present embodiment, the control unit 121 checks the following items with respect to the person to be processed in order to determine whether or not the worker is a single worker.

(Item 1). The control unit 121 determines whether or not there is a person detected within a radius R with the center on the horizontal plane image of the person to be processed.
This radius R may be appropriately set according to the situation to which the present embodiment is applied. For example, when monitoring indoor work, the radius R can be a distance corresponding to 2 m. However, at this time, when the control unit 121 refers to the coordinate history stored in the node of the person to be processed and determines that the person to be processed is moving, The value of the threshold value R is set larger than that of the threshold value. Further, at this time, if the co-workers work together in the x-axis direction on the horizontal plane image, the control unit 121 may use an ellipse obtained by reducing a circle having a radius R in the y-axis direction. Similarly, if the co-workers work side by side in the y-axis direction on the horizontal plane image, the control unit 121 may use an ellipse obtained by reducing a circle having a radius R in the x-axis direction.
(Item 2). When there is a person who satisfies (item 1), the control unit 121 determines whether the person exists in the detection area (work area frame).
In the present embodiment, the control unit 121 operates the input device 130 or the like to set an area specified by the user as a detection area where the worker exists. Thus, it is possible to prevent a non-worker moving on the other side of a glass wall and a person reflected on a poster or the like placed on a desk from being erroneously detected as a co-worker. However, the setting of the detection area is not limited to the input by the user via the input device 130 or the like. The control unit 121 may automatically set a region surrounded by a marker arranged on the captured image as a detection region. As a method of detecting using a marker, for example, a two-dimensional marker set at each of the four corners of the detection area is obtained by raster-scanning a captured image, and the two-dimensional marker is registered in the client device 120 in advance. There is a method of pattern matching with information.
When all of the above (item 1) and (item 2) are satisfied, the control unit 121 determines that the user is a co-worker, and proceeds to (process 3-1). That is, the control unit 121 associates workers who satisfy all of the above (item 1) and (item 2) with each other and determines that the worker is a joint German worker. If at least one of the conditions is not satisfied, the control unit 121 determines that the person to be processed is a single worker, and proceeds to (Processing 3-2).

  Here, the processing of determining whether a worker is a single worker or a joint worker is not limited to the above-described processing. For example, the control unit 121 may associate the persons based on at least one of the position of the person detected from the captured image, the direction of the face of the person, and the distance between the persons. For example, the control unit 121 may associate persons located within a predetermined range from a predetermined object in the captured image, for example, a workbench, or may estimate the face direction (or eyes) of the person, The persons whose face directions (or eyes) intersect may be associated with each other. Further, for example, the control unit 121 may associate the persons whose distance between the persons is within a predetermined distance. Then, the control unit 121 may determine a person associated with another person as a co-worker and a person not associated with another person as a single worker. Further, the control unit 121 may further associate the persons based on the color or shape of the person's uniform, bibs (vest-like clothes), or hat. For example, the control unit 121 associates persons with the same uniform shape among the persons whose distance between the persons is within a predetermined distance. May not be associated with each other, or a combination of different colors of the bibs within a predetermined distance may be determined by a plurality of conditions such as a co-worker, and other combinations may be determined by a plurality of conditions such as a single worker.

式（４）中のｑｘ、ｑｙはそれぞれ、処理対象としている人物の、水平面画像上でのｘ座標、及びｙ座標を表す。図１３では、ｑｘ、ｑｙはそれぞれ楕円１２０２のｘ座標、及びｙ座標を表す。式（４）中のａ、ｂは楕円の変換パラメータである。例えば、ａ＝１、ｂ＝１とすると、楕円は半径Ｒ、中心（ｑｘ、ｑｙ）の円となる。また、ａ＝１、ｂ＝１／２とすると、ｘ軸方向の長さがＲ、ｙ軸方向の長さがＲ／２となる楕円となる。本実施形態では共同作業者同士が水平面画像上のｘ軸方向に並んで作業を行うため、ａ＞ｂとなる値を用いることとする。 The determination in the above (item 1) and (item 2) will be described with reference to FIG. In FIG. 13, ellipses 1201 to 1205 represent the results of transforming the centers of the ellipses 401 to 405, which are the person detection positions, into coordinates on a horizontal plane image. An area 1206 represents a detection area specified by the user. An ellipse 1200 represents an ellipse composed of x and y that satisfies the following equation (4).

In Equation (4), qx and qy represent the x coordinate and the y coordinate of the person to be processed on the horizontal plane image, respectively. In FIG. 13, qx and qy represent the x coordinate and the y coordinate of the ellipse 1202, respectively. A and b in the equation (4) are elliptic conversion parameters. For example, if a = 1 and b = 1, the ellipse is a circle having a radius R and a center (qx, qy). If a = 1 and b = 1/2, the ellipse has a length in the x-axis direction of R and a length in the y-axis direction of R / 2. In the present embodiment, since the co-workers work together in the x-axis direction on the horizontal plane image, a value that satisfies a> b is used.

  For example, it is assumed that the control unit 121 processes a person on the ellipse 402 at the person detection position during scanning of the list. At this time, the control unit 121 performs the above (item 1) to (item 2) determination on the coordinates of the ellipse 1202 on the horizontal plane image corresponding to the ellipse 402 of the person detection position. First (item 1), the control unit 121 determines whether or not a detected person exists inside the ellipse 1200. In the situation illustrated in FIG. 13, since the coordinates of the ellipse 1201 on the horizontal plane image corresponding to the ellipse 401 at the person detection position are included in the ellipse, the control unit 121 determines that the determination is satisfied. Next, in (item 2), the control unit 121 determines whether the coordinates 601 on the horizontal plane image corresponding to the ellipse 401 of the detected position of the person who satisfies the condition of (item 1) is included in the area 1206. . In the situation described in the example of the present embodiment, since the coordinates of the ellipse 1201 are included in the region 1206, the control unit 121 determines that the determination is satisfied. As described above, since both (item 1) and (item 2) are satisfied, the control unit 121 determines that the person with the ellipse 402 is a co-worker.

(Process 3-1). The control unit 121 determines whether the last person in the list is to be processed (S1106). If it is not the end of the list, the control unit 121 sets the next person as a processing target and returns to (Process 2) (S1107). If the list is at the end of the list, the control unit 121 ends the scanning in the list, and sets the cumulative number of frames in which a single worker exists to 0 (S1108).
(Process 3-2). The control unit 121 increments the number of accumulated frames in which a single worker exists (S1109).
Next, the control unit 121 determines whether or not to notify the user that a single worker exists, and notifies the user if the condition is satisfied (S1110, S1111).
The control unit 121 performs the same processing as in the first embodiment for the display processing and the like at the time of notification.
In the present embodiment, the control unit 121 displays a UI indicating that a single worker exists on the display device 140 as a method for notifying the user that a single worker exists. For example, the control unit 121 superimposes a double circle or a red circle on the position of a single worker on the captured image 300 or the horizontal plane image 500. However, the present invention is not limited to this, and the control unit 121 may send an e-mail to the user, output to an external output, and issue an alert from a speaker.

<Other embodiments>
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or an apparatus via a network or a storage medium. The present invention can also be realized by processing in which one or more processors in a computer of the system or the apparatus read and execute a program. Further, it can be realized by a circuit (for example, an ASIC) that realizes one or more functions.

As described above, the preferred embodiment of the present invention has been described in detail, but the present invention is not limited to the specific embodiment. The functional configuration of the imaging device 110 may be implemented in the imaging device as a software configuration realized by the CPU 12 executing a process based on a program stored in the memory 13. In addition, a part or all of the functional configuration of the imaging device 110 may be implemented in the imaging device 110 as a hardware configuration. Similarly, the functional configuration of the client device 120 may be implemented in the client device 120 as a software configuration realized by the CPU 22 executing a process based on a program stored in the memory 21. Further, a part or all of the functional configuration of the client device 120 may be implemented in the client device 120 as a hardware configuration.
In addition to superimposing a double circle at the position of a single worker or superimposing a red circle, a character string such as “single worker” may be superimposed on a single worker, or a warning color such as red may be displayed. You may make it superimpose on a single worker on a captured image. Also, these may be arbitrarily combined and displayed in a superimposed manner.
The hardware configurations of the client device and the imaging device described above are merely examples, and for example, a plurality of CPUs, memories, communication I / Fs, and the like may be provided.

  As described above, according to the processing of each of the above-described embodiments, it is possible to appropriately indicate whether the person in the image is a single person or a plurality of persons. Being notified of the presence of a single worker at the work site and being able to easily check the video, the work observer can find a worker who violates the rules and pay attention to that worker. It will be possible to encourage.

21 Memory 22 CPU
120 client device 140 display device

Claims (14)

Detecting means for detecting a person from the moving image ;
Setting means for setting an area for specifying whether the number of persons detected by the detection means is one on the moving image,
Identification means for identifying whether the number of persons detected by the detection means is one in the area set by the setting means,
Display control means for controlling to display a timeline related to the moving image,
Has,
The specifying means further specifies whether or not the duration in which the number of persons detected by the detection means in the area is one exceeds a predetermined time,
The display control means, the time period of the continuation time specified as exceeding the predetermined time by the specifying means, and the time period of the continuation time specified as not exceeding the predetermined time by the specifying means, A control device for controlling the display so that information capable of identifying the information is displayed on the timeline .   Detecting means for detecting a person from the moving image;
  Setting means for setting an area for specifying whether the number of persons detected by the detection means is one on the moving image,
  Identification means for identifying whether the number of persons detected by the detection means is one in the area set by the setting means,
  Display control means for controlling to display a timeline related to the moving image;
  Has,
  The setting means can set a plurality of the area on the moving image,
  The display control unit is configured to determine that the number of people detected by the detection unit is one, that the time period in which a plurality of areas specified by the specification unit exist, and that the number of people detected by the detection unit be one. A control device which controls so as to display, on the time line, information capable of identifying a time zone in which a single area specified by the specifying means exists. The specifying means further specifies whether or not the duration in which the number of persons detected by the detection means in the area is one exceeds a predetermined time,
It said display control means is a area number detected by said detecting means is specified by the specifying means as the one, and the duration is identified as exceeding the predetermined time area more The time zone that exists and the area specified by the specifying means that the number of persons detected by the detecting means is one, and the area specified when the duration exceeds the predetermined time are singular. The control device according to claim 2, wherein control is performed such that information capable of identifying an existing time zone is displayed on the timeline . Further comprising an acquisition unit for acquiring a moving image from the imaging device,
The detecting device, the control device according to any one of claims 1 to 3, characterized in that a person is detected from the moving image acquired by the acquisition unit. The control device according to any one of claims 1 to 4 , wherein the detection unit detects a person from the moving image based on a feature amount of the person. Detecting means for detecting a person from the moving image;
Setting means for setting an area where a person works on the moving image;
Identification means for identifying whether the number of persons detected by the detection means in the area set by the setting means is one ,
By the specifying means, the number of people detected by said detection means in the area, is performed on the condition that has been identified as one, working alone in the area must be performed to work a plurality of persons A control device, comprising: display control means for controlling information such that a user can recognize that a person is present on a display unit. The specifying means further specifies whether or not the duration in which the number of persons detected by the detection means in the area is one exceeds a predetermined time,
  The display control unit further controls a timeline related to the moving image to be displayed on the display unit, and the time period of the continuation time specified by the specifying unit to be longer than the predetermined time; 7. The control device according to claim 6, wherein information is displayed on the time line to identify the time period of the continuation time specified as not exceeding the predetermined time by the means. . The setting means can set a plurality of the area on the moving image,
  The display control unit further controls a timeline related to the moving image to be displayed on the display unit, and an area specified by the specifying unit that the number of persons detected by the detection unit is one is determined. Information that can identify a plurality of time zones and a time zone in which the number of persons detected by the detection unit is one and the area specified by the specification unit is singular is displayed on the time line. The control device according to claim 6, wherein the control is performed in such a manner. The specifying means further specifies whether or not the duration in which the number of persons detected by the detection means in the area is one exceeds a predetermined time,
  The display control means may include an area specified by the specifying means that the number of persons detected by the detection means is one, and a plurality of areas specified if the duration exceeds the predetermined time. The time zone that exists and the area specified by the specifying means that the number of persons detected by the detecting means is one, and the area specified when the duration exceeds the predetermined time are singular. The control device according to claim 8, wherein control is performed such that information capable of identifying an existing time zone is displayed on the timeline. Before Symbol display control means, said by the specific means, that the is detected by the detection means persons identified as not one person on the condition in the area, the information, so as not to display on the display unit control The control device according to claim 6, wherein: A detection step of detecting a person from the moving image ;
A setting step of setting an area for specifying whether the number of persons detected in the detecting step is one on the moving image,
In the area set in the setting step, a specifying step of specifying whether the number of persons detected in the detection step is one ,
A display control step of controlling to display a timeline related to the moving image,
Including
In the specifying step, it is further specified whether or not the duration of the number of persons detected by the detection step in the area is one exceeds a predetermined time,
In the display control step, the time period of the continuation time specified as exceeding the predetermined time in the specifying step, and the time period of the continuation time specified as not exceeding the predetermined time in the specifying step, A control method characterized in that information is displayed on the timeline so that the information can be identified .   A detection step of detecting a person from the moving image;
  A setting step of setting an area for specifying whether the number of persons detected by the detection step is one on the moving image;
  A specifying step of specifying whether the number of persons detected by the detection step is one in the area set by the setting step;
  A display control step of controlling to display a timeline related to the moving image;
  Including
  In the setting step, a plurality of areas can be set on the moving image,
  In the display control step, when the number of people detected in the detection step is one, the time zone in which a plurality of areas specified in the specific step exist, and the number of people detected in the detection step is one. A control method, wherein control is performed such that information capable of identifying a time zone in which a single area specified in the specifying step exists is displayed on the time line. A control method executed by a control device,
A detection step of detecting a person from the moving image;
A setting step of setting an area where a person works on the moving image;
A specifying step of specifying whether the number of persons detected in the detection step in the area set in the setting step is one ,
In the specifying step, the number of the detected in the detection step in the area, is performed on the condition that has been identified as one, working alone in the area must be performed to work a plurality of persons A display control step of controlling information that can be recognized by the user that a person is present on the display unit;
A control method comprising: A program for causing a computer to function as each unit of the control device according to any one of claims 1 to 10 .

Images