JP2023177871A - Learning device and learning method - Google Patents

Abstract

To provide a learning device capable of generating a highly accurate learning model while allowing a user to easily and accurately determine the occurrence status of notable events without being greatly influenced by the user's subjectivity, by efficiently collecting properly labeled training target images, and a learning method.SOLUTION: In a monitoring system, image management server 3 generates a learning target image in which a determination reference image that is used as a reference when determining an entry of an object into a predetermined area is superimposed and synthesized with a picked-up image. A monitoring terminal 4 presents a learning target image to a user, and based on user operation, acquires information on the occurrence status of a notable event that appeared in the learning target image as label information. An image analysis server 2 generates a learning model by machine learning using the learning target image and the label information as learning information.SELECTED DRAWING: Figure 4

Description

The present invention relates to a learning device and a learning method that generate a learning model for detecting a state in which an object enters a predetermined area set on a photographed image as an event of interest based on a photographed image of a monitoring area. .

BACKGROUND ART Systems that detect abnormal events that occur in a monitoring area based on captured images of the monitoring area are widely used. Furthermore, in recent years, systems that detect abnormal events using learning models generated by machine learning such as deep learning have also been used.

In a system that uses a learning model to detect an abnormal event, a learning model is created before operation by performing machine learning using a learning target image (teacher image) based on a captured image of a monitoring area. Also, during operation, detection results of abnormal events can be obtained by inputting detection target images based on captured images of the monitoring area to the learning model. Furthermore, during operation, if additional learning is performed using a detection target image used for abnormal event detection as a learning target image, the accuracy of the learning model can be improved at any time.

Conventionally, as a technology for performing additional learning using such a detection target image as a learning target image, the detection target image is presented to the user as a candidate for the learning target image in additional learning, and the detection target image is used for additional learning. There is a known technique in which a user visually determines the suitability of an image as a learning target image and selects a learning target image for additional learning (see Patent Document 1).

Japanese Patent Application Publication No. 2018-205900

Now, according to the conventional technology, a detection target image as a candidate for a learning target image in additional learning is added with a detection result (correctness determination result) as to whether a predetermined event (appearance of a target object) has been detected. , presented to the user. By visually observing the detection target image, the user determines whether the image is appropriate as a learning target image depending on whether the detection result is correct or not, and selects the learning target image for additional learning.

However, the problem with conventional technology is that when a detection target image, which is an image taken in a surveillance area, is simply visually inspected, the determination of whether or not it is suitable as a learning target image is largely influenced by the user's subjectivity, resulting in variations in the determination results. was there. Furthermore, when detecting the entry of an object (for example, a person) into a predetermined area as an abnormal event, it is difficult to determine the occurrence of the abnormal event, and it is difficult to judge the occurrence of the abnormal event by simply looking at the detection target image, which is an image taken in the monitoring area. , it is not possible to easily and accurately determine the occurrence status of an abnormal event. For this reason, there is a problem in that the labeling that adds label information regarding the occurrence status of an abnormal event to the learning target image becomes inaccurate, and a highly accurate learning model cannot be created.

Therefore, the present invention enables a user to easily and accurately judge the occurrence status of an event of interest without being greatly influenced by the user's subjectivity, and efficiently collect appropriately labeled learning target images. The main objective is to provide a learning device and a learning method that can generate highly accurate learning models.

The learning device of the present invention uses a processor to generate a learning model for detecting, as an event of interest, a state in which an object enters a predetermined area set on the photographed image, based on a photographed image of a monitoring area. In the learning device, the processor generates a learning target image by superimposing and synthesizing a determination reference image, which is a reference for determining entry of a target object into the predetermined area, on the photographed image; Machine learning that is presented to a user, acquires information regarding the occurrence status of the noted event that appears in the learning target image as label information based on the user's operation, and uses the learning target image and the label information as learning information. The learning model is generated by:

Furthermore, the learning method of the present invention includes a process of generating a learning model for detecting, as an event of interest, a state in which an object enters a predetermined area set on the photographed image, based on the photographed image of the monitoring area. In this learning method, a learning target image is generated by superimposing and synthesizing a determination reference image, which is a reference for determining entry of an object into the predetermined area, on the photographed image, and the learning target image is presented to the user. Based on the user's operation, information regarding the occurrence status of the noted event that appeared in the learning target image is acquired as label information, and machine learning is performed using the learning target image and the label information as learning information. The configuration is to generate a learning model.

According to the present invention, the learning target image is obtained by superimposing and synthesizing the determination reference image on the photographed image of the monitoring area. Therefore, the user can easily and accurately judge the occurrence status of the event of interest without being greatly influenced by the user's subjectivity, efficiently collect appropriately labeled learning target images, and perform high-precision training. A learning model can be generated.

Overall configuration diagram of a monitoring system according to this embodiment Explanatory diagram showing the judgment line and judgment area set on the image taken by the camera Explanatory diagram showing images processed by this system before operation Explanatory diagram showing an overview of the processing performed by this system before operation Explanatory diagram showing images processed by this system during operation Block diagram showing an overview of the processing performed by this system during operation Block diagram showing the schematic configuration of the image analysis server, image management server, and monitoring terminal Explanatory diagram showing the monitoring screen displayed on the monitoring terminal Explanatory diagram showing the processing target setting screen displayed on the monitoring terminal Explanatory diagram showing the camera confirmation screen displayed on the monitoring terminal Explanatory diagram showing the processing condition setting screen displayed on the monitoring terminal Explanatory diagram showing the processing condition setting screen displayed on the monitoring terminal Explanatory diagram showing the notification screen displayed on the monitoring terminal Explanatory diagram showing the notification screen displayed on the monitoring terminal Explanatory diagram showing the status confirmation screen displayed on the monitoring terminal Explanatory diagram showing the status confirmation screen displayed on the monitoring terminal Explanatory diagram showing the status confirmation screen displayed on the monitoring terminal Explanatory diagram showing the status confirmation screen displayed on the monitoring terminal Explanatory diagram showing the status confirmation screen displayed on the monitoring terminal Explanatory diagram showing the status confirmation screen displayed on the monitoring terminal Explanatory diagram showing the learning model creation screen displayed on the monitoring terminal Explanatory diagram showing the rollback settings screen displayed on the monitoring terminal

A first invention made to solve the above problem is a learning model for detecting, as an event of interest, a state in which an object enters a predetermined area set on the photographed image, based on the photographed image of the monitoring area. A learning device in which a processor performs processing to generate a learning target image, wherein the processor generates a learning target image obtained by superimposing and synthesizing a determination reference image, which is a reference for determining entry of an object into the predetermined area, onto the photographed image. the learning target image is presented to the user, and based on the user's operation, information regarding the occurrence status of the noted event appearing in the learning target image is acquired as label information, and the learning target image and the label information are acquired. The learning model is generated by machine learning using as learning information.

According to this, the learning target image is obtained by superimposing and synthesizing the determination reference image on the photographed image of the monitoring area. Therefore, the user can easily and accurately judge the occurrence status of the event of interest without being greatly influenced by the user's subjectivity, efficiently collect appropriately labeled learning target images, and perform high-precision training. A learning model can be generated.

Further, in a second invention, the processor generates a detection target image by superimposing and synthesizing the determination reference image on the captured image in real time, and detects the event of interest from the detection target image using the learning model. When processing is performed and the event of interest is detected, the detection target image is set as the learning target image in additional learning, the learning target image is presented to the user, and based on the user's operation, the The configuration is such that label information is acquired and the learning model for updating is generated by machine learning using the learning target image and the label information as learning information for additional learning.

According to this, since the detection target image used in the process of detecting the event of interest becomes the learning target image for additional learning as it is, additional learning can be performed efficiently.

In a third aspect of the present invention, the processor sets the detection target image designated by the user as a target for additional learning as the learning target image in additional learning, based on a user's operation.

According to this, since the detection target image designated by the user is set as the target of additional learning, additional learning can be performed more appropriately.

In a fourth aspect of the present invention, the processor generates a learning model for detecting, as the event of interest, a state in which an object enters the predetermined area corresponding to a dangerous area in which the object moves.

According to this, a state in which a target object approaches a dangerous area in which the object is moving can be detected as an event of interest. Note that the dangerous area where an object moves is, for example, a device (conveyor) for transporting luggage or a track on which a train runs.

In a fifth aspect of the present invention, the processor superimposes an image representing a linear determination line on the photographed image as the determination reference image.

According to this, by setting a linear determination line as a criterion for detecting an event of interest, it is possible to appropriately detect an event of interest. Therefore, it is possible to detect an event of interest in which an object approaches in a manner that intersects the determination line, distinguishing it from other events of interest. Note that the number of determination lines may be one or more than one.

In a sixth aspect of the invention, the processor superimposes an image representing a polygonal determination area on the photographed image as the determination reference image.

According to this, by setting a polygonal determination area as a criterion for detecting an event of interest, it is possible to appropriately detect an event of interest.

The seventh invention also provides a seventh invention that uses a processor to generate a learning model for detecting, as an event of interest, a state in which an object enters a predetermined area set on the photographed image, based on the photographed image of the monitoring area. The learning method includes: generating a learning target image by superimposing and synthesizing a determination reference image, which is a reference for determining entry of an object into the predetermined area, onto the photographed image, and presenting the learning target image to a user. Based on the user's operation, information regarding the occurrence status of the noted event appearing in the learning target image is acquired as label information, and the learning is performed by machine learning using the learning target image and the label information as learning information. The configuration is to generate a model.

According to this, similar to the first invention, the user can easily and accurately judge the occurrence situation of the event of interest without being greatly influenced by the user's subjectivity, and can select the appropriately labeled learning target image. It is possible to collect information efficiently and generate a highly accurate learning model.

Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is an overall configuration diagram of a monitoring system according to this embodiment.

This system monitors abnormal events (events of interest) occurring in a monitoring area based on captured images of the monitoring area. This system includes a camera 1, an image analysis server 2 (learning device), an image management server 3 (learning device), and a monitoring terminal 4 (terminal device). The camera 1, image analysis server 2, image management server 3, and monitoring terminal 4 are connected via a network such as the Internet or a closed network.

Camera 1 photographs the monitoring area.

The image analysis server 2 performs image analysis processing using a learning model generated by machine learning such as deep learning, detects abnormal events that occur in the monitoring area, and uses the monitoring terminal 4 to detect abnormal events. Notify the user (monitor). Furthermore, the image analysis server 2 generates a learning model using the learning target images (teacher images) accumulated by the image management server 3.

The image management server 3 accumulates and manages learning target images (teacher images) necessary for generating a learning model used by the image analysis server 2. The learning target image is generated from the image taken by the camera 1.

A monitoring screen is displayed on the monitoring terminal 4. Images captured in real time by the camera 1 are displayed on the monitoring screen. This allows the user (monitor) to check the current status of the monitoring area. Further, a notification screen is displayed on the monitoring terminal 4. This allows the user to quickly recognize that an abnormal event has occurred in the monitoring area. Further, on the monitoring terminal 4, the user (administrator) performs operations related to setting conditions for processing performed by the image analysis server 2 and the image management server 3.

Note that the monitoring terminal 4, the image management server 3, and the image analysis server 2 can be configured on-premises, that is, installed and operated on the premises of the facility, but the image management server 3 and the image analysis server 2 are It may also be operated in the cloud.

Further, in this embodiment, various processes are performed in the image management server 3 and the image analysis server 2, but these processes may be performed in a single server. Further, the processing performed by the image management server 3 and the image analysis server 2 may be shared among a plurality of servers in a different combination from this embodiment.

Next, the criteria for detecting abnormal events in the monitoring area will be explained. FIG. 2 is an explanatory diagram showing a determination line and a determination area set on an image taken by the camera 1.

In this embodiment, the monitoring area is a workplace where a conveyor is installed. In a workplace, when a person approaches a conveyor, accidents may occur, such as hands being caught or clothing being pulled in. Therefore, in this embodiment, a state in which there is a high risk of an accident occurring due to a person approaching the conveyor is detected as an abnormal event.

In this embodiment, as shown in FIG. 2(A), a linear determination line is set on the image captured by the camera 1 as a determination criterion. In this example, two determination lines, first and second, are set. The first determination line is set at a position close to the conveyor (hazardous material), and the second determination line is set at a position away from the conveyor. Note that the determination line is a boundary line that separates a predetermined area (dangerous area) into which a person's body is prohibited from entering and other areas (safety area).

Here, there are two states: a state in which the body of the person (object) intersects (contacts) both the first and second determination lines (first intersecting state), and a state in which the body of the person intersects only the second determination line (the first intersecting state). contact) and does not intersect (contact) the first judgment line (second crossing state), and a state in which the person's body does not intersect (contact) either the first or second judgment line (non-crossing state). condition).

In this embodiment, the first intersecting state and the second intersecting state are detected as abnormal events, and notification is performed. The first determination line corresponds to a "danger" notification level, and the second determination line corresponds to a "caution" notification level. That is, a state in which a person's body intersects (contacts) both the first and second determination lines (first intersecting state) is highly dangerous, and therefore a "danger" notification is performed. On the other hand, a state in which the person's body crosses (contacts) only the second judgment line and does not cross (contact) the first judgment line (second crossing state) is of low risk, so it is not recommended to be careful. An announcement will be made.

Note that only one determination line may be set, or three or more determination lines may be set.

Further, the first and second determination lines are drawn in different colors. For example, the first determination line is drawn in red, and the second determination line is drawn in yellow. Thereby, the user can easily visually determine the state of intersection (contact) of the person's body with the first and second determination lines.

Further, as shown in FIG. 2(B), a determination area may be set on the image taken by the camera 1 as a determination criterion. The boundary line (determination line) of the determination area has a polygonal shape. In addition, the determination area is, for example, the inner area surrounded by the boundary line of a polygon is a predetermined area (dangerous area) where the body of a person is prohibited to enter, and the outer area is a area where the presence of a person is prohibited. This is a permissible area (safety area).

Further, a plurality of determination areas may be set. For example, another determination area may be set outside the determination area shown in FIG. 2(B) so as to surround it. In this case, the inner determination area may correspond to the "danger" notification level, and the outer determination area may correspond to the "caution" notification level.

In this embodiment, the monitoring area is a workplace where a conveyor (conveying device) is installed, but the monitoring area is not limited to this. For example, the monitoring area may be a platform and tracks at a railway station. In this case, a situation in which there is a high risk of an accident occurring, such as a user falling from the platform onto the tracks or colliding with a train, is detected as an abnormal event and notified.

Furthermore, in the present embodiment, an abnormal event (event of interest) occurring in the monitoring area is detected based on a photographed image of the monitoring area, but the detected event of interest is not limited to an abnormal event.

Next, an overview of the processing performed by this system before operation will be explained. FIG. 3 is an explanatory diagram showing images processed by this system before operation. FIG. 4 is an explanatory diagram showing an overview of the processing performed in this system before operation.

In this embodiment, two determination lines, a first and a second, are set on an image taken by the camera 1, and a state in which a person's body crosses (contacts) the first and second determination lines (the first a crossing condition, a second crossing condition) is detected as an abnormal event. The process of detecting abnormal events is performed by image analysis using a learning model generated by machine learning such as deep learning.

In this system, processing to generate a learning model (main learning) is performed before operation. In this embodiment, a learning model is generated by learning using a learning target image (teacher image) generated from an image taken by the camera 1.

At this time, as shown in FIG. 3, a determination reference image (an image representing a determination line), which is a reference for determining the presence or absence of an abnormal event, is superimposed and synthesized on the image taken by the camera 1 that photographed the monitoring area. A learning target image is generated. The generated learning target image is displayed on the annotation screen on the monitoring terminal 4.

The user (administrator) visually observes the displayed learning target image on the annotation screen, determines the state of the monitoring area (first intersecting state, second intersecting state, non-intersecting state), and selects the monitored area. Enter label information regarding the status of (annotation work). At this time, since the judgment reference image is superimposed on the learning target image, the user can easily and accurately judge the occurrence of abnormal events by visually observing the learning target image, which reduces the user's annotation work. It can be made more efficient.

As shown in FIG. 4, in the present embodiment, the image management server 3 performs a process of superimposing a determination reference image on an image captured by the camera 1 to generate a learning target image (image synthesis process). Further, in the image management server 3, labeling is performed in which label information regarding the state of the monitoring area (first intersecting state, second intersecting state, non-intersecting state) is added to the learning target image (annotation processing). Furthermore, in the image management server 3, the learning target image and label information are registered in the learning target image database.

Next, the image analysis server 2 acquires the learning target images and label information stored in the image management server 3 from the image management server 3, and generates a learning model by machine learning using the learning target images and label information. (learning model generation process).

Next, an overview of the processing performed by this system during operation will be explained. FIG. 5 is an explanatory diagram showing images processed by this system during operation. FIG. 6 is an explanatory diagram showing an overview of the processing performed by this system during operation.

During operation, as shown in Figure 5, a detection target image is created by superimposing a judgment reference image (an image representing a judgment line), which is a reference for determining the presence or absence of an abnormal event, on an image captured in real time by camera 1. is generated.

As shown in FIG. 6, in the present embodiment, the image analysis server 2 performs a process of superimposing and synthesizing a determination reference image on an image captured by the camera 1 to generate a detection target image (image synthesis process). In addition, the image analysis server 2 performs image analysis processing using a learning model on the detection target image, and based on the analysis results, an abnormal event, that is, a person appears on the first and second judgment lines. An abnormality detection process is performed to determine the presence or absence of a state in which bodies cross (contact) (a first crossed state, a second crossed state).

Here, when an abnormal event is detected, the monitoring terminal 4 notifies the user of the occurrence of the abnormal event. At this time, a notification screen containing notification information corresponding to the degree of risk of the abnormal event that has occurred is displayed, and then a status confirmation screen is displayed. On the status confirmation screen, a detection target image in which an abnormal event has been detected and is the target of notification is displayed (see FIG. 5).

Here, the detection target image in which an abnormal event is detected and is the target of notification becomes the learning target image in additional learning, and the user (administrator) visually checks the displayed learning target image on the status confirmation screen. , the state of the monitoring area (first intersecting state, second intersecting state, non-intersecting state) is determined, and label information regarding the state of the monitoring area is input (annotation work). At this time, since the judgment reference image is superimposed on the learning target image, the user can easily and accurately judge the state of the monitoring area by visually observing the learning target image. User annotation work can be made more efficient.

In the image analysis server 2, labeling is performed to add label information regarding the state of the monitoring area (first intersecting state, second intersecting state, non-intersecting state) to the learning target image (annotation processing). The learning target image and label information are registered in the learning target image database in the image management server 3 as learning information for additional learning.

Next, the image analysis server 2 obtains the learning target images and label information stored in the image management server 3 from the image management server 3, and generates a new learning model using the learning target images and label information. A process is performed to generate a learning model (learning model generation process). Next, the image analysis server 2 performs a process of applying the new learning model to the image analysis process (learning model management process).

Note that the learning model may be updated every time an abnormal event is detected and notified. In this embodiment, when notification is performed, the detection target image in which an abnormal event has been detected and is the target of notification, that is, the learning target image in additional learning, is labeled, so this timing The learning model may be updated using the newly added learning target image.

Next, the schematic configurations of the image analysis server 2, image management server 3, and monitoring terminal 4 will be explained. FIG. 7 is a block diagram showing a schematic configuration of the image analysis server 2, the image management server 3, and the monitoring terminal 4. As shown in FIG.

The image management server 3 includes a communication section 31, a storage section 32, and a processor 33.

The communication unit 31 communicates with the camera 1, the image analysis server 2, and the monitoring terminal 4.

The storage unit 32 stores programs and the like executed by the processor 33. The storage unit 22 also stores registration information of the learning target image database. In the learning target image database, learning target images (teacher images) for each monitoring area state (first intersecting state, second intersecting state, non-intersecting state) are registered.

The processor 33 performs various processes by executing programs stored in the storage unit 32. In this embodiment, the processor 33 performs image composition processing, annotation processing, and the like.

In the image synthesis process, the processor 33 superimposes and synthesizes the judgment reference image (judgment line image, judgment area image) on the captured image of the camera 1 to generate a learning target image (teacher image) used for main learning before operation. do. At this time, image synthesis processing may be performed by acquiring images captured by camera 1 in real time from camera 1, or image synthesis processing may be performed using images captured by camera 1 stored in its own device. Good too.

In the annotation process, the processor 33 provides label information regarding the occurrence status of an abnormal event that appears in the learning target image to the learning target image generated by the image synthesis process in response to the user's input operation performed on the monitoring terminal 4. Perform labeling by adding .

The image analysis server 2 includes a communication section 21, a storage section 22, and a processor 23.

The communication unit 21 communicates with the camera 1, the image management server 3, and the monitoring terminal 4.

The storage unit 22 stores programs executed by the processor 33 and the like. The storage unit 22 also stores information on previously updated learning models.

The processor 23 performs various processes by executing programs stored in the storage unit 22. In this embodiment, the processor 23 performs learning model generation processing, learning model management processing, image synthesis processing, image analysis processing, abnormality detection processing, annotation processing, and the like.

In the learning model generation process, the processor 23 acquires a learning target image (teacher image) from the image management server 3, and uses the learning target image to generate a learning model by machine learning such as deep learning.

In the learning model management process, the processor 23 manages learning models used in the image analysis process. In this embodiment, the processor 23 performs rollback processing to return the learning model used in the image analysis process to the previous learning model specified by the user. Rollback processing is performed when a user determines that a rollback is necessary because the learning model currently in use has many defects, specifically, a large number of false positives, and instructs a rollback. It will be executed if you do so.

In the image synthesis process, the processor 23 acquires real-time captured images from the camera 1, superimposes and synthesizes the captured images with a determination reference image (determination line image, determination area image), and generates a detection target image. .

In the image analysis process, the processor 23 uses the learning model generated in the learning model generation process to perform image analysis on the detection target image obtained in the image synthesis process, and detects an abnormal event, that is, the first and second The accuracy with which a state in which a person's body intersects (contacts) the determination line (first intersecting state, second intersecting state) occurs is obtained. Specifically, a detection target image is input to a learning model, and the accuracy of an abnormal event output from the learning model as an analysis result is obtained.

In the abnormality detection process, the processor 23 detects an abnormal event, that is, a state in which a person's body intersects (contacts) the first and second determination lines (first intersecting state, second intersecting state). At this time, the presence or absence of an abnormal event (first crossing state, second crossing state) is determined based on the accuracy of the abnormal event obtained by the image analysis process, based on the sensitivity specified by the user.

In the annotation process, the processor 23 selects a candidate image for learning in additional learning, that is, a detection target image in which an abnormal event has been detected and is the target of notification, in response to a user's input operation performed on the monitoring terminal 4. Then, labeling is performed to add label information regarding the occurrence status of the abnormal event that appeared in the learning target image.

The monitoring terminal 4 includes a display 41, an input device 42, a communication section 43, a storage section 44, and a processor 45.

The display 41 displays a screen. The input device 42 is a keyboard, a mouse, or the like, and detects a user's input operation.

The communication unit 43 communicates with the image management server 3 and the image analysis server 2.

The storage unit 44 stores programs executed by the processor 45 and the like.

The processor 45 performs various processes by executing programs stored in the storage unit 44. In this embodiment, the processor 45 performs display input control and the like.

In the display input control process, the processor 45 displays various screens such as a monitoring screen (see FIG. 8) on the display 41, and acquires input operation information according to the user's operation of the input device 42. The user's input operation information is transmitted to the image management server 3 and the image analysis server 2.

Next, the monitoring screen 101 displayed on the monitoring terminal 4 will be explained. FIG. 8 is an explanatory diagram showing the monitoring screen 101.

The monitoring screen 101 is provided with a monitoring area selection section 102. The monitoring area selection unit 102 allows the user to select a target monitoring area. In this example, the user can select a workplace as the monitoring area.

Further, the monitoring screen 101 is provided with a camera image display section 103. On the camera image display section 103, captured images (camera images) of a plurality of cameras 1 that capture images of the monitoring area (workplace) selected by the monitoring area selection section 102 are displayed side by side. The camera image display section 103 displays real-time captured images (live images). Thereby, the user (monitor) can visually check the images captured by all the cameras 1 that capture images of the target monitoring area (workplace) and check the current status of the monitoring area.

When the user performs a predetermined operation (right-click) on one of the display fields of the plurality of monitoring areas (workplaces) displayed in the monitoring area selection section 102, a menu 104 is displayed, and the user selects a menu from the menu 104. When a processing target setting is selected, a transition is made to a processing target setting screen 111 (see FIG. 9).

Next, the processing target setting screen 111 displayed on the monitoring terminal 4 will be explained. FIG. 9 is an explanatory diagram showing the processing target setting screen 111.

The processing target setting screen 111 is provided with a camera image display section 112. On the camera image display section 112, captured images (camera images) of a plurality of cameras 1 that capture images of the monitoring area (workplace) selected on the monitoring screen 101 are displayed side by side. In the camera image display section 112, it is possible to set for each camera 1 whether or not it is to be subjected to abnormality detection processing performed by the image analysis server 2.

Specifically, when the user performs a predetermined operation (right-click) on the display column for the name of each camera 1 in the camera image display section 112, a menu 113 is displayed, and the user selects the processing target from the menu 113. You can select either "Not subject to processing" or "Not subject to processing". Here, when a processing target is selected, the corresponding camera 1 is set as a target of the abnormality detection process, and when a non-processing target is selected, the corresponding camera 1 is excluded from the target of the abnormality detection process. In this example, camera 1 #1 is set to be subject to the abnormality detection process, and cameras 1 #2 to #8 are set not to be subject to the abnormality detection process.

Furthermore, when the user selects a target camera 1 by operating one of the display columns for each camera 1 in the camera image display section 112, a camera confirmation screen 121 (see FIG. 10) regarding the selected camera 1 is displayed. Transition.

Further, the processing target setting screen 111 is provided with a "back" button 114. When the user operates the "back" button 114, the screen returns to the monitoring screen 101 (see FIG. 8).

Next, the camera confirmation screen 121 displayed on the monitoring terminal 4 will be explained. FIG. 10 is an explanatory diagram showing the camera confirmation screen 121.

The camera confirmation screen 121 is provided with a camera attribute information display section 122. The camera attribute information display section 122 displays attribute information of the camera 1, specifically, the installation date and time, name, and IP address of the camera 1. This allows the user to check the attribute information of the camera 1.

Further, the camera confirmation screen 121 is provided with a camera image display section 123. The camera image display section 123 displays images taken by the camera 1. Thereby, the user can check the shooting situation of the target workplace by the target camera 1 by visually checking the captured image of the camera 1.

Further, the camera confirmation screen 121 is provided with a button 124 for "obtain camera information". When the user operates the "camera information acquisition" button 124, a process is performed to acquire the attribute information of the camera 1 from the image analysis server 2, and the attribute information of the camera 1 is displayed on the camera attribute information display section 122. , a process of acquiring a photographed image from the camera 1 is performed, and the photographed image of the camera 1 is displayed on the camera image display section 123.

Further, the camera confirmation screen 121 is provided with a "camera confirmation" tab 125 and a "processing condition setting" tab 126. On the camera confirmation screen 121, the "camera confirmation" tab 125 is in a selected state, and when the user operates the "processing condition setting" tab 126, the screen transitions to a processing condition setting screen 131 (see FIGS. 11 and 12).

Next, the processing condition setting screen 131 displayed on the monitoring terminal 4 will be explained. 11 and 12 are explanatory diagrams showing the processing condition setting screen 131.

As shown in FIGS. 11 and 12, the processing condition setting screen 131 is provided with a camera image display section 132. The camera image display section 132 displays images taken by the camera 1.

Further, the processing condition setting screen 131 is provided with a determination criterion setting section 133. The criterion setting section 133 is provided with a criterion selection field 141, a coordinate display field 142, an "add" button 143, a "delete" button 144, and a "setting" button 145.

Here, the example shown in FIG. 11 is a case where a determination line is set as a determination criterion.

In this case, when the user operates the judgment criteria selection field 141, a pull-down menu is displayed, and in the pull-down menu, the user can select the first judgment line (red judgment line) and the second judgment line (yellow judgment line). line). At this time, the transition is made to the judgment line input mode, and the positions of the two end points of the straight line representing the judgment line can be specified on the captured image displayed on the camera image display section 132 using the input device 42. . Note that the first determination line corresponds to the "danger" notification level, and the second determination line corresponds to the "caution" notification level.

The coordinate display field 142 displays the coordinates of the two end points of the straight line representing the determination line. Note that the coordinates of three or more end points may be specified and a polygonal line connecting the specified points may be used as the determination line.

When the user operates the "Add" button 143, a new determination line can be added. When the user operates the "setting" button 144, the determination line specified by the user is set. When the user operates the "delete" button 145, the set determination line can be deleted.

On the other hand, the example shown in FIG. 12 is a case where a determination area is set as a determination criterion.

In this case, when the user operates the judgment criterion selection column 141, a pull-down menu is displayed, and the user selects a judgment area from the pull-down menu. At this time, it is possible to transition to the determination area input mode and use the input device 42 to specify the positions of multiple end points of the polygon representing the determination area on the captured image displayed on the camera image display section 132. can.

The coordinate display column 142 displays the coordinates of a plurality of end points of a polygon representing the determination area.

When the user operates the "Add" button 143, a new determination area can be added. When the user operates the "set" button 144, the determination area specified by the user is set. When the user operates the "delete" button 145, the set determination area can be deleted.

Further, the processing condition setting screen 131 is provided with a mask area setting section 134. The mask area setting unit 134 allows the user to specify a mask area to be excluded from the abnormality detection process. The mask area setting section 134 is provided with a mask area selection field 146, a coordinate display field 147, an "add" button 148, a "delete" button 148, and a "setting" button 150.

When the user operates the mask area selection column 146, a pull-down menu is displayed, and the user can select a mask area from the pull-down menu. At this time, it is possible to transition to the mask area input mode and use the input device 42 to specify the positions of multiple end points of the polygon representing the mask area on the captured image displayed on the camera image display section 132. can.

The coordinate display field 147 displays the coordinates of a plurality of end points of a polygon representing the mask area.

When the user operates the "Add" button 148, a new mask area can be added. When the user operates the "setting" button 149, the mask area specified by the user is set. When the user operates the "delete" button 150, the set mask area can be deleted.

Further, the processing condition setting screen 131 is provided with a sensitivity setting section 135. The sensitivity setting unit 135 allows the user to determine the presence or absence of an abnormal event in a process (abnormality detection process) of detecting an abnormal event, that is, a state in which a person's body crosses (contacts) the first and second determination lines. You can enter the sensitivity value (0 to 100) as a reference value when Note that as the sensitivity value increases, the detection sensitivity of abnormal events increases.

Further, the processing condition setting screen 131 is provided with a "registration" button 136. When the user operates the "Register" button 136, the information entered on this screen is registered as setting information.

Next, the notification screen 161 displayed on the monitoring terminal 4 will be explained. 13 and 14 are explanatory diagrams showing the notification screen 161.

When an abnormal event, that is, a state where a person's body intersects (contacts) the first and second determination lines (first intersecting state, second intersecting state) is detected, the state shown in FIGS. 13 and 14 is detected. A notification screen 161 is displayed as a pop-up on the monitoring screen 101 (see FIG. 8).

On the notification screen 161, one of the characters "danger" and "caution" representing the notification level is displayed depending on the detected abnormal event (first crossing state, second crossing state). The notification screen 161 shown in FIG. 13 is displayed when the notification level is dangerous, that is, when a state in which a person's body intersects both the first determination line and the second determination line is detected. In this case, the text "Danger" is displayed on the notification screen 161. The notification screen 161 shown in FIG. 14 is displayed when the notification level is caution, that is, when a state in which the person's body intersects only the second determination line is detected. In this case, the text "Caution" is displayed on the notification screen 161.

The notification screen 161 is provided with a "confirm" button 162. When the user operates the "confirm" button 162, the screen changes to a status confirmation screen 171 (see FIGS. 15 to 20).

Next, the status confirmation screen 171 displayed on the monitoring terminal 4 will be explained. 15 to 20 are explanatory diagrams showing the status confirmation screen 171.

The status confirmation screen 171 is provided with a camera image display section 172. The camera image display section 172 displays a detection target image in which an abnormal event is detected and is the target of notification. The determination line is superimposed and synthesized on the detection target image. Therefore, by visually observing the detection target image, the user can easily and accurately determine the state of intersection (contact) of the person's body with the first and second determination lines.

Further, the status confirmation screen 171 is provided with an annotation operation section 173. The annotation operation unit 173 allows the user to perform an input operation for labeling candidates for learning target images in additional learning, that is, detection target images in which abnormal events have been detected and are the targets of notification. be. The annotation operation unit 173 is provided with a button 181 for inputting a correct detection, and three buttons 182, 183, and 184 for inputting an actual state in the case of a false detection.

When the user visually observes the detection target image and confirms that the detection is correct, the user operates the button 181. In addition, the user visually inspects the detection target image and determines that there is a false detection and that the person's body is located on both the first judgment line (red judgment line) and the second judgment line (yellow judgment line). After confirming that the user is in a state of intersecting (contacting) (first intersecting state), the button 182 is operated. In addition, the user visually inspects the detection target image and determines that the detection is false and that the person's body intersects (contacts) only the second judgment line (yellow judgment line) and crosses the first judgment line. After confirming that there is no (contact) state (second crossing state), the button 183 is operated. In addition, the user visually inspects the detection target image and confirms that there is a false detection and that the person's body does not intersect (contact) either the first or second determination line (non-intersection state). After confirming, button 184 is operated.

Here, the examples shown in FIGS. 15 to 17 are the status confirmation screen 171 when transitioning from the notification screen 161 (see FIG. 13) where the notification level is "dangerous". Here, the button 182 corresponding to correct detection is displayed in a grayed out state (unselectable).

In the example shown in FIG. 15, in the detection target image displayed on the camera image display section 172, the body of the person is located on both the first judgment line (red judgment line) and the second judgment line (yellow judgment line). is in a state of crossing (contacting) (first crossing state), and the notification level is "dangerous". Therefore, in this case, the user confirms that the detection is correct by visually observing the detection target image, and operates the button 181 for inputting that the detection is correct.

In addition, in the example shown in FIG. 16, in the detection target image displayed on the camera image display section 172, the body of the person intersects only the second determination line (yellow determination line). status), and the original notification level is "Caution". Therefore, in this case, the user visually checks the detection target image to confirm that it is a false detection and is in the second crossing state, and then presses the button 183 to input that it is the second crossing state. Manipulate.

Further, in the example shown in FIG. 17, in the detection target image displayed on the camera image display section 172, the person's body does not intersect (contact) with either the first or second determination line (non-intersection state). Therefore, if it was originally the case, the notification would not be made. Therefore, in this case, the user visually checks the detection target image to confirm that it is a false detection and a non-intersecting state, and operates the button 184 for inputting the non-intersecting state.

On the other hand, the examples shown in FIGS. 18 to 20 are the status confirmation screen 171 when transitioning from the notification screen 161 (see FIG. 14) where the notification level is "caution". Here, the button 183 corresponding to correct detection is displayed in a grayed out state (unselectable).

In the example shown in FIG. 18, in the detection target image displayed on the camera image display section 172, the person's body intersects only the second determination line (yellow determination line) (second crossing state) The notification level is "caution". Therefore, in this case, the user confirms that the detection is correct by visually observing the detection target image, and operates the button 181 for inputting that the detection is correct.

Furthermore, in the example shown in FIG. 19, in the detection target image displayed on the camera image display section 172, the body of the person is located on the first determination line (red determination line) and the second determination line (yellow determination line). (first crossing state), and the original notification level is "dangerous". Therefore, in this case, the user visually checks the detection target image to confirm that it is a false detection and is in the first crossing state, and presses the button 182 to input that it is the first crossing state. Manipulate.

Furthermore, in the example shown in FIG. 20, in the detection target image displayed on the camera image display section 172, the person's body does not intersect (contact) either the first or second determination line (non-intersection state). Therefore, if it was originally the case, the notification would not be made. Therefore, in this case, the user visually checks the detection target image to confirm that it is a false detection and a non-intersecting state, and operates the button 184 for inputting the non-intersecting state.

Further, the status confirmation screen 171 is provided with a learning model information display section 174. The learning model information display section 174 displays identification information of the learning model currently in operation, that is, the learning model currently used for image analysis processing, specifically, the creation date and time of the learning model.

Further, the status confirmation screen 171 is provided with a button 175 for instructing rollback of the learning model. The user operates the button 175 when the user determines that the learning model currently in use has many defects, specifically, a large number of false positives, and that a rollback to the previous learning model is necessary. As a result, the rollback setting screen 211 (see FIG. 22) is displayed as a pop-up on the status confirmation screen 171.

Further, the status confirmation screen 171 is provided with a "notification" tab 176 and a "learning" tab 177. On the status confirmation screen 171, the "Notification" tab 176 is in a selected state, and when the user operates the "Learning" tab 177, the screen changes to the learning model creation screen 201 (see FIG. 21).

Next, the learning model creation screen 201 displayed on the monitoring terminal 4 will be explained. FIG. 21 is an explanatory diagram showing the learning model creation screen 201.

The learning model creation screen 201 is provided with a learning target image list display section 202. The learning target image list display section 202 displays a list of information regarding candidates for learning target images used for additional learning. Candidates for learning target images are detection target images in which abnormal events have been detected and are the targets of notification. Specifically, the learning target image list display section 202 displays, for each learning target image candidate, the shooting time, a thumbnail image, whether the abnormal event detection result is correct (correct detection or false detection), and the learning target image. The label information added to is displayed.

When the user performs a predetermined operation (right click) on the display field of any learning target image in the learning target image list display section 202, a menu 203 is displayed. In the menu 203, the user can specify whether or not to target the learning target image candidate for additional learning. Here, if an object not to be subjected to additional learning is selected, the learning target image candidate is excluded from the learning processing target.

Further, the learning model creation screen 201 is provided with a creation memo input field 204. In the creation memo input field 204, the user can input characters representing the contents of the learning model to be created this time (creation memo).

Further, the learning model creation screen 201 is provided with a "learning model creation" button 205. When the user operates the "Create Learning Model" button 205, a learning model is created using an image selected as an additional learning target from among the candidate learning target images displayed in the learning target image list display section 202. processing is performed.

Next, the rollback setting screen 211 displayed on the monitoring terminal 4 will be explained. FIG. 22 is an explanatory diagram showing the rollback setting screen 211.

When the user operates the button 175 for instructing rollback of the learning model on the status confirmation screen 171 (see FIGS. 15 to 20), a rollback setting screen 211 shown in FIG. 22 pops up and is displayed on the status confirmation screen 171. be done.

The rollback setting screen 211 is provided with a model selection section 212. The model selection section 212 displays a list of creation dates and contents for each learning model. The user can select a learning model by operating one of the learning model columns.

Further, the rollback setting screen 211 is provided with a "switch execution" button 213 and a "back" button 214. When the user operates the "switch execution" button 213, rollback processing is performed to switch the learning model used for image analysis processing to the learning model selected by the model selection unit 212. When the user operates the "back" button 214, the screen returns to the status confirmation screen 171 (see FIGS. 15 to 20).

As described above, the embodiments have been described as examples of the technology disclosed in this application. However, the technology in the present disclosure is not limited to this, and can also be applied to embodiments in which changes, replacements, additions, omissions, etc. are made. Furthermore, it is also possible to create a new embodiment by combining the components described in the above embodiments.

The learning device and the learning method according to the present invention allow the user to easily and accurately judge the occurrence status of the event of interest, without being greatly influenced by the user's subjectivity, and efficiently generate appropriately labeled learning target images. It has the effect of being able to collect data well and generate a highly accurate learning model, and based on the captured images of the monitoring area, it detects the state of an object entering a predetermined area set on the captured image as an event of interest. The present invention is useful as a learning device and a learning method for generating a learning model for learning.

1 Camera 2 Image analysis server (learning device)
3 Image management server (learning device)
4 Monitoring terminal (terminal device)
23 processor 33 processor

Claims (7)

A learning device that uses a processor to generate a learning model for detecting a state in which an object enters a predetermined area set on the photographed image as an event of interest based on a photographed image of a monitoring area, the learning device comprising:
The processor includes:
Generate a learning target image by superimposing and synthesizing a determination reference image, which is a reference for determining entry of a target object into the predetermined area, onto the photographed image;
presenting the learning target image to the user;
Based on a user's operation, information regarding the occurrence status of the noted event appearing in the learning target image is acquired as label information,
A learning device that generates the learning model by machine learning using the learning target image and the label information as learning information. The processor includes:
Generating a detection target image by superimposing and synthesizing the judgment reference image on the captured image in real time,
performing a process of detecting the event of interest from the detection target image using the learning model;
When the event of interest is detected, setting the detection target image as the learning target image in additional learning, and presenting the learning target image to the user;
Obtaining the label information based on the user's operation,
The learning device according to claim 1, wherein the learning model for updating is generated by machine learning using the learning target image and the label information as learning information for additional learning. The processor includes:
3. The learning device according to claim 2, wherein the detection target image specified by the user as a target for additional learning is set as the learning target image in additional learning based on a user's operation. The processor includes:
2. The learning device according to claim 1, wherein a learning model is generated for detecting, as the event of interest, a state in which an object has entered the predetermined area corresponding to a dangerous area in which the object moves. The processor includes:
2. The learning device according to claim 1, wherein an image representing a linear determination line is superimposed and synthesized on the photographed image as the determination reference image. The processor includes:
2. The learning device according to claim 1, wherein an image representing a polygonal determination area is superimposed and synthesized on the photographed image as the determination reference image. A learning method that uses a processor to generate a learning model for detecting a state in which an object has entered a predetermined area set on the photographed image as an event of interest based on a photographed image of a monitoring area, the method comprising:
generating a learning target image by superimposing and synthesizing a determination reference image, which is a reference for determining entry of an object into the predetermined area, onto the photographed image;
presenting the learning target image to the user;
Based on a user's operation, information regarding the occurrence status of the noted event appearing in the learning target image is acquired as label information,
A learning method characterized in that the learning model is generated by machine learning using the learning target image and the label information as learning information.

Images