WO2023079593A1

WO2023079593A1 - Stain determination device, stain determination method, and stain determination program

Info

Publication number: WO2023079593A1
Application number: PCT/JP2021/040401
Authority: WO
Inventors: 健太久瀬; 恭平濱田; 拓弥橋口; 猛大佐賀; 淳志堀
Original assignee: 三菱電機ビルソリューションズ株式会社; 三菱電機株式会社
Priority date: 2021-11-02
Filing date: 2021-11-02
Publication date: 2023-05-11
Also published as: JPWO2023079593A1; JP7455284B2

Abstract

An acquisition unit (131) regularly acquires captured-image data (122) captured by a monitoring camera (400). An extraction unit (132) extracts feature amounts of the acquired captured-image data (122). An estimation unit (141) inputs information based on the extracted feature amounts to a trained model (135) and outputs the degree of stain on a lens of the monitoring camera (400). A determination unit (143) determines a stained state where the lens is stained to a given extent, in the case where the degree of stain has reached or exceeded a threshold relating to the degree of stain, and the degree of stain does not fall below the threshold after lapse of a prescribed period since the degree of stain reached or exceeded the threshold.

Description

Dirt Judgment Device, Dirt Judgment Method, and Dirt Judgment Program

The present disclosure relates to a dirt determination device, a dirt determination method, and a dirt determination program.

Japanese Patent No. 6616906 (Patent Literature 1) discloses a detection device that uses an identification model to determine whether photographed data photographed by a surveillance camera is flawed photographed data containing defects. there is This identification model is a model for extracting feature amounts from photographed data and detecting defective photographed data based on the extracted features. Defective photography data is photography data that looks like it was taken when the lens of the camera was dirty.

Japanese Patent No. 6616906

When making judgments using the above identification model, there is a possibility that changes in images that are not defects will be recognized as defect data. For example, when shooting the inside of an elevator car, even though you want to detect the dirt on the lens of the surveillance camera, you may not be able to detect the dirt, such as people or objects being reflected, or things that change regularly, such as opening and closing the door. Other elements may be detected as dirt.

In this case, it is possible to increase the accuracy of estimation by increasing the amount of learning data taken with elements other than dirt removed. However, when the probability of occurrence of such a state is low, it is difficult to collect learning data, making it difficult to improve estimation accuracy.

The present disclosure has been made to solve such a problem, and the object thereof is to be able to accurately determine whether or not the lens of a surveillance camera is in a state with a certain amount of dirt. An object of the present invention is to provide a contamination determination device, a contamination determination method, and a contamination determination program.

A contamination determination device according to the present disclosure includes an acquisition unit, an extraction unit, an estimation unit, and a determination unit. The acquisition unit periodically acquires image data captured by the surveillance camera. The extraction unit extracts the feature amount of the acquired photographed data. The estimation unit inputs information based on the extracted feature amount to the learned model and outputs the degree of contamination of the lens of the surveillance camera. The determining unit determines that the lens has a certain amount of dirt when the degree of dirt is equal to or greater than a threshold for the degree of dirt and the degree of dirt does not fall below the threshold even after a predetermined time has passed since the degree of dirt became equal to or greater than the threshold. It is judged to be in a dirty state.

A dirt determination method according to the present disclosure includes a step of periodically acquiring photographed data photographed by a surveillance camera, a step of extracting a feature amount of the acquired photographed data, and a step of learning information based on the extracted feature amount. a step of inputting data into the model and outputting the degree of contamination of the lens of the surveillance camera; and determining that the lens is in a dirty state with a certain degree of dirt if the degree does not fall below the threshold.

A dirt determination program according to the present disclosure provides a computer with a step of periodically acquiring photographed data photographed by a surveillance camera, a step of extracting a feature amount of the acquired photographed data, and a step based on the extracted feature amount. a step of inputting information into a learned model and outputting the degree of contamination of a lens of a surveillance camera; and a step of determining that the lens is in a dirty state with a certain degree of dirt if the degree of dirt does not fall below the threshold value.

According to the present disclosure, it is possible to accurately determine whether or not the lens of the surveillance camera is in a dirty state with a certain degree of dirt.

It is a figure which shows an example of the hardware constitutions of a dirt determination system. It is a figure which shows an example of the functional block diagram of a contamination determination system. 4 is a flowchart of first processing executed by the contamination determination device; 4 is a flowchart of determination processing executed by the contamination determination device; FIG. 4 is a diagram for explaining various learned models; FIG. It is a figure for demonstrating the time change of the dirt degree of a lens. FIG. 5 is a diagram for explaining the relationship between an image of the interior of the car and the degree of contamination. FIG. 5 is a diagram for explaining the relationship between an image of the interior of the car and the degree of contamination. FIG. 5 is a diagram for explaining the relationship between an image of the interior of the car and the degree of contamination. FIG. 5 is a diagram for explaining the relationship between an image of the interior of the car and the degree of contamination. FIG. 5 is a diagram for explaining the relationship between an image of the interior of the car and the degree of contamination. FIG. 10 is a diagram for explaining the relationship between an image of the inside of the car captured in the first process and the degree of contamination; FIG. 10 is a diagram for explaining the relationship between an image of the interior of the car captured in the second process and the degree of contamination; It is a flow chart of the 2nd processing which a dirt judging device performs. FIG. 5 is a diagram for explaining the relationship between an image of the interior of the car and the degree of contamination.

Embodiments will be described below with reference to the drawings. In the following description, the same parts are given the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated.

[Dirt determination system 1]
First, the contamination determination system 1 including the contamination determination device 200 according to the present embodiment will be described. FIG. 1 is a diagram showing an example of the hardware configuration of the contamination determination system 1. As shown in FIG.

The dirt determination system 1 includes a video recording device 100, an elevator car 401, a monitoring camera 400 installed in the car 401, a dirt determination device 200, and a terminal 300. Video recording device 100 can communicate with surveillance camera 400 and contamination determination device 200 . Terminal 300 can communicate with contamination determination device 200 .

A monitoring camera 400 is a camera for monitoring the inside of an elevator car 401 installed in a building. Surveillance camera 400 generates, for example, 30 frame images per second. The continuously generated frame images are reproduced as a moving image by being reproduced continuously.

The photographed data in this embodiment corresponds to a frame image. The video recording device 100 is a device that records photographed data (frame images) photographed by the monitoring camera 400 and determines the state of dirt on the lens of the monitoring camera 400 .

It should be noted that multiple baskets may be installed in the building. In this case, a monitor camera 400 is installed for each car. The video recording device 100 determines the dirt condition of the lens of the monitoring camera 400 for each car. Moreover, the monitoring camera 400 is not limited to being installed in the elevator car 401, and may be installed to monitor elevator halls, escalators, and other building equipment.

The video recording device 100 includes a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, a RAM (Random Access Memory) 113, a storage section 114, a communication interface 115, and an I/O interface 116. have. These are communicably connected to each other via a bus.

The CPU 111 comprehensively controls the video recording apparatus 100 as a whole. The CPU 111 develops a program stored in the ROM 112 in the RAM 113 and executes it. The ROM 112 stores a program in which processing procedures for processing performed by the video recording apparatus 100 are described.

The RAM 113 serves as a work area when the CPU 111 executes programs, and temporarily stores programs and data used when executing the programs. Also, the storage unit 114 is a non-volatile storage device, such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

The video recording device 100 can communicate with the contamination determination device 200 via the communication interface 115 . I/O interface 116 is an interface for CPU 111 to connect with surveillance camera 400 .

The contamination determination device 200 has a CPU 211 , a ROM 212 , a RAM 213 , a storage section 214 , a communication interface 215 and an I/O interface 216 . These are communicably connected to each other via a bus.

The CPU 211 comprehensively controls the contamination determination device 200 as a whole. The CPU 211 develops a program stored in the ROM 212 in the RAM 213 and executes it. The ROM 212 stores a program in which processing procedures for processing performed by the contamination determination device 200 are described.

The RAM 213 serves as a work area when the CPU 211 executes the program, and temporarily stores the program and data used when executing the program. Also, the storage unit 214 is a non-volatile storage device such as an HDD or an SSD.

The contamination determination device 200 can communicate with the video recording device 100 and the terminal 300 via the communication interface 215 . The I/O interface 216 is an interface for the CPU 211 to connect with a display device or an input device.

The display device is, for example, a display. On the display device, it is possible to confirm photographed data and the state of dirt on the lens. Input devices are, for example, a keyboard and a mouse. For example, an instruction can be given to the contamination determination device 200 by operating the input device.

Although not shown, the terminal 300 also has a CPU, a ROM, a RAM, a storage unit, a communication interface, and an I/O interface, similar to the contamination determination device 200 . The terminal 300 may be a personal computer, a tablet terminal, a smart phone, or the like. In the terminal 300 as well, it is possible to check the imaging data recorded in the monitoring camera 400, the dirtiness of the lens, and the like.

FIG. 2 is a diagram showing an example of a functional block diagram of the contamination determination system 1. As shown in FIG. The video recording device 100 has an acquisition unit 121 . Acquisition unit 121 acquires image data captured by surveillance camera 400 and stores it as image data 122 in storage unit 114 .

The contamination determination device 200 includes an acquisition unit 131, an extraction unit 132, a normalization unit 133, an estimation unit 141, a model generation unit 134, a determination unit 143, a prediction unit 145, and a notification unit 144.

Through this series of processes, the dirt determination device 200 uses the captured data 122 captured by the monitoring camera 400 to generate the learned model 135 . In addition, it acquires the photographed data 122 photographed by the monitoring camera 400, and finally notifies the terminal 300 of the dirt determination result.

At the terminal 300, it is possible to confirm the determination result notified by the contamination determination device 200. The storage unit 214 stores the learned model 135 generated by the model generation unit 134 and the estimation result 142 generated by the estimation unit 141 . The flow of processing will be described below using flowcharts and the like.

[Flowchart of first process]
The contamination determination device 200 can set execution modes including a first mode and a second mode. When the first mode is set, the determination unit 143 makes a determination every time T1. When the second mode is set, the determination unit 143 makes a determination every time T2, which is longer than the time T1. In this embodiment, time T1=10 minutes and time T2=1 day.

Specifically, the contamination determination device 200 activates the first process when the first mode is set, and activates the second process when the second mode is set. In this embodiment, it is assumed that both the first mode and the second mode are set.

FIG. 3 is a flowchart of the first process executed by the contamination determination device 200. FIG. The first process is a process of judging contamination with a period of time T1. Hereinafter, "step" is also simply referred to as "S".

As shown in FIG. 3, when the first process starts, in S11, the contamination determination device 200 determines whether or not time T1 has elapsed. When the contamination determination device 200 determines that the time T1 has elapsed (YES in S11), the process proceeds to S12. If the contamination determination device 200 does not determine that the time T1 has elapsed (NO in S11), the process returns to S11.

In S12, the contamination determination device 200 performs determination processing (see FIG. 4). The determination process is a process of determining the dirty state of the lens of the monitoring camera 400 based on the image data 122 captured by the monitoring camera 400 .

In S13, the notification unit 144 notifies the terminal 300 of the determination result of the determination unit 143, and returns the process to S11. In this way, the first process performs the determination process each time the time T1 elapses.

FIG. 4 is a flowchart of determination processing executed by the contamination determination device 200. FIG. As shown in FIG. 4, when the determination process starts, in S21, the acquisition unit 131 acquires the photographed data 122 photographed by the monitoring camera 400, and the process proceeds to S22. The photographed data 122 is data obtained by photographing the inside of the elevator car 401 .

In S22, the extraction unit 132 extracts the feature amount of the acquired photographing data 122, and advances the process to S23. In S23, the normalization unit 133 normalizes the extracted feature amount, and the process proceeds to S24. The processing of the extraction unit 132 and the normalization unit 133 is the same as the processing during learning, which will be described later with reference to FIG.

The estimation unit 141 inputs information based on the extracted feature amount to the learned model 135 and outputs the degree of contamination of the lens of the surveillance camera 400 . The degree of dirt indicates how dirty the lens of the monitoring camera 400 is by using a numerical value from 0 to 1. FIG.

Specifically, in S24, the estimating unit 141 stores the normalized data as information based on the extracted feature amount for all the learned models 135 (the learned model shown in FIG. 5 to be described later) stored in the storage unit 214. models A to E, etc.), output the degree of contamination, and advance the process to S25. In S25, the estimating unit 141 selects the lowest contamination degree from among all the output contamination degrees, and advances the process to S26.

In this way, the estimating unit 141 selects one of all the learned models 135 (learned models A to E, etc.) based on the contamination degree output results. Specifically, the estimating unit 141 selects the model with the smallest output result from among the plurality of models. The details will be described later with reference to FIG. 5. The reason is that the model with the smallest output result (dirt degree) is considered to be the optimum model.

Determination unit 143 determines that there is contamination when the degree of contamination becomes equal to or greater than the threshold and the degree of contamination becomes less than the threshold within a predetermined period of time (e.g., 8 minutes) after the degree of contamination becomes equal to or greater than the threshold. state (S26 to S28 below).

Details will be described later using FIGS. 6 to 10, including specific examples, but in the present embodiment, an element unrelated to dirt may be erroneously detected as "dirt". Since it is assumed that these elements will not be detected with the passage of time, the transition of the degree of contamination is confirmed until a predetermined period of time has elapsed.

Here, the "threshold" is a threshold set in advance regarding the degree of contamination, and is set to "0.3" in the present embodiment. Further, the “dirt state” is a state in which the lens of the monitoring camera 400 is soiled to a certain degree, and specifically refers to a state in which the degree of dirt is equal to or greater than a threshold.

In S26, the estimating unit 141 determines whether or not the degree of contamination becomes equal to or greater than the threshold and the degree of contamination does not fall below the threshold even after a predetermined period of time has elapsed since the degree of contamination became equal to or greater than the threshold.

Specifically, when the degree of contamination is equal to or greater than the threshold, steps S21 to S25 are repeated every specific period of time (eg, 30 seconds) until a predetermined period of time (8 minutes) elapses. Then, among the contamination degrees obtained until the predetermined period (8 minutes) has passed, the smallest contamination degree is selected as the correct contamination degree. In S26, it is determined whether or not the value selected as the correct degree of contamination is less than the threshold value.

If the estimating unit 141 determines that the contamination degree is equal to or greater than the threshold value and that the contamination degree does not become less than the threshold value even after a predetermined period of time has elapsed since the contamination degree became equal to or greater than the threshold value (YES in S26), the process to S27. In S<b>27 , the determination unit 143 determines that the dirt exists, and terminates the determination process.

The estimating unit 141 did not determine that the degree of contamination became equal to or greater than the threshold and that the degree of contamination did not become less than the threshold even after a predetermined period of time had passed since the degree of contamination became equal to or greater than the threshold (that is, the predetermined time had elapsed). (NO in S26), the process proceeds to S28. In S28, the determination unit 143 determines that there is no contamination, and terminates the determination process.

After confirming that the terminal 300 is dirty, the elevator maintenance staff or building manager cleans the lens of the monitoring camera 400 to remove the dirt.

[Trained model]
FIG. 5 is a diagram for explaining various learned models. The learned model 135 is a model that has been subjected to learning processing in order to output the degree of contamination when information based on the feature amount is input.

As shown in FIG. 5, the trained model 135 includes trained models A to E and the like. Each of the trained models A to E, etc. is a model that has undergone learning processing using a group of photographed data 122 photographed in one of a plurality of photographing environments.

The learned model A is a model that has undergone learning processing using 122 groups of shooting data taken in the morning as the shooting environment. The learned model B is a model that has undergone learning processing using the 122 group of shooting data taken in the daytime as the shooting environment. The learned model C is a model for which learning processing has been performed using a group of 122 photographed data taken at night as the photographing environment. The brightness in the car 401 differs in the morning, daytime, and night depending on how light enters.

The learned model D is a model in which learning processing is performed using a group of 122 photographed data photographed with the door of the car 401 open (door open state) as the photographing environment. The trained model E is a model in which learning processing is performed using a group of photographed data 122 photographed in a state in which the door of the car 401 is closed (closed door state) as the photographing environment.

In the present embodiment, learning processing is performed using a set of photographed data (a group of photographed data) whose degree of contamination has been determined in advance. Acquisition unit 131 acquires photographed data 122 for which the degree of contamination has been determined in advance as learning data.

At that time, for example, when the learned model D (door open state) is generated, the degree of contamination has been determined in advance, and photographed data 122 photographed with the door open is acquired as learning data. The imaging data 122 for the learning process may be stored in the storage unit 114 as teaching data after linking the degree of dirt with the imaging environment in advance.

Next, the extraction unit 132 extracts the feature amount of the captured image data 122 obtained. For example, the feature amount may be extracted based on the color characteristics of the photographed data 122 . When the photographing data 122 is a color image, each pixel has color characteristic data. Specifically, there are saturation, lightness, hue, and the like. In addition, the photographed data as a whole includes coloration, variation, structure, and the like. By combining these color characteristics, a feature amount that shows a difference between the low degree of contamination and the high degree of contamination is extracted. When the degree of dirt is high, it is presumed that the image of the dirt adhering to the lens (the imaged area of the dirt portion) is black, that is, the brightness is lowered. Therefore, it is conceivable that the feature amount will be greatly different from that in the case where dirt is not attached. In this way, it is sufficient to extract feature amounts so that a difference appears when the degree of contamination is high.

Next, the normalization unit 133 normalizes the feature quantity extracted by the extraction unit 132 . The model generating unit 134 performs learning processing based on the normalized photographing data 122 and the associated degree of contamination to generate a learned model.

When the shooting environment of the shooting data to be determined is close to the shooting environment of the learned model, it can be expected that the determination accuracy of the determination unit 143 will be high. For example, as shown in FIG. 7, if the acquired photographed data is photographed with the door open, then the learned model D undergoes learning processing using the photographed data 122 group photographed with the door open. can be expected to improve the accuracy of determination.

Then, it can be expected that the model with the lowest calculated degree of contamination is the model with the highest determination accuracy of the determination unit 143 . For example, the state of the door is different between the photographed data photographed with the door open and the learned model E using the photographed data with the door closed, so there is a possibility that the degree of contamination will be erroneously detected. is high. On the other hand, since the state of the door is the same between the photographed data photographed with the door open and the learned model D using photographed data similarly with the door open, the degree of contamination in the region of the door is Less chance of false positives.

Therefore, as described above, the estimating unit 141 uses all the learned models 135 (learned models A to E, etc.) to output the respective contamination degrees. Then, the model with the smallest degree of contamination is selected as the optimum model (the smallest degree of contamination is determined to be the correct degree of contamination).

In addition, the learned model is not limited to the above-mentioned learned models A to E, but also a model that has undergone learning processing while passengers are on board, or a model that has undergone learning processing in an environment in which multiple shooting environments are combined. may include Alternatively, the contamination level may be output by the estimating unit 141 using only one trained model. In this case, the learning process becomes simple.

It should be noted that similar regions in the image may be divided as a group. In this case, the estimation unit 141 outputs the contamination level for each divided area. The estimating unit 141 calculates “degree of contamination”×“percentage of the divided region in the entire image” for each region, and adds them up to calculate the “degree of contamination”.

[Time change of degree of contamination]
FIG. 6 is a diagram for explaining the temporal change in the degree of contamination of the lens. In FIG. 6, the vertical axis indicates the degree of contamination. The horizontal axis indicates time.

As shown in FIG. 6, when the lens is not dirty at all (also referred to as "normal"), the degree of contamination is 0. The threshold is 0.3.

At time t1 (scene 1), the contamination level is 0. After that, scene 2 is time t2 when the degree of contamination rises sharply to 0.9. At this time, the degree of contamination is equal to or greater than the threshold value (0.3). After that, scene 3 is time t3 when the degree of contamination drops sharply and becomes 0.1. At this time, the degree of contamination (0.1) is below the threshold (0.3). In such a case, it is determined that there is no contamination.

Next, scene 4 is time t4 when the degree of contamination rises sharply to 0.8. At this time, the degree of contamination (0.8) is equal to or greater than the threshold (0.3). After that, scene 5 is time t5 when the degree of contamination drops sharply to 0.4. At this time, the contamination level (0.4) remains equal to or greater than the threshold value (0.3). In such a case, it is judged to be in a dirty state.

The situation in Scenes 1 to 5 will be explained below using images of the interior of the car 401. 7 to 11 are diagrams for explaining the relationship between the image of the inside of the car 401 and the degree of dirtiness.

In scene 1, as shown in FIG. 7, in an image 80a (photographed data) of the inside of the car 401, the door of the car 401 is open (door open state). There are no passengers in car 401 . Also, the lens of the monitoring camera 400 is not soiled, and the degree of soiling=0 is output.

In scene 2, as shown in FIG. 8, in an image 80b of the interior of the car 401, four passengers 53 are in the car 401, and the door of the car 401 is closed (door closed state) at the entrance/exit 51. ing. In addition, dirt adheres to the lens. At this time, the contamination level is increased to 0.9.

In scene 3, as shown in FIG. 9, in an image 80c of the inside of the car 401, the door of the entrance 51 is open (door open state) and all the passengers inside the car 401 have gotten off. At this time, the dirt on the lens is attached to the same extent as scene 2. At this time, the contamination degree is lowered to 0.1.

In both scenes 1 and 3, the doors are open and there are 0 passengers. Since the only difference in the images between scene 1 and scene 3 is dirt on the lens, the degree of dirt of 0.1 can be considered purely due to dirt on the lens.

On the other hand, scene 2 and scene 3 are the same in that the lenses are dirty, but differ in that scene 2 has the door closed and there are four passengers. Therefore, the contamination degree of 0.9 in Scene 2 is considered to include erroneous detection by the open/closed state of the door of car 401 and passengers inside car 401 in addition to dirt on the lens.

As described above, in the present embodiment, when the door of the car 401 is open, the number of passengers is 0, and the lenses are not dirty (Scene 1), the degree of contamination is calculated as 0. After that, the dirt degree of 0.1 adheres to the lens, but when four passengers board and the door is closed, the dirt degree increases to 0.9.

In this case, although the degree of contamination is equal to or greater than the threshold value (0.3), the degree of contamination is not actually contamination, but simply increases due to an image change due to the door being closed and passengers boarding the vehicle. be. Therefore, in scene 3 where the door is open and the number of passengers is 0, only pure dirt on the lens (degree of dirt = 0.1) is detected.

In this way, the degree of contamination changes according to at least one of the contamination of the lenses, the amount of load in the car 401 (for example, the number of passengers), and the open/closed state of the door of the car 401. Therefore, in the present embodiment, the estimating unit 141 determines that the degree of contamination has become equal to or greater than the threshold and that the degree of contamination has become less than the threshold within a predetermined period of time after the degree of contamination becomes equal to or greater than the threshold. If so, it is determined that there is no contamination (S26, S28). Various factors that change the degree of contamination will be described in more detail with reference to FIG. 15 .

Next, in scene 4, as shown in FIG. 10, in an image 80d of the interior of the car 401, two passengers 53 are in the car 401 and the door is closed. Furthermore, the lower left corner of the lens of the surveillance camera 400 is soiled due to mischief by the passenger 53 . At this time, the degree of contamination has increased to 0.8.

In scene 5, as shown in FIG. 11, in an image 80e of the interior of the car 401, the door is open and all the passengers in the car 401 have gotten off. At this time, the contamination degree is lowered to 0.4.

In both scenes 3 and 5, the doors are open and there are 0 passengers. The only difference between the images of scene 3 and scene 5 is the dirt on the lower left corner of the lens that the passenger 53 put on for mischief. As a result, the contamination level increases from 0.1 to 0.4, exceeding the threshold value (0.3).

On the other hand, scene 4 and scene 5 are the same in that the lens is dirty, but differ in that scene 4 has the door closed and there are two passengers. Therefore, the contamination level of 0.8 in scene 4 is considered to include erroneous detection by the open/closed state of the door of car 401 and the passengers inside car 401 in addition to the dirt on the lens.

In this way, when the door is open, passengers = 0, and the lens is dirty (Scene 3), the degree of contamination is calculated as 0.1. After that, two passengers board the vehicle, the door is closed, and the fouling degree increases to 0.8 due to fouling of the lens caused by mischief.

In this case, the degree of contamination exceeds the threshold (0.3), but an increase in the degree of contamination due to image changes due to the passenger getting on the vehicle with the door closed is also included. Therefore, in scene 5 where the door is open and the number of passengers is 0, only pure dirt on the lens (degree of dirt = 0.4) is detected. Also in this case, since the degree of contamination (0.4) does not fall below the threshold value (0.3), it is determined that there is contamination.

In this way, the determining unit 143 determines that the soiled state is present when the degree of soiling is equal to or greater than the threshold and the degree of soiling does not fall below the threshold even after a predetermined time has elapsed since the degree of soiling became equal to or greater than the threshold. (S26, S27).

Here, when trying to estimate the degree of contamination using a trained model, there is a possibility that false detections (increased values) may occur due to factors other than contamination (e.g. door open/close status and passengers). . In this case, it is conceivable to improve the accuracy of estimating the degree of contamination by performing more accurate learning processing in order to distinguish between contamination and elements other than contamination. For that purpose, it is conceivable to increase the learning data in a state where elements other than dirt are excluded to improve the accuracy of learning. However, when the probability of occurrence of such a state is low, it is difficult to collect learning data, making it difficult to improve estimation accuracy.

For this reason, in the present embodiment, methods other than estimation using a learned model are also used to exclude elements other than dirt, thereby increasing the accuracy of estimating the degree of dirt. As described above, the degree of contamination caused by "dirt" does not decrease over time, and the degree of contamination caused by "factors other than contamination" decreases over time. For example, the "door open/closed state", which is an element other than dirt, returns to the door open state with the lapse of time, and the number of passengers returns to 0 with the lapse of time. In this way, there are cases where the threshold is exceeded due to factors other than contamination. Therefore, by waiting for a predetermined period of time, erroneous detection due to factors other than contamination can be reduced, and the degree of contamination due only to contamination can be detected as much as possible. By doing so, it is possible to accurately determine whether or not there is dirt without adding learning data.

Next, the relationship between the image taken inside the car 401 and the degree of contamination in the first and second processes will be described. FIG. 12 is a diagram for explaining the relationship between the image of the inside of the car 401 photographed in the first process and the degree of contamination.

In the first process, as described using the flowcharts of FIGS. 3 and 4, the determination process is performed every T1 (10 minutes). On the other hand, in the second process, the determination process is performed every T2 (one day) as will be described later with reference to the flowchart of FIG.

First, an example in which determination processing is performed every 10 minutes in the first processing will be described. As shown in FIG. 12, in the image 80j, the doorway 51 is closed and there is no passenger 53 in the car 401. As shown in FIG. Also, the lens of the surveillance camera 400 is not dirty, and the degree of dirt is zero.

After this state, judgment processing will be performed every 10 minutes. At that time, if the degree of contamination is determined to be equal to or greater than the threshold, it is determined whether or not the degree of contamination remains equal to or greater than the threshold even after a predetermined period of time has elapsed. As described with reference to FIGS. 6 to 11, even when it is determined that the degree of contamination is equal to or greater than the threshold as in scene 2 (t2), scene 3 (t3) until a predetermined time elapses. , if the degree of contamination is less than the threshold value, it is determined that there is no contamination.

On the other hand, when it is determined that the degree of contamination is greater than or equal to the threshold as in scene 4 (t4), the degree of contamination remains greater than or equal to the threshold in scene 3 (t5) even after the predetermined time has passed. , is determined to be in a dirty state.

In the state of image 80j, for example, let us say that the lens of surveillance camera 400 has been soiled by a passenger's mischief. An image 80k is obtained after 10 minutes have elapsed from the state of the image 80j. In the image 80k, the entrance/exit 51 is closed and no passenger 53 is inside the car 401 . However, dirt adheres to the right side of the lens of the monitoring camera 400, and the degree of dirt is 0.4. In this case, since the degree of contamination is equal to or greater than the threshold value (0.3), it is determined that there is contamination.

Thus, in the first process, since the determination process is performed every 10 minutes, it is possible to detect sudden stains such as when the lens is soiled by mischief.

Next, an example in which determination processing is performed for each day in the second processing will be described. FIG. 13 is a diagram for explaining the relationship between the image of the inside of the car 401 photographed in the second process and the degree of contamination. As shown in FIG. 13, in the image 80f, the doorway 51 is in the open state and there is no passenger 53 in the car 401. As shown in FIG. Also, the lens of the surveillance camera 400 is not dirty, and the degree of dirt is zero.

Determination processing is performed every day, and image 80g is the state after 30 days have passed since the state of image 80f (determination processing has been performed 30 times). In the image 80 g as well, the doorway 51 is in the open state and no passenger 53 is inside the car 401 . However, the lens of the monitoring camera 400 is dirty, and the degree of dirt is 0.1.

It should be noted that when the judgment process is executed, there are passengers, and if the result is equal to or higher than the threshold, it is judged whether the degree of contamination will be less than the threshold within a predetermined period. In this example, the degree of contamination drops to 0.1 at the timing when the passenger gets off.

An image 80h is a state in which 30 days have passed since the state of the image 80g (30 determination processes have been performed). Counting from the state of the image 80f, 60 days have passed. In the image 80h, the entrance/exit 51 is still in the open state, and there is no passenger 53 inside the car 401 . However, the lens of the surveillance camera 400 is further contaminated, and the degree of dirt is 0.2.

An image 80i is a state in which 30 days have passed since the state of the image 80h (30 determination processes have been performed). Counting from the state of the image 80f, 90 days have passed. In the image 80 g as well, the doorway 51 is in the open state and no passenger 53 is inside the car 401 . However, the lens of the surveillance camera 400 is further contaminated, and the degree of dirt is 0.3. In this case, since the degree of contamination has reached the threshold value, it is determined to be in the contamination state.

According to the images 80f to 80i, the degree of contamination increases by 0.1 every 30 days. For this reason, for example, it is possible to predict that the degree of contamination will reach the threshold value 90 days after 30 days from the time-dependent change in the degree of contamination from 0 days to 60 days. This allows maintenance personnel to know when to clean the lens. The second process will be described below using a flowchart.

[Flowchart of second processing]
As described above, the first process is the process of judging contamination with a cycle of time T1 (10 minutes). On the other hand, in the second process, dirt is determined in a period of time T2 (one day). In addition, the second process estimates future lens contamination.

The second process will be described below. FIG. 14 is a flowchart of the second process executed by the contamination determination device 200. FIG.

As shown in FIG. 14, when the second process starts, in S31, the contamination determination device 200 determines whether or not time T2 has elapsed. When the contamination determination device 200 determines that the time T2 has elapsed (YES in S31), the process proceeds to S32. If the contamination determination device 200 does not determine that the time T2 has elapsed (NO in S31), the process returns to S31.

In S32, the contamination determination device 200 performs determination processing, and advances the processing to S33. That is, the contamination determination device 200 performs determination processing and the like each time the time T2 elapses.

In S33, the prediction unit 145 of the contamination determination device 200 stores the degree of contamination in the storage unit 214, and advances the process to S33.

In S34, the prediction unit 145 estimates the time when it is determined that there is dirt based on the time-series data of the degree of dirt stored in the storage unit 214, and advances the process to S35.

In the example shown using FIG. 13, the contamination level is 0 on the 0th day. On the 30th day, the fouling degree is 0.1. On the 60th day, the fouling degree=0.2. Assume that it is now the 60th day. The storage unit 214 stores dirt degree data from the 0th day to the 60th day.

According to these data, the degree of dirt increases by 0.1 every 30 days, so the prediction unit 145 predicts that the degree of dirt will reach the threshold of 0.3 on the 90th day (after 30 days). do. That is, it is predicted that the time when it is determined to be in the dirty state is 30 days later.

For example, the prediction unit 145 may generate a prediction model using the least squares method or the like using time-series data of the degree of contamination. The prediction unit 145 predicts the time when it is determined that the soiled state is present using the prediction model.

In S35, the notification unit 144 transmits the determination result of the determination unit 143 and the estimation result of the prediction unit 145 to the terminal 300, and returns the process to S31.

[Various factors that change the degree of contamination]
As described with reference to FIGS. 6 to 11, etc., the degree of contamination changes according to at least one of "dirt on lenses", "loading amount in car 401", and "open/closed state of door of car 401". do. Here, various factors that change the degree of contamination will be described in more detail with reference to FIG. 15 .

FIG. 15 is a diagram for explaining the relationship between the image of the interior of the car 401 and the degree of contamination. As shown in FIG. 15, in an image 80j of the inside of the car 401, the doorway 51 is open and there are no passengers inside the car 401. As shown in FIG. Also, the lens of the monitoring camera 400 is not soiled, and the degree of soiling=0 is output.

In the examples of FIGS. 6 to 11, an example in which a passenger rides in the car 401 has been described. There is also The "loading amount in the car 401" includes not only the number of passengers but also the amount of luggage. Note that the "loading amount in the car 401" may indicate the total area of passengers and luggage estimated from the image 80k.

In the example of image 80k, the degree of contamination has increased to 0.2 due to the luggage being loaded in the car 401 together with the passengers. In this case, the degree of contamination varies according to the amount of load in the car 401, but the degree of contamination returns to 0 again when the passenger and the cargo leave the car 401 upon arrival at the destination floor. Therefore, it is not determined to be "dirty" based on the passenger and luggage.

Also, the above-mentioned "door open/closed state of car 401" includes "state of landing". In an image 80l of the inside of the car 401, the door is open like the image 80j, but the state of the hall has changed. Specifically, in the image 80j, the platform is bright due to sunlight during the daytime, whereas in the image 80l, the platform is dark at night, and the degree of contamination increases to 0.15. As described above, the degree of contamination differs between the door open state and the door closed state, but in the door open state the degree of contamination also differs depending on the time zone.

In the example of FIG. 5, as the shooting environment, different learned models are used for the morning, noon, and night (learned models A to C), and different learned models are used for the door open state and the door closed state (learned model By constructing the models D and E) as described above, the accuracy of determination of the degree of contamination is improved. In this example, for example, by changing the learning model according to the time zone in the door open state, it is possible to further improve the accuracy of determining the degree of contamination.

In addition, the "state of the boarding area" includes the congestion status of the boarding area. For example, the degree of contamination differs between peak hours such as when arriving and leaving work (a plurality of waiting passengers are reflected when the door is open) and off-peak hours (when there are no people at the boarding point). In this case as well, for example, by using different learning models for the shooting environment during peak hours and during other hours, it is possible to further improve the accuracy of determining the degree of contamination.

In addition to the above, the degree of contamination also changes depending on "the state of the monitoring camera 400 installed in the car 401". For example, in an image 80m of the inside of the car 401, the installation direction of the surveillance camera 400 installed in the car 401 is shifted due to mischief by a passenger or the like. ing. As a result, the degree of contamination is changed to 0.4 due to the difference between the image 80j and the image 80m.

In this case, since the degree of contamination does not decrease even after a predetermined period of time has passed, it is erroneously detected as a "dirty state." In this case, the maintenance staff of the elevator or the manager of the building checks the installation state of the monitoring camera 400 and restores the monitoring camera 400 to the normal installation state. As a result, the degree of contamination returns to 0, and the degree of contamination can be normally detected. The "state of the monitoring camera 400 installed in the car 401" may be the installation direction or installation position of the monitoring camera 400 as described above, or a state in which the monitoring camera 400 is out of focus. can be assumed.

[Main configuration and effects]
Main configurations and effects of the above-described embodiment will be described below.

(1) The contamination determination device 200 includes an acquisition unit 131 , an extraction unit 132 , an estimation unit 141 and a determination unit 143 . Acquisition unit 131 periodically acquires photographed data 122 photographed by surveillance camera 400 . The extraction unit 132 extracts the feature amount of the acquired photographed data 122 . The estimation unit 141 inputs information based on the extracted feature amount to the learned model 135 and outputs the degree of contamination of the lens of the surveillance camera 400 . The determining unit 143 determines that when the degree of contamination is equal to or greater than the threshold for the degree of contamination, and the degree of contamination does not become less than the threshold even after a predetermined time has passed since the degree of contamination became equal to or greater than the threshold, the lens has a certain degree of contamination. It is determined that there is some dirt. By waiting for a predetermined period of time, erroneous detection due to factors other than dirt can be reduced, and it is possible to accurately determine whether or not the lens of the surveillance camera is in a dirty state with a certain degree of dirt.

(2) The learned model 135 includes a plurality of models (learned models A to E, etc.). Each of the plurality of models is subjected to learning processing to output the degree of contamination when information based on the feature amount is input using a group of photographed data 122 photographed in one of a plurality of photographing environments. model. The estimating unit 141 selects one model from among a plurality of models, and outputs the degree of contamination using the selected model. By using an appropriate model according to the imaging environment, it is possible to reduce erroneous detection due to changes in the surrounding environment and determine whether or not there is dirt.

(3) The estimation unit 141 selects the model with the smallest output result from among the multiple models. By using an appropriate model according to the imaging environment, it is possible to reduce erroneous detection due to changes in the surrounding environment and determine whether or not there is dirt.

(4) The contamination determination device 200 can set execution modes including a first mode and a second mode. When the first mode is set, the determination unit 143 performs determination for each first cycle. When the second mode is set, the determination unit 143 performs determination for each second cycle longer than the first cycle. As a result, dirt that suddenly occurs can be detected in the first mode, and dirt that accumulates over time can be detected in the second mode. By detecting dirt that accumulates over time, it is possible to predict when the lens will become dirty and need cleaning.

(5) The photographed data 122 is data in which the inside of the elevator car 401 is photographed. The degree of contamination changes according to at least one of the contamination of the lenses, the amount of load (the number of passengers and the amount of luggage) in the car 401, and the door opening/closing state of the car 401. FIG. As a result, it is possible to discriminate dirt on the lens that does not reduce the degree of dirt even after the passage of a predetermined time from the amount of load and door open/closed state that may reduce the degree of dirt before the passage of the predetermined time. Whether or not can be determined with high accuracy.

(6) The dirt determination method consists of a step of periodically acquiring the photographed data 122 photographed by the monitoring camera 400, a step of extracting the feature amount of the acquired photographed data 122, and information based on the extracted feature amount. to the learned model 135 and outputting the degree of contamination of the lens of the monitoring camera 400; determining that the lens is in a dirty state with a certain amount of dirt when the degree of dirt does not become less than the threshold value even when the lens is dirty. By waiting for a predetermined period of time, erroneous detection due to factors other than dirt can be reduced, and it is possible to accurately determine whether or not the lens of the surveillance camera is in a dirty state with a certain degree of dirt.

(7) The dirt determination program provides the computer with a step of periodically acquiring the photographed data 122 photographed by the monitoring camera 400, a step of extracting the feature amount of the acquired photographed data 122, and a step of extracting the extracted feature amount to the learned model 135 and outputting the degree of contamination of the lens of the monitoring camera 400; and a step of determining that the lens is in a dirty state with a certain degree of dirt when the degree of dirt does not become less than the threshold value even after a lapse of time. By waiting for a predetermined period of time, erroneous detection due to factors other than dirt can be reduced, and it is possible to accurately determine whether or not the lens of the surveillance camera is in a dirty state with a certain degree of dirt.

The embodiments disclosed this time should be considered illustrative in all respects and not restrictive. The scope of the present disclosure is indicated by the scope of claims rather than the above description, and is intended to include all changes within the meaning and scope of equivalence to the scope of claims.

1 Dirt determination system, 51, 52 entrance, 53 passengers, 80a to 80i images, 100 video recording device, 111, 211 CPU, 112, 212 ROM, 113, 213 RAM, 114, 214 storage unit, 115, 215 communication interface, 116, 216 I/O interface, 121 acquisition unit, 122 captured data, 131 acquisition unit, 132 extraction unit, 133 normalization unit, 134 model generation unit, 135 learned model, 141 estimation unit, 142 estimation result, 143 determination unit , 144 notification unit, 200 dirt determination device, 300 terminal, 400 surveillance camera.

Claims

an acquisition unit that periodically acquires photographed data photographed by a surveillance camera;
an extraction unit that extracts the feature amount of the acquired photographed data;
an estimating unit that inputs information based on the extracted feature amount to a trained model and outputs the degree of contamination of the lens of the surveillance camera;
When the degree of contamination becomes equal to or greater than a threshold value related to the degree of contamination and the degree of contamination does not become less than the threshold even after a predetermined time has passed since the degree of contamination became equal to or greater than the threshold, A contamination determination device, comprising: a determination unit that determines that a contamination state is present.
The trained model includes a plurality of models,
Each of the plurality of models performs learning processing to output the degree of contamination when information based on the feature amount is input using a group of photographed data photographed in one of a plurality of photographing environments. is a model that has been done,
The contamination determination device according to claim 1, wherein the estimating unit selects one of the plurality of models based on the output result of the degree of contamination.
The contamination determination device according to claim 2, wherein the estimating unit selects, from among the plurality of models, a model that produces the smallest output result of the degree of contamination.
The contamination determination device can set an execution mode including a first mode and a second mode,
The determination unit is
When the first mode is set, a determination is made for each first cycle,
The dirt determination device according to any one of claims 1 to 3, wherein when the second mode is set, the determination is performed every second cycle longer than the first cycle.
The photographed data is data in which the inside of the elevator car is photographed,
5. The contamination according to any one of claims 1 to 4, wherein the degree of contamination varies according to at least one of the contamination of the lenses, the load capacity in the car, and the open/closed state of the door of the car. judgment device.
a step of periodically acquiring photographed data photographed by a surveillance camera;
a step of extracting a feature amount of the acquired photographed data;
a step of inputting information based on the extracted feature quantity into a trained model and outputting the degree of contamination of the lens of the surveillance camera;
When the degree of contamination becomes equal to or greater than a threshold value related to the degree of contamination and the degree of contamination does not become less than the threshold even after a predetermined time has passed since the degree of contamination became equal to or greater than the threshold, and determining that the presence of dirt is present.
to the computer,
a step of periodically acquiring photographed data photographed by a surveillance camera;
a step of extracting a feature amount of the acquired photographed data;
a step of inputting information based on the extracted feature quantity into a trained model and outputting the degree of contamination of the lens of the surveillance camera;
When the degree of contamination becomes equal to or greater than a threshold value related to the degree of contamination and the degree of contamination does not become less than the threshold even after a predetermined time has passed since the degree of contamination became equal to or greater than the threshold, A contamination determination program for executing a step of determining that a contamination state is present.