JP7455284B2 - Dirt determination device - Google Patents

Description

The present disclosure relates to a dirt determination device, a dirt determination method, and a dirt determination program.

Patent No. 6,616,906 (Patent Document 1) discloses a detection device that uses an identification model to determine whether photographic data photographed by a surveillance camera is defective photographic data that includes a defect. There is. This identification model is a model for extracting feature amounts from photographic data and detecting defective photographic data based on the extracted features. The defective photographic data is photographic data that was photographed when the lens of the camera was contaminated.

Patent No. 6616906

When making a determination using the above-mentioned identification model, there is a possibility that a change in an image that is not a defect may be recognized as defect data. For example, if you are photographing the inside of an elevator car, you may want to detect dirt on the surveillance camera lens; Other elements may be detected as dirt.

In this case, it is possible to increase the estimation accuracy by performing learning by increasing the amount of learning data taken with elements other than dirt removed. However, when the probability of occurrence of such a state is low, it is difficult to collect learning data and it becomes difficult to improve estimation accuracy.

The present disclosure has been made in order to solve such problems, and the purpose is to accurately determine whether or not the lens of a surveillance camera is in a dirty state where there is a certain amount of dirt. An object of the present invention is to provide a dirt determination device, a dirt determination method, and a dirt determination program.

A dirt determination device according to the present disclosure includes an acquisition section, an extraction section, an estimation section, and a determination section. The acquisition unit periodically acquires photographic data photographed by the surveillance camera. The extraction unit extracts feature amounts of the acquired photographic data. The estimation unit inputs information based on the extracted feature amounts to the trained model and outputs the degree of dirt on the lens of the surveillance camera. The determination unit determines that there is a certain level of dirt on the lens when the degree of dirt is equal to or greater than a threshold regarding the degree of dirt, and the degree of dirt does not become less than the threshold even after a predetermined time has elapsed since the degree of dirt became equal to or greater than the threshold. It is determined that there is dirt.

The dirt determination method according to the present disclosure includes the steps of periodically acquiring photographic data taken by a surveillance camera, extracting feature quantities of the acquired photographic data, and learning information based on the extracted feature quantities. The step of outputting the degree of contamination of the lens of the surveillance camera by inputting the degree of contamination to a model, and the step of outputting the degree of contamination of the lens of the surveillance camera. and determining that the lens is in a contamination state where there is a certain level of contamination if the degree of contamination does not become less than a threshold value.

A dirt determination program according to the present disclosure includes a step of periodically acquiring photographic data photographed by a surveillance camera, a step of extracting a feature amount of the acquired photographic data, and a step based on the extracted feature amount. a step of inputting information into a trained model and outputting the degree of contamination of the lens of the surveillance camera; and a step of inputting information into the trained model and outputting the degree of contamination of the lens of the surveillance camera; If the degree of contamination does not become less than the threshold value even if the degree of contamination does not become less than the threshold value, the step of determining that the lens is in a contamination state where there is a certain degree of contamination is executed.

According to the present disclosure, it is possible to accurately determine whether or not the lens of a surveillance camera is in a dirty state with a certain amount of dirt.

FIG. 1 is a diagram showing an example of a hardware configuration of a dirt determination system. FIG. 1 is a diagram showing an example of a functional block diagram of a dirt determination system. It is a flowchart of the 1st process performed by a dirt judging device. It is a flowchart of the judgment process which a dirt judgment device performs. FIG. 3 is a diagram for explaining various learned models. FIG. 3 is a diagram for explaining changes over time in the degree of contamination of a lens. FIG. 3 is a diagram for explaining the relationship between an image taken of the inside of a car and the degree of dirt. FIG. 3 is a diagram for explaining the relationship between an image taken of the inside of a car and the degree of dirt. FIG. 3 is a diagram for explaining the relationship between an image taken of the inside of a car and the degree of dirt. FIG. 3 is a diagram for explaining the relationship between an image taken of the inside of a car and the degree of dirt. FIG. 3 is a diagram for explaining the relationship between an image taken of the inside of a car and the degree of dirt. FIG. 6 is a diagram for explaining the relationship between an image of the inside of the car and the degree of dirt in the first process. FIG. 7 is a diagram for explaining the relationship between an image of the inside of the car and the degree of dirt in the second process. It is a flowchart of the 2nd process performed by a dirt judging device. FIG. 3 is a diagram for explaining the relationship between an image taken of the inside of a car and the degree of dirt.

Embodiments will be described below with reference to the drawings. In the following description, the same parts are given the same reference numerals. Their names and functions are also the same. Therefore, detailed descriptions thereof will not be repeated.

[Dirty determination system 1]
First, a dirt determination system 1 including a dirt determination device 200 according to the present embodiment will be described. FIG. 1 is a diagram showing an example of the hardware configuration of the stain determination system 1. As shown in FIG.

The dirt determination system 1 includes a video recording device 100, an elevator car 401, a surveillance camera 400 installed in the car 401, a dirt determination device 200, and a terminal 300. The video recording device 100 can communicate with the surveillance camera 400 and the dirt determination device 200. The terminal 300 can communicate with the dirt determination device 200.

The surveillance camera 400 is a camera for monitoring the inside of an elevator car 401 installed in a building. Surveillance camera 400 generates, for example, 30 frame images per second. The continuously generated frame images are played back as a moving image by being played back continuously.

The photographic data in this embodiment corresponds to a frame image. The video recording device 100 is a device that records photographic data (frame images) photographed by the surveillance camera 400 and determines the state of dirt on the lens of the surveillance camera 400.

Note that a plurality of cages may be installed within the building. In this case, a surveillance camera 400 is installed for each car. The video recording device 100 determines the state of dirt on the lens of the surveillance camera 400 for each car. Moreover, the surveillance camera 400 is not limited to one installed in the elevator car 401, but may be installed to monitor an elevator landing, an escalator, or other building equipment.

The video recording device 100 includes a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, a RAM (Random Access Memory) 113, a storage section 114, a communication interface 115, and an I/O interface 116. have These are communicably connected to each other via a bus.

The CPU 111 comprehensively controls the entire video recording device 100. The CPU 111 expands the program stored in the ROM 112 into the RAM 113 and executes it. The ROM 112 stores a program in which processing procedures for processing performed by the video recording device 100 are written.

The RAM 113 serves as a work area when the CPU 111 executes a program, and temporarily stores programs and data used when executing the program. Furthermore, the storage unit 114 is a nonvolatile storage device, such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

The video recording device 100 can communicate with the stain determination device 200 via the communication interface 115. The I/O interface 116 is an interface for connecting the CPU 111 to the surveillance camera 400.

The dirt determination device 200 includes a CPU 211 , a ROM 212 , a RAM 213 , a storage section 214 , a communication interface 215 , and an I/O interface 216 . These are communicably connected to each other via a bus.

The CPU 211 comprehensively controls the stain determination device 200 as a whole. The CPU 211 expands the program stored in the ROM 212 into the RAM 213 and executes it. The ROM 212 stores a program in which a procedure for processing performed by the stain determination device 200 is written.

The RAM 213 serves as a work area when the CPU 211 executes a program, and temporarily stores programs and data used when executing the program. Furthermore, the storage unit 214 is a nonvolatile storage device, such as an HDD or an SSD.

The dirt determination device 200 can communicate with the video recording device 100 and the terminal 300 via the communication interface 215. The I/O interface 216 is an interface for connecting the CPU 211 to a display device or an input device.

The display device is, for example, a display. On the display device, it is possible to check photographic data, the state of dirt on the lens, etc. The input device is, for example, a keyboard or a mouse. For example, instructions can be given to the stain determination device 200 by operating an input device.

Although not shown, the terminal 300 also includes a CPU, a ROM, a RAM, a storage section, a communication interface, and an I/O interface, like the dirt determination device 200. The terminal 300 may be a personal computer, a tablet terminal, a smartphone, or the like. Also on the terminal 300, it is possible to check the photographic data recorded on the surveillance camera 400, the state of dirt on the lens, etc.

FIG. 2 is a diagram showing an example of a functional block diagram of the stain determination system 1. The video recording device 100 includes an acquisition section 121. The acquisition unit 121 acquires photographic data taken by the surveillance camera 400 and stores it in the storage unit 114 as photographic data 122.

The dirt determination device 200 includes an acquisition section 131, an extraction section 132, a normalization section 133, an estimation section 141, a model generation section 134, a determination section 143, a prediction section 145, and a notification section 144.

Through these series of processes, the dirt determination device 200 generates the learned model 135 using the photographic data 122 photographed by the surveillance camera 400. It also acquires photographic data 122 photographed by the surveillance camera 400, and finally notifies the terminal 300 of the dirt determination result.

At the terminal 300, the determination result notified by the dirt determination device 200 can be confirmed. The storage unit 214 stores the trained model 135 generated by the model generation unit 134 and the estimation result 142 generated by the estimation unit 141. The flow of processing will be explained below using flowcharts and the like.

[Flowchart of first process]
The dirt determination device 200 can set execution modes including a first mode and a second mode. The determination unit 143 performs determination at every time T1 when the first mode is set. When the second mode is set, the determination unit 143 performs determination every time T2, which is longer than time T1. In this embodiment, time T1=10 minutes and time T2=1 day.

Specifically, the dirt determination device 200 starts the first process when the first mode is set, and starts the second process when the second mode is set. In this embodiment, it is assumed that both the first mode and the second mode are set.

FIG. 3 is a flowchart of the first process executed by the stain determination device 200. The first process is a process of determining dirt at a period of time T1. Hereinafter, the "step" will also be simply referred to as "S".

As shown in FIG. 3, when the first process starts, in S11, the stain determination device 200 determines whether or not time T1 has elapsed. When the stain determination device 200 determines that the time T1 has elapsed (YES in S11), the process proceeds to S12. When the stain determination device 200 does not determine that the time T1 has elapsed (NO in S11), the process returns to S11.

In S12, the stain determination device 200 performs determination processing (see FIG. 4). The determination process is a process of determining the dirt status of the lens of the surveillance camera 400 based on the photographic data 122 photographed by the surveillance camera 400.

In S13, the notification unit 144 notifies the terminal 300 of the determination result of the determination unit 143, and returns the process to S11. In this way, in the first process, the determination process is performed every time the time T1 elapses.

FIG. 4 is a flowchart of the determination process executed by the stain determination device 200. As shown in FIG. 4, when the determination process starts, in S21, the acquisition unit 131 acquires the photographic data 122 photographed by the surveillance camera 400, and advances the process to S22. The photographic data 122 is data obtained by photographing the inside of the elevator car 401.

In S22, the extraction unit 132 extracts the feature amount of the acquired photographic data 122, and advances the process to S23. In S23, the normalization unit 133 normalizes the extracted feature amount, and advances the process to S24. The processing of the extraction unit 132 and the normalization unit 133 is similar to the processing during learning, which will be described later with reference to FIG.

The estimation unit 141 inputs information based on the extracted feature amounts to the learned model 135 and outputs the degree of dirt on the lens of the surveillance camera 400. The degree of dirtiness indicates how dirty the lens of the surveillance camera 400 is using a numerical value between 0 and 1.

Specifically, in S24, the estimation unit 141 uses the normalized data as information based on the extracted feature amount for all the trained models 135 stored in the storage unit 214 (the trained models 135 in FIG. (Models A to E, etc.), the degree of contamination is output for each, and the process proceeds to S25. In S25, the estimation unit 141 selects the smallest degree of contamination among all the output degrees of contamination, and advances the process to S26.

In this way, the estimation unit 141 selects one of all the learned models 135 (learned models A to E, etc.) based on the output result of the degree of contamination. Specifically, the estimating unit 141 selects the model with the smallest output result from among the plurality of models. The details will be described later using FIG. 5, but the reason is that the model that provides the smallest output result (degree of contamination) is considered to be the optimal model.

The determining unit 143 determines that dirt is present when the dirt degree becomes equal to or greater than the threshold value and becomes less than the threshold value before a predetermined time (e.g., 8 minutes) elapses after the dirt degree becomes equal to or greater than the threshold value. (S26 to S28 below).

Details will be described later with reference to FIGS. 6 to 10, including specific examples, but in this embodiment, elements unrelated to dirt may be erroneously detected as "stain". Since it is assumed that these elements become undetectable over time, the change in the degree of contamination is checked until a predetermined period of time has elapsed.

Here, the "threshold value" is a threshold value set in advance regarding the degree of contamination, and in this embodiment, "0.3" is set. Moreover, the "stain state" is a state in which the lens of the surveillance camera 400 has a certain amount of dirt, and specifically refers to a state in which the degree of dirt is equal to or higher than a threshold value.

In S26, the estimating unit 141 determines whether the degree of contamination becomes equal to or greater than the threshold value and whether the degree of contamination does not become less than the threshold value even after a predetermined period of time has elapsed since the degree of contamination became equal to or greater than the threshold value.

Specifically, if the degree of contamination is equal to or greater than the threshold value, steps S21 to S25 are repeated at specific time intervals (for example, 30 seconds) until a predetermined time period (8 minutes) has elapsed. Then, among the degrees of contamination obtained before a predetermined period (8 minutes) has elapsed, the smallest degree of contamination is selected as the correct degree of contamination. In S26, it is determined whether the value selected as the correct degree of dirt is less than a threshold value.

If the estimation unit 141 determines that the degree of contamination is equal to or greater than the threshold value and that the degree of contamination does not become less than the threshold value even after a predetermined time has elapsed since the degree of contamination exceeded the threshold value (YES in S26), the estimation unit 141 performs processing. Proceed to S27. In S27, the determination unit 143 determines that there is dirt, and ends the determination process.

The estimating unit 141 did not determine that the degree of contamination is equal to or greater than the threshold value, and that the degree of contamination does not become less than the threshold value even after a predetermined period of time has passed since the degree of contamination has exceeded the threshold value (that is, the degree of contamination does not become less than the threshold value even after the degree of contamination has exceeded the threshold value). (NO in S26), the process advances to S28. In S28, the determination unit 143 determines that there is no dirt, and ends the determination process.

After confirming that there is dirt on the terminal 300, the elevator maintenance worker or the building manager cleans the lens of the surveillance camera 400 to remove the dirt attached thereto.

[Trained model]
FIG. 5 is a diagram for explaining various learned models. The learned model 135 is a model that has been subjected to a learning process to output the degree of contamination when information based on the feature amount is input.

As shown in FIG. 5, the trained models 135 include trained models A to E and the like. Each of the trained models A to E, etc. is a model that has been subjected to learning processing using a group of photographic data 122 photographed in any of a plurality of photographing environments.

The trained model A is a model in which learning processing has been performed using 122 groups of photographic data taken in the morning as a photographing environment. The trained model B is a model in which learning processing has been performed using 122 groups of photographic data taken in the daytime as a photographing environment. The learned model C is a model in which a learning process is performed using 122 groups of photographic data taken at night as a photographing environment. The brightness inside the car 401 differs in the morning, noon, and night depending on the amount of light entering.

The trained model D is a model in which a learning process is performed using a group of photographic data 122 photographed with the door of the car 401 open (door open state) as a photographic environment. The trained model E is a model in which a learning process is performed using a group of photographic data 122 photographed with the door of the car 401 closed (door closed state) as a photographic environment.

In this embodiment, learning processing is performed using a set of photographic data (a group of photographic data) whose degree of contamination has been determined in advance. The acquisition unit 131 acquires photographed data 122 whose degree of contamination has been determined in advance as learning data.

At this time, for example, when generating the learned model D (door open state), the degree of dirt has been determined in advance and photographic data 122 taken with the door open is acquired as learning data. The photographic data 122 for learning processing may be stored in the storage unit 114 as teacher data after linking the degree of dirt and the photographing environment in advance.

Next, the extraction unit 132 extracts the feature amount of the acquired photographic data 122. For example, the feature amount may be extracted based on the color characteristics of the photographic data 122. When the photographic data 122 is a color image, each pixel has color characteristic data. Specifically, there are saturation, brightness, hue, etc. In addition, the overall photographic data includes color scheme, variation, structure, etc. By combining these color characteristics, feature amounts that show a difference between a low degree of contamination and a high degree of contamination are extracted. If the degree of contamination is high, it is assumed that the image of the contamination attached to the lens (the photographed area of the contamination portion) will be black, that is, the brightness will be reduced. Therefore, it is thought that the feature amount will be significantly different compared to the case where no dirt is attached. In this way, feature amounts may be extracted such that differences appear when the degree of contamination is high.

Next, the normalization unit 133 normalizes the feature amount extracted by the extraction unit 132. The model generation unit 134 performs a learning process based on the normalized photographic data 122 and the linked degree of dirt to generate a learned model.

If the shooting environment of the shooting data to be determined is close to the shooting environment of the learned model, it can be expected that the judgment accuracy of the judgment unit 143 will be high. For example, as shown in FIG. 7, if the acquired photographic data was photographed with the door open, then the learned model D is subjected to learning processing using 122 groups of photographic data photographed with the door open. By using , it can be expected that the determination accuracy will be higher.

It can be expected that the model for which the calculated degree of contamination is the smallest is the model for which the determination unit 143 has the highest determination accuracy. For example, because the condition of the door is different between photographic data taken with the door open and trained model E using photographic data with the door closed, there is a possibility that the degree of dirt will be incorrectly detected. is high. On the other hand, since the door condition is the same between photographic data taken with the door open and trained model D that also uses photographic data with the door open, the degree of dirt in the door area is Less chance of false positives.

Therefore, as described above, the estimation unit 141 uses all the learned models 135 (learned models A to E, etc.) and outputs the degree of dirt for each. Then, the model with the smallest degree of contamination is selected as the optimal model (the lowest degree of contamination is determined to be the correct degree of contamination).

Note that the trained models are not limited to the trained models A to E above, but also include models in which learning processing was performed with a passenger on board, and models in which learning processing was performed in an environment in which multiple shooting environments were combined. May include. Alternatively, the estimation unit 141 may output the degree of contamination using only one learned model. In this case, the learning process becomes simple.

Note that similar regions in the image may be divided as a group. In this case, the estimation unit 141 outputs the degree of contamination for each divided area. The estimating unit 141 calculates, for each area, "stain level" x "proportion occupied by the divided area in the entire image", and adds these together to calculate the "stain level".

[Time change in degree of contamination]
FIG. 6 is a diagram for explaining the change over time in the degree of contamination of the lens. In FIG. 6, the vertical axis indicates the degree of contamination. The horizontal axis indicates time.

As shown in FIG. 6, when the lens is not dirty at all (also referred to as "normal time"), the degree of dirt is 0. The threshold value is 0.3.

At time t1 (scene 1), the degree of contamination is 0. After that, scene 2 occurs at time t2 when the degree of contamination rapidly increases to 0.9. At this time, the degree of contamination is equal to or higher than the threshold value (0.3). After that, scene 3 occurs at time t3 when the degree of contamination rapidly decreases to 0.1. At this time, the degree of contamination (0.1) is below the threshold value (0.3). In such a case, it is determined that there is no dirt.

Next, scene 4 occurs at time t4 when the degree of contamination rapidly increases to 0.8. At this time, the degree of contamination (0.8) is equal to or higher than the threshold value (0.3). After that, scene 5 occurs at time t5 when the degree of contamination rapidly decreases to 0.4. At this time, the degree of contamination (0.4) remains equal to or higher than the threshold value (0.3). In such a case, it is determined that there is dirt.

Below, the situations of scenes 1 to 5 will be explained using images taken inside the car 401. 7 to 11 are diagrams for explaining the relationship between images taken inside the car 401 and the degree of dirt.

In scene 1, as shown in FIG. 7, in an image 80a (photographed data) of the inside of the car 401, the door of the car 401 is in an open state (door open state) at the entrance 51. There are no passengers in the car 401. Furthermore, there is no dirt attached to the lens of the surveillance camera 400, and the dirt degree=0 is output.

In scene 2, as shown in FIG. 8, in an image 80b taken inside the car 401, four passengers 53 get into the car 401, and the door 51 of the car 401 is in a closed state (door closed state). ing. Also, the lens has dirt attached to it. At this time, the degree of contamination has increased to 0.9.

In scene 3, as shown in FIG. 9, in an image 80c taken of the inside of the car 401, the door of the entrance 51 is open (door open state), and all passengers in the car 401 are getting off the car. At this time, the lens is contaminated to the same extent as in Scene 2. At this time, the degree of contamination has decreased to 0.1.

In both scenes 1 and 3, the door is open and there are no passengers. Since the difference between the images of Scene 1 and Scene 3 is only the dirt on the lens, it is considered that the dirt degree = 0.1 is purely due to the dirt on the lens.

On the other hand, scenes 2 and 3 are the same in that the lenses are dirty, but they differ in that in scene 2 the door is closed and there are four passengers. Therefore, the reason why the degree of contamination is 0.9 in scene 2 is considered to include not only the contamination of the lens but also the open/closed state of the door of the car 401 and the false detection caused by the passenger inside the car 401.

As described above, in this embodiment, when the car 401 is in an open state, there are 0 passengers, and the lenses are not dirty (scene 1), the degree of dirt is calculated to be 0. After that, dirt with a dirt level of 0.1 adheres to the lens, but when four more passengers get on board and the door is closed, the dirt level increases to 0.9.

In this case, although the degree of contamination is higher than the threshold value (0.3), in reality, the degree of contamination is only increasing due to an image change due to the door being closed and a passenger getting on board, rather than due to contamination. be. Therefore, in scene 3 when the door is open and the number of passengers is zero, only pure lens dirt (stain level = 0.1) is detected.

In this way, the degree of contamination changes depending on at least one of the contamination of the lens, the load inside the car 401 (for example, the number of passengers), and the open/closed state of the door of the car 401. Therefore, in the present embodiment, the estimation unit 141 determines that the degree of contamination becomes equal to or greater than the threshold value, and that the degree of contamination becomes less than the threshold value by the time a predetermined time elapses after the degree of contamination becomes equal to or greater than the threshold value. If so, it is determined that there is no dirt (S26, S28). Note that various factors that cause the degree of contamination to change will be explained in more detail using FIG. 15.

Next, in scene 4, as shown in FIG. 10, in an image 80d taken inside the car 401, two passengers 53 are in the car 401, and the door is closed. Furthermore, the lower left corner of the lens of the surveillance camera 400 has been soiled due to a prank by the passenger 53. At this time, the degree of contamination has increased to 0.8.

In scene 5, as shown in FIG. 11, in an image 80e of the interior of the car 401, the door is open and all passengers in the car 401 have exited the car. At this time, the degree of contamination has decreased to 0.4.

In both scenes 3 and 5, the door is open and there are no passengers. The only difference between the images of Scene 3 and Scene 5 is the dirt on the lower left of the lens, which was added as a prank by passenger 53. As a result, the degree of contamination has increased from 0.1 to 0.4, which exceeds the threshold (0.3).

On the other hand, scenes 4 and 5 are the same in that the lenses are dirty, but they differ in that in scene 4 the door is closed and there are two passengers. Therefore, the reason why the degree of contamination is 0.8 in scene 4 is considered to include not only the contamination of the lens but also the open/closed state of the door of the car 401 and the false detection caused by the passenger inside the car 401.

In this way, when the door is open, there are no passengers, and the lens is dirty (scene 3), the degree of dirt is calculated to be 0.1. After that, two passengers boarded the vehicle, the door was closed, and the lens became dirty due to mischief, increasing the degree of contamination to 0.8.

In this case, the degree of contamination exceeds the threshold value (0.3), but it also includes an increase in the degree of contamination due to an image change caused by the door being closed and a passenger getting on board. Therefore, in scene 5 when the door is open and the number of passengers is zero, only pure lens dirt (stain level = 0.4) is detected. Also in this case, since the degree of contamination (0.4) does not become less than the threshold value (0.3), it is determined that there is contamination.

In this way, the determining unit 143 determines that the dirt presence state is present when the dirt degree is equal to or greater than the threshold and the dirt degree does not become less than the threshold even after a predetermined time has elapsed since the dirt degree became equal to or greater than the threshold. It is determined that (S26, S27).

If you try to estimate the degree of dirt using a trained model, there is a possibility that false detections (increasing values) may occur due to factors other than dirt (for example, door opening/closing status or passengers). . In this case, in order to be able to distinguish between dirt and factors other than dirt, it is possible to improve the accuracy of estimating the degree of dirt by performing a more accurate learning process. To this end, it is possible to improve the accuracy of learning by increasing the amount of learning data with elements other than dirt removed. However, when the probability of occurrence of such a state is low, it is difficult to collect learning data and it becomes difficult to improve estimation accuracy.

For this reason, in the present embodiment, a method other than estimation using the trained model is also used to exclude factors other than dirt, thereby increasing the accuracy of estimating the degree of dirt. As described above, the degree of contamination due to "dirt" does not decrease over time, and the degree of contamination due to "elements other than contamination" decreases over time. For example, the "door open/closed state", which is a factor other than dirt, returns to the door open state over time, and the number of passengers returns to zero over time. In this way, cases occur where the threshold value is exceeded due to factors other than dirt. Therefore, by waiting for a predetermined period of time, false detections due to factors other than dirt are reduced, and the system is configured to detect the degree of dirt only due to dirt as much as possible. By doing so, it is possible to accurately determine whether or not there is dirt without adding learning data.

Next, the relationship between the images taken inside the car 401 and the degree of dirt in the first process and the second process will be described. FIG. 12 is a diagram for explaining the relationship between the image taken inside the car 401 and the degree of dirt in the first process.

In the first process, as explained using the flowcharts of FIGS. 3 and 4, the determination process is performed every T1 (10 minutes). On the other hand, in the second process, the determination process is performed every T2 (one day) as described later using the flowchart of FIG.

First, an example in which the determination process is performed every 10 minutes in the first process will be described. As shown in FIG. 12, in the image 80j, the doorway 51 is in a closed state, and there is no passenger 53 inside the car 401. Furthermore, there is no dirt attached to the lens of the surveillance camera 400, and the degree of dirt is 0.

After this state, determination processing is performed every 10 minutes. At that time, if it is determined that the degree of contamination is equal to or greater than the threshold value, it is determined whether or not the degree of contamination remains equal to or greater than the threshold value even after a predetermined period of time has elapsed. As explained using FIGS. 6 to 11, even if it is determined that the degree of contamination is equal to or higher than the threshold value as in scene 2 (t2), scene 3 (t3) until the predetermined time elapses. If the degree of contamination is less than the threshold value, it is determined that there is no contamination.

On the other hand, in the case where the degree of contamination is determined to be equal to or higher than the threshold value as in scene 4 (t4), the degree of contamination remains equal to or greater than the threshold value in scene 3 (t5) even after the predetermined time has elapsed. , it is determined that there is dirt.

In the state of the image 80j, for example, suppose that the lens of the surveillance camera 400 is dirty due to a prank by a passenger. An image 80k is a state 10 minutes have passed since the state of the image 80j. In the image 80k, the doorway 51 is in a closed state, and there is no passenger 53 inside the car 401. However, there is dirt on the right side of the lens of the surveillance camera 400, and the dirt level is 0.4. In this case, since the degree of contamination is equal to or greater than the threshold value (0.3), it is determined that there is contamination.

In this way, in the first process, since the determination process is performed every 10 minutes, it is possible to detect sudden stains such as when the lens is dirty due to a prank.

Next, an example in which the determination process is performed every day in the second process will be described. FIG. 13 is a diagram for explaining the relationship between the image taken inside the car 401 and the degree of dirt in the second process. As shown in FIG. 13, in the image 80f, the doorway 51 is in an open state, and there is no passenger 53 inside the car 401. Furthermore, there is no dirt attached to the lens of the surveillance camera 400, and the degree of dirt is 0.

The determination process is executed every day, and the image 80g is the state in which 30 days have passed (the determination process has been performed 30 times) from the state of the image 80f. Also in the image 80g, the doorway 51 is in an open state, and there is no passenger 53 inside the car 401. However, the lens of the surveillance camera 400 is contaminated, and the degree of contamination is 0.1.

Note that, if there is a passenger at the timing when the determination process is executed and the level becomes equal to or higher than the threshold value, it is determined whether the degree of contamination becomes less than the threshold value within a predetermined period of time. In this example, the degree of contamination decreases to 0.1 at the timing when the passenger gets off the vehicle.

An image 80h is a state in which 30 days have passed since the state of the image 80g (30 determination processes have been performed). Counting from the state of image 80f, 60 days have passed. In the image 80h, the doorway 51 is in an open state, and there is no passenger 53 inside the car 401. However, more dirt has accumulated on the lens of the surveillance camera 400, and the dirt level is 0.2.

The image 80i is a state in which 30 days have passed since the state of the image 80h (30 determination processes have been performed). Counting from the state of image 80f, 90 days have passed. Also in the image 80g, the doorway 51 is in an open state, and there is no passenger 53 inside the car 401. However, more dirt has accumulated on the lens of the surveillance camera 400, and the dirt level is 0.3. In this case, since the degree of contamination has reached the threshold value, it is determined that there is contamination.

According to images 80f to 80i, the degree of contamination increases by 0.1 every 30 days. Therefore, for example, based on the change in the degree of contamination over time from day 0 to day 60, it is possible to predict that the degree of contamination will reach the threshold value after another 30 days or 90 days. This allows maintenance personnel to know when to clean the lens. The second process will be explained below using a flowchart.

[Flowchart of second process]
As described above, the first process is a process of determining dirt at intervals of time T1 (10 minutes). On the other hand, in the second process, dirt is determined at a cycle of time T2 (one day). Furthermore, the second process estimates future lens dirt.

The second process will be explained below. FIG. 14 is a flowchart of the second process executed by the stain determination device 200.

As shown in FIG. 14, when the second process starts, in S31, the stain determination device 200 determines whether or not time T2 has elapsed. When the stain determination device 200 determines that the time T2 has elapsed (YES in S31), the process proceeds to S32. When the stain determination device 200 does not determine that the time T2 has elapsed (NO in S31), the process returns to S31.

In S32, the dirt determination device 200 performs a determination process and proceeds to S33. In other words, the dirt determination device 200 performs a determination process, etc., every time the time T2 elapses.

In S33, the prediction unit 145 of the stain determination device 200 stores the degree of stain in the storage unit 214, and advances the process to S33.

In S34, the prediction unit 145 estimates the time when it is determined that there is dirt based on the time series data of the degree of dirt stored in the storage unit 214, and the process proceeds to S35.

In the example shown using FIG. 13, the degree of staining is 0 on day 0. On the 30th day, the degree of staining was 0.1. On the 60th day, the degree of staining was 0.2. Assume that it is currently the 60th day. The storage unit 214 stores data on the degree of contamination from day 0 to day 60.

According to these data, the degree of contamination increases by 0.1 every 30 days, so the prediction unit 145 predicts that the degree of contamination will reach the threshold value of 0.3 on the 90th day (30 days later). do. In other words, it is predicted that the time when it will be determined that there is dirt is 30 days later.

For example, the prediction unit 145 may generate a prediction model using the least squares method or the like using time-series data on the degree of contamination. The prediction unit 145 uses the prediction model to predict the time when it will be determined that there is dirt.

In S35, the notification unit 144 transmits the determination result of the determination unit 143 and the estimation result of the prediction unit 145 to the terminal 300, and returns the process to S31.

[Various factors that change the degree of contamination]
As explained in FIGS. 6 to 11, the degree of contamination changes depending on at least one of "dirty lenses", "loading amount in the car 401", and "open/closed state of the door of the car 401". do. Here, various factors that cause the degree of contamination to change will be explained in more detail using FIG. 15.

FIG. 15 is a diagram for explaining the relationship between an image taken inside the car 401 and the degree of dirt. As shown in FIG. 15, in an image 80j of the inside of the car 401, the doorway 51 is in an open state, and there are no passengers inside the car 401. Furthermore, there is no dirt attached to the lens of the surveillance camera 400, and the dirt degree=0 is output.

In the examples shown in FIGS. 6 to 11, an example was explained in which a passenger boarded the car 401. However, as shown in an image 80k taken inside the car 401, there is a case where a large baggage is loaded in the car 401 along with the passenger. There is also. The above-mentioned "loading amount in the car 401" includes not only the number of passengers but also the amount of luggage. Note that the "loading amount in the car 401" may indicate the total area of passengers and luggage estimated from the image 80k.

In the example of image 80k, the degree of dirt has increased to 0.2 due to luggage being loaded into the car 401 along with passengers. In this case, the degree of contamination changes depending on the load inside the car 401, but when the passengers and luggage arrive at the destination floor they exit the car 401, so the degree of contamination returns to zero again. Therefore, it is not determined that the vehicle is in a "dirty state" based on the passengers and luggage.

Moreover, the above-mentioned "door opening/closing state of the car 401" also includes the "landing state". In the image 80l taken inside the car 401, the door is in an open state like the image 80j, but the state of the landing has changed. Specifically, in the image 80j, the landing area is bright due to sunlight during the day, whereas in the image 80l, the dirt level has increased to 0.15 because the landing area is dark at night. As described above, the degree of contamination differs between when the door is open and when the door is closed, and when the door is open, the degree of contamination also differs depending on the time of day.

In the example shown in Fig. 5, different trained models are used for morning, noon, and night shooting environments (trained models A to C), and different trained models are used for the door open state and the door closed state (learned models A to C). By configuring the models D and E), the accuracy of determining the degree of contamination is improved. In this example, for example, by making the learning model different depending on the time period when the door is open, it is possible to further improve the accuracy of determining the degree of dirt.

Further, the "landing state" includes the crowded state of the landing. For example, the degree of dirtiness is different during peak hours such as when people come to work and when they leave work (when multiple waiting passengers are visible when the door is open) and during quiet times (when there are no people on board). Even in this case, for example, the accuracy of determining the degree of dirt can be further improved by using different learning models for peak time periods and other time periods as the photographing environment.

In addition to the above, the degree of contamination also changes depending on "the state of the surveillance camera 400 installed in the car 401." For example, in the image 80m taken inside the car 401, the installation direction of the surveillance camera 400 installed inside the car 401 has shifted due to a prank by a passenger, and the slightly left side of the car 401 is photographed compared to the image 80j. ing. As a result, the degree of contamination changes to 0.4 due to the difference between the image 80j and the image 80m.

In this case, since the degree of contamination does not decrease even after a predetermined period of time has elapsed, it is erroneously detected as a "contamination presence state". In this case, the elevator maintenance worker or the building manager checks the installation state of the surveillance camera 400 and performs work to return the surveillance camera 400 to the normal installation state. As a result, the degree of contamination returns to 0, and the degree of contamination can be detected normally. The "state of the surveillance camera 400 installed in the car 401" may be the installation direction and position of the surveillance camera 400 as described above, or a state where the surveillance camera 400 is out of focus. It can be assumed.

[Main composition and effects]
The main configuration and effects of the embodiment described above will be explained below.

(1) The stain determination device 200 includes an acquisition section 131, an extraction section 132, an estimation section 141, and a determination section 143. The acquisition unit 131 periodically acquires photographic data 122 photographed by the surveillance camera 400. The extraction unit 132 extracts the feature amount of the acquired photographic data 122. The estimation unit 141 inputs information based on the extracted feature amounts to the learned model 135 and outputs the degree of dirt on the lens of the surveillance camera 400. The determination unit 143 determines that a certain amount of dirt is present on the lens when the degree of dirt is equal to or higher than a threshold regarding the degree of dirt, and the degree of dirt does not become less than the threshold even after a predetermined period of time has passed since the degree of dirt became equal to or greater than the threshold. It is determined that there is a certain dirt state. By waiting for a predetermined period of time, false detections due to factors other than dirt can be reduced, and it is possible to accurately determine whether or not there is a certain amount of dirt on the lens of the surveillance camera.

(2) The trained model 135 includes a plurality of models (trained models A to E, etc.). Each of the plurality of models uses 122 groups of photographic data taken in one of a plurality of photographing environments, and performs a learning process to output the degree of contamination when information based on feature quantities is input. It is a model that The estimation unit 141 selects one of the plurality of models and outputs the degree of dirt using the selected model. By using an appropriate model according to the shooting environment, false detections due to changes in the surrounding environment can be reduced, and it is possible to determine whether or not there is dirt.

(3) The estimation unit 141 selects the model that provides the smallest output result from among the plurality of models. By using an appropriate model according to the shooting environment, false detections due to changes in the surrounding environment can be reduced, and it is possible to determine whether or not there is dirt.

(4) The dirt determination device 200 can set execution modes including a first mode and a second mode. If the first mode is set, the determination unit 143 performs determination every first cycle. When the second mode is set, the determination unit 143 performs determination every second period that is longer than the first period. Thereby, dirt that suddenly occurs can be detected in the first mode, and dirt that accumulates over time can be detected in the second mode. By detecting dirt that accumulates over time, it is possible to predict when the lens will need to be cleaned due to dirt.

(5) The photographic data 122 is data obtained by photographing the inside of the elevator car 401. The degree of dirt changes depending on at least one of the dirt on the lens, the loading amount (number of passengers and amount of luggage) in the car 401, and the open/closed state of the door of the car 401. This allows us to identify dirt on the lens that does not reduce the degree of dirt even after a predetermined period of time has passed, and the loading amount and door opening/closing conditions that may reduce the degree of dirt before the elapse of a predetermined period of time. It is possible to accurately determine whether or not it is.

(6) The stain determination method includes the steps of periodically acquiring photographic data 122 photographed by the surveillance camera 400, extracting feature quantities of the acquired photographic data 122, and information based on the extracted feature quantities. is inputted into the learned model 135 to output the degree of contamination of the lens of the surveillance camera 400, and the degree of contamination is equal to or higher than a threshold value regarding the degree of contamination, and a predetermined period of time has elapsed since the degree of contamination exceeds the threshold value. If the degree of contamination does not become less than a threshold value even if the degree of contamination does not become less than a threshold value, determining that the lens is in a contamination state where there is a certain degree of contamination. By waiting for a predetermined period of time, false detections due to factors other than dirt can be reduced, and it is possible to accurately determine whether or not there is a certain amount of dirt on the lens of the surveillance camera.

(7) The dirt determination program includes a step of periodically acquiring photographic data 122 photographed by the surveillance camera 400, a step of extracting a feature amount of the acquired photographic data 122, and a step of extracting the extracted feature amount. inputting information based on the learned model 135 into the learned model 135 and outputting the degree of contamination of the lens of the surveillance camera 400; If the degree of contamination does not become less than the threshold value even after a period of time has elapsed, a step of determining that the lens is in a contamination state with a certain degree of contamination is executed. By waiting for a predetermined period of time, false detections due to factors other than dirt can be reduced, and it is possible to accurately determine whether or not there is a certain amount of dirt on the lens of the surveillance camera.

The embodiments disclosed this time should be considered to be illustrative in all respects and not restrictive. The scope of the present disclosure is indicated by the claims rather than the above description, and it is intended that equivalent meanings and all changes within the scope of the claims are included.

1 dirt determination system, 51, 52 doorway, 53 passenger, 80a-80i image, 100 video recording device, 111,211 CPU, 112,212 ROM, 113,213 RAM, 114,214 storage unit, 115,215 communication interface, 116, 216 I/O interface, 121 acquisition unit, 122 imaging data, 131 acquisition unit, 132 extraction unit, 133 normalization unit, 134 model generation unit, 135 learned model, 141 estimation unit, 142 estimation result, 143 determination unit , 144 notification unit, 200 dirt determination device, 300 terminal, 400 surveillance camera.

Claims (4)

an acquisition unit that periodically acquires photographic data photographed by a surveillance camera;
an extraction unit that extracts a feature amount of the acquired photographic data;
an estimator that inputs information based on the extracted feature amounts into a learned model and outputs a degree of dirt on a lens of the surveillance camera;
When the degree of contamination becomes equal to or greater than the threshold value regarding the degree of contamination, and the degree of contamination does not become less than the threshold value even after a predetermined period of time has elapsed since the degree of contamination became equal to or greater than the threshold value, a certain level of contamination is applied to the lens. and a determination unit that determines that there is dirt.
The trained model includes a plurality of models,
Each of the plurality of models performs a learning process to output the degree of contamination when information based on the feature amount is input using a group of photographic data taken in one of a plurality of photographing environments. It is a model that was carried out,
The estimating unit is a dirt determination device that selects one of the plurality of models based on an output result of the dirt degree by each of the plurality of models. The dirt determination device according to claim 1, wherein the estimation unit selects a model that provides the smallest output result of the dirt degree from among the plurality of models. The dirt determination device is capable of setting execution modes including a first mode and a second mode,
The determination unit includes:
If the first mode is set, a determination is made every first cycle,
The dirt determination device according to claim 1 or 2, wherein when the second mode is set, the determination is performed every second cycle that is longer than the first cycle. The photographic data is data in which the inside of an elevator car is photographed;
The dirt according to any one of claims 1 to 3, wherein the degree of dirt changes depending on at least one of the dirt on the lens, the loading amount in the car, and the open/closed state of the door of the car. Judgment device.

Images