JP6989294B2 - Monitoring system and monitoring method - Google Patents

Description

The present disclosure relates to a monitoring system and a monitoring method in which a server and a plurality of cameras installed in a monitoring area are connected so as to be able to communicate with each other.

Currently, the computing power of the camera is said to be 0.3T (tera) ops. T (tera) is a value indicating 10 to the 12th power. Ops is known as a unit indicating arithmetic processing power. In the future, it is considered that high-performance GPUs (Graphics Processing Units) and FPGAs (Field Programmable Gate Arrays) mounted on game machines and the like will be adopted for use in camera arithmetic processing units. In that case, for example, after one year, it is expected that the arithmetic processing capacity of the camera will be dramatically improved to about 2.6 Tops, which is more than 10 times.

Further, it has been pointed out that when the camera performs image recognition processing using deep learning as an example of machine learning, 1.3 Tops is required for the arithmetic processing capacity of the camera. Due to this high computing power, it was conventionally thought that it would be difficult for a camera to perform image recognition processing using deep learning, but one year later, the camera's computing power will use deep learning. It is considered possible to perform image recognition processing.

On the other hand, when high-quality (for example, 4K) captured image data captured by a camera is transferred to a server and the server performs image recognition processing, the amount of communication transmitted on the network as the size of the captured image data increases. (Traffic) also inevitably increases, and as a result, communication efficiency decreases and delays occur. Therefore, it is expected that the camera performs image recognition processing by using deep learning in its own device without transferring high-quality (for example, 4K) captured image data.

Generally, when image recognition processing is performed using deep learning, a device such as a camera learns an object (that is, an object to be recognized) contained in captured image data, and model parameters used in the image recognition processing (that is, an object to be recognized). For example, the training model is updated by changing the weight coefficient and threshold value). A device such as a camera improves the accuracy of detecting an object (that is, an object to be recognized as an image) contained in the captured image data based on this updated learning model.

As a prior art for recognizing an object using captured image data captured by a camera and acquiring object movement information, for example, an object tracking device of Patent Document 1 has been proposed. This object tracking device uses a time-series image group acquired from a camera capable of photographing an object, and includes image information relating to the acquired image and position information relating to the position of the object in real space. Learning is performed by a teacher data set that includes motion information and information that is considered to be the correct answer. Further, the object tracking device uses a tracking classifier that outputs at least the position information that is considered to be the correct answer in the real space of the object by inputting the image information related to the image for each image to be tracked. Acquires the momentary position information of the object in the real space.

Japanese Unexamined Patent Publication No. 2016-206795

In the future, it is expected that the captured image data handled by the camera will have a high definition and a large capacity such as 4K or 8K, and the data size will increase. With the increase in the size of the captured image data, if the parameters used for detecting the captured image data are learned by the server instead of the camera, the processing load will be concentrated on the server, and a large amount of data will be stored. By transmitting to the server one by one, the traffic on the network increases, which causes a problem that a corresponding delay occurs during data communication. No particular consideration has been given to the technical measures for such problems in the prior art as in Patent Document 1.

The present disclosure has been devised in view of the above-mentioned conventional circumstances, and when detecting at least one object in each captured image captured by a plurality of cameras installed in the monitoring area, learning of parameters used for the detection. The purpose is to provide a monitoring system and a monitoring method that distributes such processing among a plurality of cameras, suppresses an increase in traffic on the network, and helps reduce the processing load of a server connected to a plurality of cameras. do.

The present disclosure is a monitoring system in which a server and a plurality of cameras installed in a monitoring area are connected to each other so as to be able to communicate with each other, and the server is a free resource for the processing capacity for generating a learning model of each of the cameras. It has a memory for holding information about the amount of the camera and an image captured by imaging the surveillance area by each camera, and by each camera based on information about the amount of free resources of the camera. The process to be executed by the camera with respect to the detection of at least one object appearing in the obtained captured image is determined for each camera, and the determined execution instruction of the process is transmitted for each camera. Provided is a monitoring system that executes a process corresponding to the execution instruction based on the execution instruction of the process transmitted from the server.

Further, the present disclosure is a monitoring method using a monitoring system in which a server and a plurality of cameras installed in a monitoring area are connected so as to be able to communicate with each other, and the server uses a learning model of each of the cameras. Information on the amount of free resources for the processing power to be generated and the captured images obtained by imaging the surveillance area by each of the cameras are held in memory, and based on the information on the amount of free resources of the cameras. The process to be executed by the camera with respect to the detection of at least one object appearing in the captured image obtained by each of the cameras is determined for each camera, and the determined execution instruction of the process is transmitted for each camera. Each camera provides a monitoring method that executes a process corresponding to the execution instruction based on the execution instruction of the process transmitted from the server.

According to the present disclosure, when detecting at least one object in each captured image captured by a plurality of cameras installed in a monitoring area, processing such as learning of parameters used for the detection is distributed among the plurality of cameras. However, it is possible to suppress the increase in traffic on the network and help reduce the processing load of the server connected to multiple cameras.

A block diagram showing an example of the system configuration of the monitoring system of the first embodiment. Explanatory diagram of an outline example of learning and detection A block diagram showing in detail an example of the internal configuration of the camera according to the first embodiment. A block diagram showing in detail an example of the internal configuration of the server of the first embodiment. Explanatory diagram of an overview example of learning on a device Explanatory diagram of a schematic example of camera detection Explanatory diagram of an example of processing outline when performing distribution during learning using multiple cameras in a surveillance system Explanatory diagram of an outline example of resource management in a monitoring system A sequence diagram showing in detail an example of an operation procedure in which the server gives an instruction to execute a process to the camera in the first embodiment. A sequence diagram showing in detail an example of an operation procedure in which the server controls the feedback amount of the model parameters in the first embodiment. Explanatory diagram of an outline example of sharing learning results in a monitoring system A diagram showing an example of the UI screen displayed during local learning A diagram showing an example of the UI screen displayed on the display of the server during integrated learning. A block diagram showing in detail an example of the internal configuration of the processing execution unit of the camera of the second embodiment. A flowchart showing in detail an example of the operation procedure of local learning of the camera Explanatory diagram of an outline example of sharing learning results in a monitoring system A diagram showing an example of the UI screen displayed during local learning A diagram showing an example of the UI screen displayed on the display of the server during integrated learning.

(Background to the first embodiment)
In the future, it is expected that the captured image data handled by the camera will have a high definition and a large capacity such as 4K or 8K, and the data size will increase. With the increase in the size of the captured image data, if the parameters used for detecting the captured image data are learned by the server instead of the camera, the processing load is concentrated on the server, and a large amount of data is stored. By transmitting to the server one by one, the traffic on the network increases, which causes a problem that a corresponding delay occurs during data communication. No particular consideration has been given to the technical measures for such problems in the prior art as in Patent Document 1.

Therefore, in the first embodiment, when detecting at least one object in each captured image captured by a plurality of cameras installed in the monitoring area, processing such as learning of parameters used for the detection is performed between the plurality of cameras. An example of a monitoring system and a monitoring method that distributes the data, suppresses an increase in traffic on the network, and helps reduce the processing load of a server connected to a plurality of cameras will be described.

(Embodiment 1)
FIG. 1 is a block diagram showing an example of the system configuration of the monitoring system 5 of the first embodiment.

The monitoring system 5 is, for example, a security monitoring system, and is installed indoors of banks, stores, companies, facilities, etc., or outdoors of parking lots, parks, etc. Indoors such as banks, stores, companies, facilities, etc., or outdoors such as parking lots and parks are monitoring areas of the monitoring system 5. The monitoring system 5 of the present embodiment uses artificial intelligence (AI) technology, and has at least one camera 10 and a server that recognize at least one object (in other words, an object) appearing in a captured image. It has a configuration including 30 and a recorder 50. The at least one camera 10, the server 30, and the recorder 50 are communicably connected to each other via the network NW.

Hereinafter, when it is necessary to distinguish between the plurality of cameras 10, they are referred to as cameras 10A, 10B, 10C, .... The plurality of cameras 10 may be installed as a monitoring area in the same place in a building, for example, or some cameras 10 may be installed in a place different from other cameras 10. Here, it is assumed that the installation conditions (for example, the installation angle and the angle of view of the camera) of the cameras 10A, 10B, and 10C installed in different places as the monitoring area are the same. For example, the cameras 10A, 10B, and 10C are all mounted on the wall surface so as to be located above the doorway where the automatic door is installed, and image a person entering and exiting the doorway so as to look down from slightly above. The installation status of the cameras 10A, 10B, and 10C is not limited to the case where the automatic door is located in the information of the installed entrance / exit.

First of all, learning to generate a neural network (in other words, a learning model) used for deep learning as an example of machine learning of artificial intelligence (AI) technology, and a learned learning model (hereinafter, "learned"). The outline of detection (that is, inference) that inputs data to (called a model) and outputs the result will be described.

FIG. 2 is an explanatory diagram of a schematic example of learning and detection.

The learning process (hereinafter, simply referred to as “learning”) is a process performed by deep learning as an example of machine learning of artificial intelligence (AI) technology, for example. In other words, as one of machine learning, learning is performed using deep learning (that is, deep learning) in a neural network (hereinafter, abbreviated as "NN"), which has been attracting attention in recent years. In machine learning by deep learning, "learning with teacher" using teacher data and "learning without teacher" using teacher data are performed. As a result of machine learning, a trained model is generated. On the other hand, detection is a process of inputting data into a generated trained model and obtaining a result.

Learning may be done in real time, but it is usually done offline (ie, asynchronous) because it requires a lot of arithmetic processing. On the other hand, the detection process (hereinafter, simply referred to as "detection") is usually performed in real time. Further, the device on which learning is performed may be any of, for example, a camera 10, a server 30, and a recorder 50, and here, the case where learning is performed by the camera 10 is shown. On the other hand, the detection is performed by the camera 10. Even if the captured image data captured by the camera 10 is transferred to the server 30 or the recorder 50, if the traffic on the network NW does not occur, the server 30 or the recorder 50 may perform the detection.

At the time of learning, the device 150 inputs a lot of learning data (for example, image data captured by the camera 10). The device 150 performs machine learning (for example, deep learning processing) based on the input learning data, and updates the model parameter P of the neural network (NN140) which is a learning model. The model parameter P is a weighting coefficient (that is, a bias), a threshold value, or the like set in each of the plurality of neurons constituting the NN 140. When the device 150 performs machine learning (for example, deep learning processing), the teacher data is used to acquire correctness for each learning data, or to calculate an evaluation value (that is, a score). The device 150 changes the learning degree of the model parameter P according to the correctness of the training data or the high or low score. After training, the NN 140 is used as a trained model for detection on the device 150.

At the time of detection (that is, at the time of inference), the device 150 inputs input data (for example, image data captured in real time by the camera 10), executes inference in the NN 140, and the inference result obtained by the execution (that is, that is). , Judgment result of the detected object) is output. The determination result includes, for example, information on a correct report or a false report according to the presence or absence of an object included in the captured image data, and information on a score indicating an evaluation value of the object. The correct report is a report showing that the object was detected correctly with high accuracy at the time of detection. A false alarm is a report showing that an object was erroneously detected with high accuracy when it was detected.

FIG. 3 is a block diagram showing in detail an example of the internal configuration of the camera 10 of the first embodiment.

For example, the camera 10 captures a subject image in a monitoring area and acquires captured image data. Specifically, the camera 10 includes a lens 11, an image sensor 12, a signal processing unit 13, a processing execution unit 14, a resource monitoring unit 15, a crop encoding unit 17, and a network I / F 16. Is.

The camera 10 forms an image of the incident subject image from the surveillance area SA on the image sensor 12 via the lens 11 arranged so that the subject image from the surveillance area SA can be incident, and the subject image (the subject image ( That is, the optical image) is converted into an electric signal and imaged. At least the lens 11 and the image sensor 12 constitute an image pickup unit of the camera 10. The camera 10 uses the electric signal obtained by the image sensor 12 to generate an RGB signal in the signal processing unit 13 and to perform various predetermined image processing such as white balance and contrast adjustment to obtain captured image data. Is generated and output.

The processing execution unit 14 is configured by using, for example, a GPU (Graphics Processing Unit) or an FPGA (Field Programmable Gate Array). In the future, when a GPU or FPGA with high performance and high arithmetic processing power is adopted as the processor of the camera 10, the arithmetic processing capacity of the camera 10 will be dramatically improved, and the deep learning processing can be sufficiently executed in the camera 10. Is expected to be. The processing execution unit 14 includes a learning model or a trained model as an NN140 generated or updated by processing execution in the GPU or FPGA, and has at least one object (that is, an object) appearing in the captured image with respect to the input captured image data. , Object) judgment result is output.

The resource monitoring unit 15 monitors information (for example, the amount of free resources) regarding the processing capacity of the camera 10 based on the usage status of the GPU, FPGA, memory, etc. in the processing execution unit 14.

The crop encoding unit 17 cuts out a part of an object (that is, an object) that appears in the captured image data at the time of detection, and outputs it as captured image data or thumbnail data to be processed.

The network I / F16 controls the connection with the network NW. The camera 10 determines the object (that is, an object) output from the processing execution unit 14 to the server 30 and the recorder 50 via the network I / F16, and the free resource monitored by the resource monitoring unit 15. Send the amount, thumbnail data, etc. Further, the camera 10 receives the model parameter P, which is the result of learning, from the server 30, the recorder 50, and another camera 10 via the network I / F16.

FIG. 4 is a block diagram showing in detail an example of the internal configuration of the server 30 of the first embodiment.

The server 30 includes a processor (for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit) or a DSP (Digital Signal Processor)) 31, a memory 32, a communication unit 33, an operation unit 36, and a display unit 37. It is configured to include a learning DB (database) 34 and a table memory 35. The processor 31 collectively executes processing and control of each part of the server 30 in cooperation with the memory 32. The memory 32 has a non-volatile memory and a volatile memory. In the non-volatile memory, for example, information regarding the unit price cost of each of the cameras 10A, 10B, 10C notified from the plurality of cameras 10A, 10B, 10C (for example, information regarding the power cost of the cameras 10A, 10B, 10C) is stored. .. The information regarding the electric power cost, which will be described in detail later, is an index value indicating, for example, how much electric power (that is, cost) is required as a result of how much the cameras 10A, 10B, and 10C are used.

When the server 30 performs machine learning (for example, deep learning processing), the processor 31 executes a program stored in the non-volatile memory and generates a learning model (neural network: NN). Further, the server 30 receives the model parameter P which is the result of learning from the plurality of cameras 10, and the installation status of each camera 10 installed in the monitoring area SA (that is, the installation environment such as the installation angle and the angle of view). Integrate the model parameters P that are the same.

In the learning DB (database) 34, a model parameter P (for example, a weighting coefficient or a threshold value) which is a result of learning transmitted from a plurality of cameras 10 and received by the server 30 is stored.

The table memory 35 stores a table in which information regarding the processing capacity of the plurality of cameras 10 (for example, the amount of free resources) is registered.

The operation unit 36 has various buttons such as a learning button bt5 (see, for example, FIG. 13) that can be operated by the user, and accepts the input operation of the user.

The display unit 37 displays a UI (user interface) screen 310 (see, for example, FIG. 12 or FIG. 13) that presents the processing result of the integrated learning in the server 30.

FIG. 5 is an explanatory diagram of a schematic example of learning in the device 150.

Here, a case where the device 150 learns the “car” appearing in the captured image as the object obj will be illustrated and described. As described above, the learning is a process normally performed offline (asynchronously), and may be performed by any of the camera 10, the server 30, and the recorder 50. In this embodiment, the camera 10 learns as an example of the device 150. The device 150 includes a processing execution unit 164, a resource monitoring unit 165, a network I / F 166, and a parameter gradient calculation unit 168.

The network I / F166 controls the connection with the network NW and receives the learning data via the network NW. Here, the learning data is the captured image data gz1 and gz2 with the car as the object obj. The captured image data gz1 and gz2 are teacher data to which a score (evaluation value) and a positive report or a false report are added, respectively. For example, the captured image data gz1 is a captured image including a "car" as an object, and is teacher data having a high score or a positive report. On the other hand, the captured image data gz2 is an image of a "tree" that is not a target vehicle, and is teacher data having a low score or a false alarm.

The processing execution unit 164 updates the model parameter P (for example, weighting coefficient, threshold value, etc.) of the learning model by executing inference based on these teacher data input via the network I / F166. .. Further, the processing execution unit 164 transmits the updated model parameter P to other devices such as the camera 10, the server 30, and the recorder 50 via the network I / F166. By performing "supervised learning" in this way, the learning ability is enhanced, and the processing execution unit 164 can generate a high-quality learning model.

The parameter gradient calculation unit 168 calculates the gradient of the object appearing in the captured image of the teacher data. For example, the captured image captured by the camera from the side and the captured image captured by the camera from the front are different even if they are the same object. That is, the model parameter P of the learning model used when detecting the same object also differs depending on the installation state of the camera (for example, the installation angle and the angle of view). Therefore, the parameter gradient calculation unit 168 calculates a gradient representing the imaging direction (hereinafter referred to as “parameter gradient”), and sets the parameter gradient Pt to the camera 10, the server 30, the recorder 50, etc. via the network I / F166. Send to other devices. The parameter gradient Pt may be transmitted with or separately from the model parameters. In any case, the installation status of the camera does not change frequently, so the parameter gradient Pt may be transmitted at least once. By using the parameter gradient Pt, different learning models can be used depending on the installation status of the camera.

The resource monitoring unit 165 monitors the amount of free resources based on the usage status of the GPU, memory, etc. in the processing execution unit 164. When the device 150 is the camera 10, the processing execution unit 164 and the parameter gradient calculation unit 168 shown in FIG. 5 correspond to the processing execution unit 14 of FIG. 3, and the resource monitoring unit 165 shown in FIG. 5 corresponds to the processing execution unit 14. Corresponds to the resource monitoring unit 15 shown in FIG. 5, and the network I / F166 shown in FIG. 5 corresponds to the network I / F16 shown in FIG.

FIG. 6 is an explanatory diagram of a schematic example of detection of the camera 10.

Here, a case where the camera 10 detects a “car” appearing in the captured image as an object will be described as an example. The processing execution unit 14 of the camera 10 has a learning model (that is, a trained model) after machine learning (for example, deep learning processing) has been performed. The processing execution unit 14 inputs the captured image og of the subject captured through the lens 11, detects it using the trained model (that is, infers the object appearing in the captured image og), and the detection result (that is, that is). Inference result) is output. The crop encoding unit 17 cuts out an image to be an object included in the captured image og of the subject, and outputs the cut-out image as a result of detection.

Here, the cut-out image tg2 of the "car" and the cut-out image tg1 of the "tree" cut out by the crop encoding unit 17 are output. Since the cut-out image tg2 of the "car" includes a captured image of the car as an object, it has a high score and a positive report. On the other hand, the cut-out image tg1 of the "tree" does not include the captured image of the target vehicle, and therefore has a low score and a false alarm.

Next, the specific operation of the monitoring system 5 of the present embodiment will be described with reference to the drawings.

FIG. 7 is an explanatory diagram of an example of processing outline when performing distribution during learning using a plurality of cameras in the monitoring system 5.

As described above, in learning, a process of updating the model parameter P of a learning model (that is, a neural network) generated by using deep learning processing as an example of machine learning of artificial intelligence (AI) technology is performed. .. As an example, a case where three cameras 10A, 10B, and 10C perform learning as a learning device is shown. The device for learning is not limited to the camera, but may be a server or a recorder. Each camera 10A, 10B, 10C performs, for example, "unsupervised learning" on the input image data. In unsupervised learning, the camera 10 raises an alarm if the model parameters of the learning model do not converge. At this time, the user cancels the alarm and performs "supervised learning". In supervised learning, the user inputs a positive or false report of image data. In the input of the teacher data, instead of inputting the positive report or the false report of the image data, the score (evaluation value) may be input together with the positive report or the false report. The score is a value for evaluating that the captured image data is captured image data including an object, and is expressed by, for example, a score of 80 points, 10 points, or a probability of 50%, 20%, or the like.

The three cameras 10A, 10B, and 10C each transmit the model parameter P, which is the result of learning, to the server 30. Further, the parameter gradient Pt described above is added to the transmitted model parameter P.

The server 30 updates the model parameter P of the learning model based on the model parameter P transmitted from the three cameras 10A, 10B, and 10C. At this time, the model parameters having the same parameter gradient Pt, that is, the model parameters having the same camera installation status are integrated. Therefore, the model parameters of the learning model with the same parameter gradient are updated. Here, the installation status of the cameras 10A, 10B, and 10C is the same, and the server 30 integrates the updated model parameters of the cameras 10A, 10B, and 10C.

The server 30 feeds back the integrated model parameters to the three cameras 10A, 10B, and 10C. As a result, the model parameters stored in the three cameras 10A, 10B, and 10C are the same. The model parameters are transmitted asynchronously from the three cameras 10A, 10B, and 10C to the server 30.

FIG. 8 is an explanatory diagram of an outline example of resource management in the monitoring system 5.

In the three cameras 10A, 10B, and 10C, the resource monitoring unit 15 determines the amount of free resources (in other words, the surplus capacity indicating the surplus degree of the processing capacity) with respect to the processing capacity of the GPU or FPGA that generates the learning model, respectively. I'm watching. The three cameras 10A, 10B, and 10C notify the server 30 of the amount of free resources monitored by the resource monitoring unit 15 asynchronously or periodically. The amount of free resources is expressed as a percentage of processing power. As an example, when the amount of free resources of the camera 10A is 90%, the amount of free resources of the camera 10B is 20%, and the amount of free resources of the camera 10C is 10%, the server 30 is the free resource. A learning instruction is output to the camera 10A so as to preferentially train the camera 10A having a large amount, that is, to increase the learning amount.

Further, when the network NW has a wide band or the network NW is free, the server 30 has captured image data (with correct or false report information) captured by the camera 10C, which has a small amount of free resources of 10%. ) Is received, the captured image data may be transmitted to the camera 10A having a large amount of free resources of 90% to instruct learning. As a result, proper learning can be realized without imposing a biased processing load between the cameras.

Further, the server 30 may instruct the camera to directly transfer the captured image data captured by the camera having a small amount of free resources to the camera having a large amount of free resources for learning. As a result, learning can be distributed within the monitoring system, and efficient learning is possible without imposing a heavy load on a specific camera.

Further, the server 30 may instruct the camera to directly transfer the captured image data captured by the camera having a small amount of free resources to the camera having a large amount of free resources for detection. As a result, the detection can be distributed in the surveillance system, and efficient detection is possible without imposing a heavy load on a specific camera.

Further, the server 30 may instruct the camera to directly transfer the captured image data captured by the camera having a small amount of free resources to the camera having a large amount of free resources for analysis. Here, the analysis is a process of tracking an object (that is, an object) obj appearing in a captured image, or recognizing whether or not the object corresponds to a suspicious person, and the content of the analysis. Is not particularly limited in this embodiment. As a result, the analysis can be distributed within the surveillance system, and efficient analysis can be performed without imposing a heavy load on a specific camera.

Further, the server 30 monitors the overall processing capacity of the monitoring system 5, and when the amount of free resources in the entire system is large, the server 30 instructs each camera 10 to increase the learning amount, while the free space in the entire system. If the amount of resources is small, each camera 10 may be instructed to reduce the amount of learning. This enables proper learning without imposing a heavy load on the entire monitoring system.

Further, the server 30 may instruct all cameras 10 to share the result of detection by each camera 10. As a result, the detection results can be distributed to each camera 10, and the detection accuracy can be improved by using the detection results for the next and subsequent detections.

Further, when the amount of free resources of the camera 10 is large, the server 30 instructs to increase the feedback amount (for example, the number of feedbacks) of the result of the integrated learning transmitted to the camera 10, while the free resources of the camera 10 are used. If the amount is small, it may be instructed to reduce the feedback amount (for example, the number of feedbacks) of the result of the integrated learning transmitted to the camera 10. This makes it possible to feed back (return) an appropriate amount of learning results to the camera without imposing a heavy load on the camera.

Further, each of the three cameras 10A, 10B, and 10C notifies the server 30 of information regarding the unit price cost (for example, information regarding the power cost). The power cost is a value unique to the camera and is expressed in units of watts / frames (W / frame), for example. As an example, the camera 10A has 1/200, the camera 10B has 1/200, and the camera 10C has 1/400. Since the power cost does not usually change significantly depending on the usage status of the camera, one notification is sufficient. Further, the unit of power cost may be expressed in frames / watts (frame / watt).

When the power costs of the camera 10A and the camera 10B are similarly high, the server 30 preferentially allocates learning to the camera 10C having a low power cost.

In the server 30, when the amount of free resources of the cameras 10A, 10B, and 10C is the same or the same, for example, the amount of free resources of the camera 10A is 10%, and the amount of free resources of the cameras 10B, 10C is all. When it is 45%, a learning instruction is output to the camera 10C so that the camera 10C, which does not require power cost, learns preferentially.

Regardless of the amount of free resources of the camera, the server 30 may instruct the camera to perform learning with a camera that gives priority to cost and has a low power cost. Further, although the server 30 manages the free resources and the power cost of each device, each camera or the recorder may manage the free resources and the power cost. In that case, the free resources and the power cost are managed by all the devices 150 in the monitoring system 5. Can be shared with. Therefore, in consideration of free resources and power cost, each device can execute an instruction for processing, and various operations are possible.

FIG. 9 is a sequence diagram showing in detail an example of an operation procedure in which the server 30 gives an instruction to execute a process to the camera 10 in the first embodiment.

In the operation procedure of FIG. 9, the server 30 determines the camera 10 to be distributed processing from among the plurality of cameras 10 based on the information of the free resources of the camera 10, and executes the processing on the corresponding camera 10. Instruct. The number N of cameras may be any number, and here, two cameras (cameras 10A and 10B) are exemplified for the sake of simplicity. Instead of the server 30, the recorder 50 may determine the camera 10 instructing the processing execution.

The camera 10A repeatedly (for example, constantly or periodically) notifies the server 30 of the information of the free resource monitored by the resource monitoring unit 15 (T1). Similarly, the camera 10B repeatedly (for example, constantly or periodically) notifies the server 30 of the information of the free resource monitored by the resource monitoring unit 15 (T2).

The server 30 registers and manages information on free resources of the cameras 10A and 10B in the table memory 35 (T3). The server 30 determines the presence or absence of at least one camera having a predetermined value (for example, 70%) or more of free resources (T4). Here, it is assumed that only the camera 10B has a free resource of a predetermined value or more.

When there is a camera having a free resource of a predetermined value or more (T4, YES), the server 30 generates an execution instruction for both detection and learning for the corresponding camera (T5). The server 30 transmits an execution instruction for both detection and learning to the camera 10B via the network NW (T6). The camera 10B executes the corresponding process (T7).

On the other hand, when there is no camera having a free resource of a predetermined value or more in the procedure T4 (T4, NO), the server 30 generates a detection execution instruction for all the cameras (here, the cameras 10A and 10B) (T8). .. Since the cameras 10A and 10B do not have enough free resources to execute learning, only detection is performed. The server 30 transmits a detection execution instruction to all cameras (here, cameras 10A and 10B) (T9). The cameras 10A and 10B execute the corresponding processes, respectively (T10, T11).

When the camera 10B executes the corresponding process in the procedure T7, the camera 10B generates a learning result (T12) and transmits the generated learning result to the server 30 (T13).

FIG. 10 is a sequence diagram showing in detail an example of an operation procedure in which the server 30 controls the feedback amount of the model parameters in the first embodiment.

In the operation procedure of FIG. 10, the server 30 controls the feedback amount of the model parameter based on the information of the free resource of the camera 10. The number N of the cameras may be any number, and here, two cameras (cameras 10A and 10B) are used for the sake of simplicity. Instead of the server 30, the recorder 50 may determine the camera 10 to feed back the model parameters.

The server 30 receives model parameters, which are learning results, from the cameras 10A and 10B, respectively, and stores them in the learning DB 34 (T21). The server 30 calculates the feedback amount of the model parameter of the learning model used at the time of inference (detection) processing from many model parameters which are the learning results according to the amount of free resources of each camera 10A and 10B for each camera. (T22).

The server 30 transmits the calculated feedback amount of model parameter data to the camera 10B (T23). Similarly, the server 30 transmits the calculated feedback amount of model parameter data to the camera 10A (T24). The camera 10B additionally registers and stores the model parameters received from the server 30 in the memory of the processing execution unit 14 (T25). Similarly, the camera 10A additionally registers and stores the model parameters received from the server 30 in the memory of the processing execution unit 14 (T26).

Here, the amount of feedback is determined by the server 30 based on the information of the free resources of each camera, but it is not limited to the free resources, but the number of positive reports detected based on the teacher data and the number of false alarms detected based on the teacher data. May be determined accordingly.

FIG. 11 is an explanatory diagram of an outline example of sharing learning results in the monitoring system 5.

Each camera 10 (10A, 10B, 10C) performs local learning using the captured image data obtained by imaging, and updates the model parameters. In addition, each camera 10 can perform learning using only the captured image data for which the correct report has been obtained, and can improve the accuracy of the model parameters that are the result of the learning. Further, the camera 10 can display the UI screen 320 (see FIG. 12) for evaluating the captured image data in the local learning on the display 19 connected as an option. Further, the camera 10 can display the UI screen 320 at the time of local learning on the display unit 37 of the server 30.

FIG. 12 is a diagram showing a UI screen 320 displayed during local learning.

The UI screen 320 is displayed, for example, on the display unit 37 of the server 30 or the display unit of a PC (not shown) communicably connected to the camera 10 during local learning of the camera 10, and specifically, from the captured image data. For each of the cut out learning data, the correctness judgment, the camera ID, and the reject button bx are displayed. The thumbnail of the captured image data is not displayed here because the camera 10 stores the original captured image data, but it may be displayed. The object (object) obj to be detected is a "person".

In the correctness determination process, the server 30 determines that the target object obj can be detected in the captured image data, and determines that the target object obj cannot be detected in the captured image data. The server 30 may determine a correct report or a false report by inputting to the UI screen 320 displayed on the display unit 37 of the server 30 by the user.

The camera ID is the identification information of the camera imaged to obtain the learning data.

The reject button bx is selected by the user and a check mark is displayed. The learning data with a check mark added to the reject button bx is not used for learning when the user presses the learning button bt5.

The camera 10 was not automatically used for learning to adopt the captured image data of the false report, but was used for learning to adopt the captured image data of the positive report. However, instead of the camera 10, the user uses the reject button bx. The captured image data may be instructed. For example, the user may instruct to use it for learning to adopt the captured image data of the positive report instead of using it for learning to adopt the captured image data of the false alarm. As a result, learning can be performed using the captured image data of the false alarm. Further, the camera 10 may be used for learning by combining the captured image data of the positive report and the captured image data of the false report. Thereby, the captured image data used for learning can be selected in light of the quality of the captured image data.

The server 30 receives the model parameter P transmitted from each camera 10 (10A, 10B, 10C), performs integrated learning to sum the received model parameters P, and adds the summed model parameter P to the learning DB 34. do. Here, the model parameters to be integrated are model parameters obtained based on the image data captured by the cameras having the same installation status. On the other hand, the model parameters obtained based on the image data captured by the cameras having different installation conditions are not added up and are individually registered as model parameters for different learning models.

FIG. 13 is a diagram showing a UI screen 310 displayed on the display unit 37 of the server 30 during integrated learning.

The server 30 can display the UI screen 310 (see FIG. 13) at the time of integrated learning on the display unit 37. The UI screen 310 displays a correct / incorrect determination, a thumbnail, a camera ID, and a reject button bx for each learning data cut out from the captured image data. Here, the case where the object (object) to be detected is a “person” is shown.

In the correctness determination process, the server 30 determines that the object OBj is detected in the captured image data as a correct report, and determines that the target object obj is not detected in the captured image data as a false alarm. The server 30 may determine a correct report or a false report by inputting to the UI screen 310 displayed on the display unit 37 of the server 30 by the user.

Thumbnails are reduced images of learning data. Since it is a thumbnail, the amount of data transfer can be suppressed when it is transmitted from the camera 10 to the server 30. The camera ID is the identification information of the camera imaged to obtain the learning data.

The reject button bx is selected by the user and a check mark is displayed. The learning data with a check mark added to the reject button bx is not used for learning when the user presses the learning button bt5.

The server 30 automatically learns to adopt the captured image data of the positive report (that is, learns to accumulate the model parameters used for detecting the captured image data of the positive report), and the captured image data of the false report. (That is, learn not to accumulate the model parameters used to detect falsely captured image data). However, regarding the selection of the captured image data to be used for learning, the user may independently use the reject button bx to instruct the captured image data to be used for learning. Further, the user may instruct to perform learning to adopt the captured image data of the positive report without performing learning to exclude the captured image data of the false alarm.

In this way, the server 30 performs integrated learning of the model parameters, so that the accuracy of learning the model parameters is improved. The server 30 feeds back the updated model parameters that are the result of the integrated learning to the corresponding camera 10. As a result, the more positive reports of the captured image data obtained by each camera 10, the higher the detection accuracy of the cameras.

Further, when the updated model parameter P, which is the result of the integrated learning, is fed back to each camera 10, the server 30 controls the amount of feedback according to the number of positive reports of each camera 10. That is, the server 30 transmits updated model parameters to the camera 10 having a large number of false alarms so that the amount of feedback (for example, the number of feedbacks) is large. This increases the number of positive reports and improves the detection accuracy of the camera.

On the other hand, the server 30 transmits updated model parameters to the camera 10 having a large number of positive reports so that the amount of feedback (for example, the number of feedbacks) is small. As a result, the processing load of the camera can be reduced. As described above, the server 30 transmits and shares the same updated model parameters to cameras having the same installation environment.

Further, when the server 30 gives an instruction to execute learning to each camera 10, the server 30 instructs each camera 10 to instruct the amount of learning according to the number of positive reports of each camera 10. A learning execution instruction is given to the camera 10 having a large number of false alarms so that the learning amount is large. This increases the number of positive reports and improves the detection accuracy of the camera. On the other hand, the server 30 gives an instruction to execute learning to the camera 10 having a large number of positive reports so that the amount of learning is small. As a result, the processing load of the camera can be reduced.

Further, the server 30 may integrate and manage the detection result of detecting the object appearing in the captured image captured by each camera 10. When integrating the detection results, the movement of the object may be represented by a vector, and the detection results may be managed by the vector.

As described above, in the monitoring system 5 of the first embodiment, the server 30 and the plurality of cameras 10 installed in the monitoring area SA are connected to each other so as to be communicable with each other. The server 30 has a table memory 35 that holds free resources (that is, information regarding processing capacity) of each camera 10 and data of an captured image obtained by imaging the monitoring area SA by each camera 10. The server 30 performs a process performed by the camera 10 for each camera 10 regarding the detection of at least one object (object) obj appearing in the captured image obtained by each camera 10 based on the information regarding the processing capacity of the camera 10. It is determined, and the execution instruction of the determined process is transmitted for each camera 10. Each camera 10 executes a process corresponding to the execution instruction based on the process execution instruction transmitted from the server 30.

As a result, when the monitoring system 5 detects at least one object in each captured image captured by the plurality of cameras 10 installed in the monitoring area SA, the monitoring system 5 performs a plurality of processes such as learning of parameters used for the detection. It can be distributed among the cameras 10, suppresses the increase in traffic on the network, and can support the reduction of the processing load of the server 30 connected to the plurality of cameras 10.

Further, the above-mentioned processing is learning to learn the model parameter P used for detecting at least one object (object) obj appearing in the captured image. As a result, the monitoring system 5 can distribute the learning with a heavy load to a plurality of cameras 10.

Further, the server 30 transmits a learning execution instruction to each of the plurality of cameras 10. Each of the plurality of cameras 10 executes learning according to a learning execution instruction. The server 30 receives the result of learning executed by the plurality of cameras 10. As a result, the server 30 can obtain the learning result from the plurality of cameras 10 without learning by the own device, for example.

Further, the server 30 executes the learning by itself and also transmits the learning execution instruction to the plurality of cameras 10. Each of the plurality of cameras 10 executes learning according to the learning execution instruction. The server 30 receives the result of learning executed by the plurality of cameras 10. As a result, the server 30 can add the learning result of its own device to the learning result obtained from the plurality of cameras 10, and can improve the efficiency of learning from the next time onward.

Further, the server 30 transmits the learning result to the plurality of cameras. The plurality of cameras 10 share the learning result. This allows multiple cameras to take advantage of the same learning results.

Further, some of the plurality of cameras 10 among the plurality of cameras 10 are installed in the same installation situation. The server 30 transmits the learning result to each of a plurality of cameras 10 having the same installation status. Some of the plurality of cameras 10 having the same installation status share the learning result transmitted from the server 30. As a result, the surveillance system 5 can improve the detection accuracy of the object by the plurality of cameras 10 having the same installation condition.

Further, the server 30 controls the learning processing amount according to the number of detected objects (objects) obj detected by the camera 10. As a result, the server 30 can reduce the amount of learning that increases the load on the camera, which has a large number of detected objects and a heavy processing load. On the other hand, the server 30 can increase the amount of learning for a camera having a large number of detected objects and a small processing load. Therefore, the processing load of the camera can be made uniform.

Further, the server 30 controls the amount of learning processing in the camera 10 according to the number of positive reports of the detection of the object (object) obj detected by the camera 10. As a result, the server 30 can improve the accuracy of the learning result (in other words, the accuracy of the detection from the next time onward) by using a lot of the learning results of the positive report.

Further, the server 30 controls the amount of learning in the camera 10 according to the number of false alarms of the detection of the object (object) obj detected by the camera 10. As a result, the server 30 can improve the accuracy of the learning result (in other words, the accuracy of the detection from the next time onward) by not using the learning result of the false alarm.

Further, the server 30 controls the amount of learning processing in the camera 10 according to the amount of processing capacity of the camera 10. As a result, the server 30 can distribute learning to a plurality of cameras without imposing a heavy load on a specific camera, and efficient learning can be realized.

Further, the server 30 holds information on the processing capabilities of the server 30 constituting the monitoring system 5 and the plurality of cameras 10 in the table memory 35. The server 30 controls the amount of learning processing according to the amount of processing capacity of each of the server 30 and the plurality of cameras 10. As a result, the server 30 can distribute learning to a plurality of devices without imposing a large load on a specific device of the monitoring system, and efficient learning can be realized.

Further, in the above processing, learning to learn the model parameter P used for detecting at least one object (object) obj appearing in the captured image og, and learning to learn at least one object (object) obj appearing in the captured image og. Includes detection to detect and analysis to analyze at least one object obj detected by detection. As a result, the server 30 can distribute the processing to the plurality of cameras 10 not only in learning but also in detection and analysis.

Further, the server 30 transmits data of the captured image held in the table memory 35 to at least one camera 10 having a relatively high processing capacity as compared with the other cameras among the plurality of cameras 10 for learning. Give an execution instruction. As a result, the server 30 can transmit the captured image data to another camera having high processing power when the network band is wide or the network is free, and as a result, the learning speed is increased. Can be improved.

Further, the server 30 transmits and detects the data of the captured image held in the table memory 35 to at least one camera 10 having a relatively high processing capacity as compared with the other cameras among the plurality of cameras 10. Give an execution instruction. As a result, the server 30 can transmit the captured image data to another camera having high processing power when the network band is wide or the network is free, and as a result, the detection speed is increased. Can be improved.

Further, the server 30 transmits data of the captured image held in the table memory 35 to at least one camera 10 having a relatively high processing capacity as compared with the other cameras among the plurality of cameras 10 and analyzes the data. Give an execution instruction. As a result, the server 30 can transmit the captured image data to another camera having high processing power when the network band is wide or the network is free, and as a result, the speed of analysis can be increased. Can be improved.

Further, the above processing is detection for detecting at least one object (object) obj that appears in the captured image og. The server 30 transmits the detection result to each of the plurality of cameras 10. The plurality of cameras 10 share the detection result. As a result, the server 30 can distribute the detection to a plurality of cameras without imposing a heavy load on a specific camera, and can improve the efficiency of the detection.

Further, the server 30 integrates the learning results executed by the plurality of cameras 10. Thereby, the monitoring system 5 can improve the accuracy of the learning result aggregated by the integration in the server 30.

Further, some of the plurality of cameras 10 among the plurality of cameras 10 are installed in the same installation situation. The server 30 integrates the results of learning performed by a plurality of cameras 10 having the same installation status of the cameras 10. As a result, the server 30 can improve the detection accuracy of the object by the camera in the same installation condition.

The server 30 receives notification of information on the installation status of each camera from a plurality of cameras 10 having the same installation status, and integrates the learning results executed by the plurality of cameras 10 of the plurality of cameras. .. This makes it easier for the server 30 to integrate the learning results from the cameras in the same installation situation.

Further, the server 30 and the plurality of cameras 10 share information regarding the processing capacity of the plurality of cameras 10 and information regarding the unit price cost of the plurality of cameras 10 (for example, information on the power cost of the individual cameras 10). As a result, the server 30 can be instructed to execute processing by each device such as the server and a plurality of cameras in consideration of free resources and power cost, and various operations are possible.

(Background to the second embodiment)
In the prior art such as Patent Document 1 described above, it is disclosed that the score of the evaluation function for the object is used in order to obtain the object motion information which is the correct answer of the object to be tracked in the captured image. However, no particular consideration has been given to controlling the learning amount of the parameters required for detecting the object according to the score indicating the accuracy of detecting the object. For this reason, for example, learning the parameters used for detection, which originally does not require learning, causes variations in the learning accuracy of the parameters, and there is a concern that the detection accuracy of the object may be affected.

Therefore, in the second embodiment, the parameters used for the detection are obtained according to the score indicating the detection accuracy of the objects obtained in the detection of at least one object in the captured image captured by the camera installed in the monitoring area. An example of a monitoring system and a monitoring method for appropriately controlling the learning amount of the camera and improving the learning accuracy in the camera, and a camera and a parameter registration method will be described.

(Embodiment 2)
Since the system configuration of the monitoring system 5 of the second embodiment is the same as the system configuration of the monitoring system 5 of the first embodiment described above, the description thereof is simplified or omitted by using the same reference numerals, and the description is different. The contents will be explained.

FIG. 14 is a block diagram showing in detail an example of the internal configuration of the processing execution unit 14 of the camera 10 of the second embodiment.

The processing execution unit 14, which is the main configuration of the camera 10, includes a neural network (that is, NN140), a teacher data set memory 151, and a parameter memory 152.

The NN 140 has an object inference function 141, a score derivation function 142, a correctness determination function 143, and a parameter learning function 144.

In the object inference function 141 as an example of the detection unit, the NN 140 infers (that is, detects) what the object appearing in the captured image is according to the model parameters.

In the score derivation function 142 as an example of the derivation unit, the NN 140 derives a score (evaluation value) indicating the detection accuracy of the object at the time of inference using the teacher data registered in the teacher data set memory 151, and the score is derived. Is output.

In the correctness determination function 143, the NN 140 derives the correctness determination of the object at the time of inference using the teacher data registered in the teacher data set memory 151, and outputs the determination result.

In the parameter learning function 144 as an example of the parameter learning unit, the NN 140 learns to adopt the model parameters used for inferring an object having a high score. Further, in the parameter learning function 144, the NN 140 learns to exclude the model parameters used for the inference of the object having a low score. The NN 140 registers and stores the learned model parameters in the parameter memory 152. The model parameters used for inferring an object having a score higher than the first predetermined value (for example, 80 points) registered in the parameter memory 152 are transmitted to the server 30 in the sharing of the learning result described later and used in the integrated learning. Will be done.

FIG. 15 is a flowchart showing in detail an example of the operation procedure of the local learning of the camera 10.

In FIG. 15, the camera 10 captures an object from a subject image by the image sensor 12 (S1) and generates captured image data (S2).

The processing execution unit 14 inputs captured image data and infers (detects) at least one object (that is, an object) appearing in the captured image (S3). The processing execution unit 14 performs scoring processing of at least one object at the time of inference (detection) (S4). In this scoring process, the process execution unit 14 outputs the score (evaluation value) of the object using the teacher data registered in the teacher data set memory 151.

As a result of the scoring process, the processing execution unit 14 infers the model parameters used for inferring the object having a higher score than the first predetermined value (for example, 80 points) and the object having the lower score in the second predetermined value (for example, 10 points). The model parameters of the NN140 are learned using the model parameters used in (S5). Further, in step S5, the processing execution unit 14 registers and accumulates the model parameters having a higher score than the first predetermined value in the parameter memory 152. After this, the camera 10 ends the process shown in FIG.

The top score is, for example, 80 to 100 points. The lower score is, for example, 0 to 10 points. The processing execution unit 14 adopts, for example, a model parameter having a high score and excludes a model parameter having a low score.

An object with a high score is an object with a high possibility of a positive report, and an object with a low score is an object with a high possibility of a false alarm. Therefore, by learning to adopt the model parameters used for inference (detection) of the object with the highest score, the model parameters applied to the estimation of the object with high possibility of correct information will be used, and the model will be used. The learning accuracy of parameters can be improved.

In addition, by learning to eliminate the model parameters used to infer low-score objects, the model parameters applied to the estimation of objects with a high possibility of false alarms are no longer used, and the learning accuracy of model parameters is improved. Can be made to.

Further, the processing execution unit 14 may learn by combining the model parameters used for inferring the object with the higher score and the model parameters used for inferring the object with the lower score. In this way, by learning by combining the model parameters used for inferring the high-scoring object and the model parameters used for inferring the low-scoring object, the learning accuracy of the model parameters can be further improved. ..

FIG. 16 is an explanatory diagram of an outline example of sharing learning results in the monitoring system 5.

Each camera 10 (10A, 10B, 10C) performs local learning using the captured image data obtained by imaging. In the local learning, each camera 10 executes a scoring process regarding the detection accuracy of the object detected in the captured image data, and learns the model parameters of the NN 140 according to the obtained score. Further, each camera 10 learns by adopting only the model parameters used for inference (detection) of the object with the higher score. As a result, the learning accuracy of the model parameters can be improved.

Further, in local learning, a UI screen 340 (see FIG. 17) for evaluating captured image data can be displayed. For example, the camera 10 may display the UI screen 340 on a display (not shown) connected to the camera 10 as an option, or may transfer the UI screen 340 to the display unit 37 of the server 30 to display the UI screen 340. It is possible.

FIG. 17 is a diagram showing an example of the UI screen 340 displayed during local learning.

The UI 340 displays a score, a camera ID, and a reject button bx for each learning data cut out from the captured image data. The thumbnail of the image data is not displayed here because the camera 10 stores the original captured image data, but it may be displayed. The target (object) of detection is a "person".

The score is quantified in the range of 0 to 100 points. The score was calculated by each camera 10 performing scoring processing, but it may be acquired by inputting from the UI 320 by the user. The camera ID is the identification information of the camera imaged to obtain the learning data. When the reject button bx is selected by the user, a check mark is displayed. The learning data with a check mark added to the reject button bx is not used for learning when the user presses the learning button bt5.

The camera 10 did not automatically learn to adopt the captured image data of the lower score, but learned to adopt the captured image data of the upper score. However, for example, instead of the camera 10, the user may use the reject button bx to instruct the captured image data to be used for learning. For example, the user may instruct to perform learning to adopt the captured image data of the upper score without performing learning to exclude the captured image data of the lower score.

Further, although each camera 10 has learned to adopt only the captured image data of the upper score, it may be instructed to perform learning to exclude only the captured image data of the lower score, for example. As a result, learning can be performed using the captured image data from which the captured image data of the lower score is excluded. Further, it may be set to use the captured image data of the upper score and the captured image data of the lower score in combination. Thereby, the camera or the user can individually select the image data used for learning in light of the quality of the captured image data.

The server 30 receives the model parameters transmitted from each camera 10 (10A, 10B, 10C), performs integrated learning to sum the received model parameters, and adds the summed model parameters to the learning DB 34. Here, the model parameters to be integrated are model parameters obtained based on the image data captured by the cameras having the same installation status. On the other hand, the model parameters obtained based on the image data captured by the cameras having different installation conditions are not added up and are individually registered as model parameters for different learning models.

FIG. 18 is a diagram showing an example of a UI screen 350 displayed on the display unit 37 of the server 30 during integrated learning.

The server 30 can display the UI 310 (see FIG. 18) at the time of integrated learning on the display unit 37. The UI 350 displays a score, a thumbnail, a camera ID, and a reject button bx for each learning data cut out from the captured image data. Here, the case where the detection target (object) is a “person” is shown.

The score is quantified in the range of 0 to 100 points. For example, when the target is a "person", the score of the image data showing the person is as high as 80 to 100 points. On the other hand, the score of the image data showing a "tree" instead of a person is as low as 10 points. Thumbnails are reduced images of learning data. Since it is a thumbnail, the amount of data transfer can be suppressed when it is transmitted from the camera 10 to the server 30. The camera ID is the identification information of the camera imaged to obtain the learning data. The reject button bx is selected by the user and a check mark is displayed. The learning data with a check mark added to the reject button bx is not used for learning when the user presses the learning button bt5.

The server 30 does not automatically learn to adopt the captured image data of the upper score, but performs learning to exclude the captured image data of the lower score. However, for example, the user may instruct the captured image data to be used for learning by using the reject button bx. For example, the user may instruct to perform learning to adopt the captured image data of the upper score without performing learning to exclude the captured image data of the lower score.

In this way, the server 30 performs integrated learning of the model parameters, so that the accuracy of learning the model parameters is improved. The server 30 feeds back the updated model parameters that are the result of the integrated learning to the corresponding camera 10. As a result, the more positive reports of the image data obtained by each camera 10, the higher the detection accuracy of the cameras.

Further, when the updated model parameter, which is the result of the integrated learning, is fed back to each camera 10, the server 30 controls the amount of feedback according to the number of positive reports of each camera 10. That is, the server 30 transmits updated model parameters to the camera 10 having a large number of false alarms so that the amount of feedback (for example, the number of feedbacks) is large. This increases the number of positive reports and improves the detection accuracy of the camera.

On the other hand, the server 30 transmits updated model parameters to the camera 10 having a large number of positive reports so that the amount of feedback (for example, the number of feedbacks) is small. This can reduce the processing load of the camera. As described above, the server 30 transmits and shares the same updated model parameters to cameras having the same installation environment.

Further, when the server 30 gives an instruction to execute learning to each camera 10, the server 30 instructs each camera 10 to process the learning according to the number of positive reports of each camera 10. A learning execution instruction is given to the camera 10 having a large number of false alarms so that the learning amount is large. This increases the number of positive reports and improves the detection accuracy of the camera. On the other hand, the server 30 gives an instruction to execute learning to the camera 10 having a large number of positive reports so that the amount of learning is small. This can reduce the processing load of the camera.

Further, the server 30 may integrate and manage the detection result of detecting the target appearing in the image captured by each camera 10. When integrating the detection results, the movement of the target may be represented by a vector, and the detection results may be managed by the vector.

As described above, the camera 10 of the second embodiment is a camera used in the monitoring system 5 installed in the monitoring area SA and connected to the server 30 so as to be able to communicate with each other. The camera 10 captures the subject light from the monitoring area SA in the image sensor 12. The camera 10 detects at least one object appearing in the captured image by using the captured image og based on the imaging of the subject light in the object inference function 141 as an example of the detection unit. The camera 10 holds a teacher data set prepared for each type of object in the teacher data set memory 151. The camera 10 derives a score indicating the detection accuracy of the detected object by using the teacher data set in the score derivation function 142 as an example of the derivation unit. The camera 10 learns the model parameters used for detecting the object according to the derived score in the parameter learning function 144 as an example of the parameter learning unit. The camera 10 registers and stores the learning result of the model parameter in the parameter memory 152 in the parameter learning function 144.

As a result, the camera 10 is a parameter used for detection according to a score indicating the detection accuracy of the object obtained by detecting at least one object in the captured image captured by the camera installed in the surveillance area. The amount of learning can be appropriately controlled and the learning accuracy in the camera can be improved.

Further, the parameter learning function 144 learns to adopt the model parameter from which the score higher than the first predetermined value is derived. In this way, the camera 10 learns to adopt the model parameters used for inferring the high-scoring object so that the model parameters applied to the estimation of the object with high possibility of positive information are used. Therefore, the learning accuracy of model parameters can be improved.

Further, the parameter learning function 144 learns so as to exclude the model parameter from which the score lower than the second predetermined value is derived. In this way, the camera 10 learns to exclude the model parameters used for inferring the lower score object, so that the model parameters applied to the estimation of the object having a high possibility of false alarm are not used, and the model is used. The learning accuracy of parameters can be improved.

Further, the parameter learning function 144 learns to adopt the model parameter from which the score higher than the first predetermined value is derived, and excludes the model parameter from which the score lower than the second predetermined value is derived. To learn. In this way, the camera 10 further improves the learning accuracy of the model parameters by learning by combining the model parameters used for inferring the high-scoring object and the model parameters used for inferring the low-scoring object. Can be made to.

Although various embodiments have been described above with reference to the drawings, it goes without saying that the present invention is not limited to such examples. It is clear that a person skilled in the art can come up with various modifications or modifications within the scope of the claims, which naturally belong to the technical scope of the present invention. Understood. Further, each component in the above-described embodiment may be arbitrarily combined as long as the gist of the invention is not deviated.

For example, in the above-described embodiment, the monitoring system is applied to a security monitoring system that detects and tracks a suspicious person such as a thief, but in an unmanned automated (FA) production line. It may be applied to a monitoring system for product inspection or the like.

The present disclosure is useful as a monitoring system capable of suppressing an increase in traffic on a network and assisting in reducing the processing load of a server connected to a plurality of cameras.

5 Monitoring system 10, 10A, 10B, 10C Camera 11 Lens 12 Image sensor 13 Signal processing unit 14 Processing execution unit 15 Resource monitoring unit 16 Network I / F
17 Crop encoding unit 30 Server 31 Processor 32 Memory 33 Communication unit 34 Learning DB
35 Table memory (memory)
36 Operation unit 37 Display unit 50 Recorder 150 Device NW network P Model parameters

Claims (21)

A monitoring system in which a server and multiple cameras installed in the monitoring area are connected so that they can communicate with each other.
The server
It has a memory that holds information about the amount of free resources for the processing capacity to generate the learning model of each of the cameras and the captured image obtained by imaging the surveillance area with each of the cameras.
Based on the information about the amount of free resources of the camera, the process performed by the camera with respect to the detection of at least one object appearing in the captured image obtained by each camera is determined and determined for each camera. The execution instruction of the process is transmitted for each camera, and the process is executed.
Each of the cameras executes a process corresponding to the execution instruction based on the execution instruction of the process transmitted from the server.
Monitoring system. The process is learning to learn parameters used for detecting at least one object appearing in the captured image.
The monitoring system according to claim 1. The server transmits the learning execution instruction to the plurality of cameras, respectively.
The plurality of cameras execute the learning according to the learning execution instruction, respectively.
The server receives the result of the learning performed by the plurality of cameras.
The monitoring system according to claim 2. The server executes the learning, and also sends an execution instruction for the learning to the plurality of cameras.
The plurality of cameras execute the learning according to the learning execution instruction, respectively.
The server receives the result of the learning performed by the plurality of cameras.
The monitoring system according to claim 2. The server transmits the learning result to the plurality of cameras, respectively.
The plurality of cameras share the learning result transmitted from the server.
The monitoring system according to claim 3 or 4. Among the plurality of cameras, some of the plurality of cameras are installed under the same installation conditions.
The server transmits the learning result to each of the plurality of cameras.
Some of the plurality of cameras share the learning result transmitted from the server.
The monitoring system according to claim 5. The server controls the amount of learning processing according to the number of detected objects detected by the camera.
The monitoring system according to claim 2. The server controls the amount of processing of the learning in the camera according to the number of positive reports of the detection of the object detected by the camera.
The monitoring system according to claim 7. The server controls the amount of processing of the learning in the camera according to the number of false alarms of detection of the object detected by the camera.
The monitoring system according to claim 7. The server controls the amount of learning in the camera according to information about the amount of free resources in the camera.
The monitoring system according to claim 2. The server holds information on the amount of free resources of each of the server and the plurality of cameras constituting the monitoring system in the memory, and relates to the amount of the free resources of each of the server and the plurality of cameras. Controlling the processing amount of the learning according to the information,
The monitoring system according to claim 2. The processing includes learning to learn parameters used for detecting at least one object appearing in the captured image, detection to detect at least one object appearing in the captured image, and at least one detected by the detection. Analyzing objects, including,
The monitoring system according to claim 1. The server transmits the captured image held in the memory to at least one camera having a relatively large amount of free resources as compared with the other cameras among the plurality of cameras, and performs the learning. Give an execution instruction,
The monitoring system according to claim 12. The server transmits the captured image held in the memory to at least one camera having a relatively large amount of free resources as compared with the other cameras among the plurality of cameras, and performs the detection. Give an execution instruction,
The monitoring system according to claim 12. The server transmits the captured image held in the memory to at least one camera having a relatively large amount of free resources as compared with the other cameras among the plurality of cameras, and performs the analysis. Give an execution instruction,
The monitoring system according to claim 12. The process is a detection for detecting at least one object appearing in the captured image.
The server transmits the detection result to the plurality of cameras, respectively.
The plurality of cameras share the result of the detection.
The monitoring system according to claim 1. The server integrates the results of learning performed by the plurality of cameras.
The monitoring system according to claim 2. Among the plurality of cameras, some of the plurality of cameras are installed under the same installation conditions.
The server integrates the results of learning performed by the plurality of cameras.
The monitoring system according to claim 17. The server receives notifications of information about the installation status of each of the plurality of cameras from the plurality of cameras, and integrates the learning results performed by the plurality of cameras.
The monitoring system according to claim 18. The server and the plurality of cameras share information regarding the amount of free resources of the plurality of cameras and information regarding the unit price cost of the plurality of cameras.
The monitoring system according to claim 1. It is a monitoring method using a monitoring system in which a server and multiple cameras installed in the monitoring area are connected so that they can communicate with each other.
The server
Information on the amount of free resources for the processing capacity to generate the learning model of each of the cameras and the captured image obtained by imaging the surveillance area with each of the cameras are stored in memory.
Based on the information about the amount of free resources of the camera, the process performed by the camera with respect to the detection of at least one object appearing in the captured image obtained by each camera is determined and determined for each camera. The execution instruction of the process is transmitted for each camera, and the process is executed.
Each of the cameras executes a process corresponding to the execution instruction based on the execution instruction of the process transmitted from the server.
Monitoring method.

Images