JP7347775B2 - Monitoring device, monitoring system, monitoring method, and program - Google Patents

Description

The present invention relates to a monitoring device, a monitoring system, a monitoring method, and a program.

In factories such as glass factories and steel mills, raw materials such as silica sand, which is the raw material for glass, and iron ore, which is the raw material for steel, are stored in raw material yards installed either outdoors or indoors. Raw materials stored in raw material yards are transported to blast furnaces and the like using yard machines such as stackers and reclaimers. When monitoring sites such as raw material yards and blast furnaces (patrol, patrol inspection, environmental preparation, cleaning, maintenance inspection, etc.), workers regularly visit the site on foot or in a mobile vehicle (car, etc.). , workers monitor the site by visually observing, photographing with a camera, and checking the images.

There are several problems with the on-site monitoring method. For example, on-site monitoring that is left to humans to do covers a wide area and takes a lot of time. Furthermore, since raw materials come in various forms, the operator may have to check intuitively, and monitoring standards may vary depending on the operator's experience. Furthermore, since there are various types of raw materials, it is necessary not only to simply monitor but also to understand the properties of the raw materials.

In order to solve such problems, there is a method of automatically and continuously monitoring the site using a visible camera and an infrared camera installed at the site (for example, see Patent Document 1).

Japanese Patent Application Publication No. 7-298248

The following analysis is provided by the inventor.

However, in addition to the visible cameras and infrared cameras described in Patent Document 1 as monitoring means, various types of cameras can be used to monitor various conditions at the site, depending on the form and type of raw materials and customer requests. It is hoped that this will become the case.

The main object of the present invention is to provide a monitoring device, a monitoring system, a monitoring method, and a program that can contribute to monitoring various situations in the field.

The monitoring device according to the first viewpoint includes an imaging unit that converts the video data into image data when the video data from the surveillance camera that photographs the base is video data, and the video data is three-dimensional data. a point cloud forming unit that sometimes converts the three-dimensional data into point cloud data; and when in the first mode, the image data converted by the imaging unit or the point cloud forming unit converting the three-dimensional data into point cloud data; a predictive model creation unit that creates a predictive model through machine learning using point cloud data or the still image data when the photographic data is still image data; and a second mode different from the first mode. The image data converted by the imaging section, the point cloud data converted by the point cloud forming section, or the still image data when the photographed data is still image data. , a model determination unit that determines a change in the monitored object included in the image data, the point cloud data, or the still image data by comparing the prediction model created by the prediction model creation unit; When in the second mode, a predetermined form of the monitoring target included in the point cloud data converted by the point cloud forming unit is measured, and a measured value and a preset reference value are obtained; a reference value determination unit that determines a change in the monitored object by comparing the values, and the type of the image data, the point cloud data, or the still image data used to create the prediction model is The prediction model corresponds to the type of the image data, the point cloud data, or the still image data to be compared in the second mode, and the prediction model is data related to an object having features extracted by the machine learning. There is .

The monitoring system related to the second viewpoint includes a monitoring camera that photographs the base, and a monitoring device related to the first viewpoint.

The monitoring method according to the third viewpoint includes a step of converting the video data into image data when the video data from a surveillance camera that photographs the base is video data, and a step of converting the video data into image data when the video data is three-dimensional data. converting the three-dimensional data into point cloud data; and when in the first mode, the converted image data, the converted point group data, or the photographic data is still image data. a step of performing machine learning to create a predictive model using the still image data; and in a second mode different from the first mode, the converted image data or the converted point cloud. data, or the still image data when the photographed data is still image data, and the created prediction model, measuring a predetermined form of the monitoring object included in the converted point cloud data in the second mode, and measuring the measured value and a preset value. of the image data, the point cloud data, or the still image data used to create the prediction model, The type corresponds to the type of the image data, the point cloud data, or the still image data to be compared in the second mode, and the prediction model is based on the object having the features extracted by the machine learning. This is such data .

The program related to the fourth viewpoint includes a process of converting video data into image data when the photographed data from a surveillance camera photographing the base is video data, and a process of converting the video data into image data when the photographed data is three-dimensional data. A process of converting three-dimensional data into point cloud data, and when the converted image data, the converted point group data, or the photographic data is still image data in the first mode. A process of creating a predictive model by machine learning using the still image data, and the converted image data or the converted point cloud data in a second mode different from the first mode. , or, by comparing the still image data when the photographed data is still image data and the created prediction model, the information contained in the image data, the point cloud data, or the still image data a process of determining a change in the monitored object; and, in the second mode, measuring a predetermined form of the monitored object included in the converted point cloud data, and combining the measured measurement value with a preset A process of determining a change in the monitored object by comparing with a reference value determined by The type of still image data corresponds to the type of the image data, the point cloud data, or the still image data to be compared in the second mode, and the prediction model is based on the features extracted by the machine learning. This is data related to the objects that are held .

Note that the program can be recorded on a computer-readable storage medium. The storage medium can be non-transient, such as a semiconductor memory, a hard disk, a magnetic recording medium, an optical recording medium, etc. Further, the present disclosure can also be implemented as a computer program product.

According to the first to fourth viewpoints, it is possible to contribute to monitoring various situations at the site.

1 is a block diagram schematically showing the configuration of a monitoring system according to Embodiment 1. FIG. 1 is a block diagram schematically showing a detailed configuration of a monitoring device in a monitoring system according to a first embodiment; FIG. FIG. 3 is a diagram schematically showing a processing path of video data in learning mode of the monitoring device in the monitoring system according to the first embodiment. FIG. 3 is a diagram schematically showing a processing path of image data in a learning mode of the monitoring device in the monitoring system according to the first embodiment. FIG. 3 is a diagram schematically showing a three-dimensional data processing path in a learning mode of the monitoring device in the monitoring system according to the first embodiment. FIG. 3 is a diagram schematically showing a processing path of video data in the operation mode of the monitoring device in the monitoring system according to the first embodiment. FIG. 3 is a diagram schematically showing a processing path of image data in the operation mode of the monitoring device in the monitoring system according to the first embodiment. FIG. 3 is a diagram schematically showing a three-dimensional data processing route in the operation mode of the monitoring device in the monitoring system according to the first embodiment. FIG. 3 is a flowchart diagram schematically showing the operation of the imaging unit of the monitoring device in the monitoring system according to the first embodiment. FIG. 3 is a flowchart diagram schematically showing the operation of the point cloud forming unit of the monitoring device in the monitoring system according to the first embodiment. FIG. 3 is a flowchart diagram schematically showing the operation of a model determination unit of the monitoring device in the monitoring system according to the first embodiment. FIG. 3 is a flowchart diagram schematically showing the operation of a reference value determination unit of the monitoring device in the monitoring system according to the first embodiment. FIG. 2 is a block diagram schematically showing the configuration of a monitoring system according to a second embodiment. FIG. 3 is a block diagram schematically showing the configuration of a monitoring device according to a third embodiment. FIG. 2 is a block diagram schematically showing the configuration of hardware resources.

In the present disclosure described below, the monitoring device according to Mode 1 and its modified modes can be appropriately selected and combined.

The monitoring device according to Mode 1 includes an imaging unit that converts the moving image data into image data when the photographed data from the monitoring camera photographing the base is moving image data. The monitoring device includes a point group forming unit that converts the three-dimensional data into point group data when the photographic data is three-dimensional data. In the monitoring device, when in the first mode, the image data converted by the imaging unit, the point cloud data converted by the point cloud forming unit, or the photographed data is image data. The apparatus includes a predictive model creation unit that performs machine learning to create a predictive model using other image data. In a second mode different from the first mode, the monitoring device may display the image data converted by the imaging section, the point cloud data converted by the point cloud forming section, or the point cloud data converted by the point cloud forming section. By comparing the other image data when the imaging data is image data and the prediction model created by the prediction model creation unit, the image data, the point cloud data, or the other image data The model determination unit includes a model determination unit that determines changes in the monitored object included in the monitor. When in the second mode, the monitoring device measures a predetermined form of the monitoring target included in the point cloud data converted by the point cloud converting unit, and calculates a predetermined shape with the measured measurement value. The monitor includes a reference value determination unit that determines a change in the monitored object by comparing the reference value and the reference value determined by the monitor.

As a modified mode of the monitoring device according to Mode 1, a first route control section is provided that controls the output route of data from the monitoring camera. The monitoring device includes a second path control section that controls an output path of data from the imaging section, the point cloud formation section, or the first path control section. The first route control unit outputs the video data to the imaging unit when the photographic data is video data. The first path control section outputs the three-dimensional data to the point group forming section when the photographic data is three-dimensional data. The first route control section outputs the other image data when the photographic data is image data to the second route control section. When in the first mode, the second path control section controls the image data from the imaging section, the point cloud data from the point cloud formation section, or the first path control section. The other image data is output to the model creation section. The second route control unit outputs the image data from the imaging unit or the other image data from the first route control unit to the model determination unit when in the second mode. Alternatively, the point cloud data from the point cloud forming section is output to the model determining section and the reference value determining section. The monitoring device includes a data acquisition unit that acquires the photographic data from the surveillance camera and outputs the acquired data to the first route control unit. The monitoring device includes an output unit that outputs a determination result from at least one of the model determination unit and the reference value determination unit. The output unit outputs the determination result to the data acquisition unit. The data acquisition unit checks a change in the monitored object based on the determination result from the output unit, and adjusts a time interval for acquiring the photographed data from the surveillance camera depending on the degree of change. The photographic data includes at least one of time information, position information, camera information, and object information. The determination result includes at least one of the time information, the position information, the camera information, and the object information. The imaging unit converts the video data into the image data by cutting out the image data from the video data at a preset sampling period and subjecting the cut out image data to predetermined image processing. The predetermined image processing is at least one of refinement, resolution enhancement, noise removal, resolution reduction, opening processing, and morphology conversion. The point cloud generation unit converts the three-dimensional data into first-order point cloud data, generates model data based on the converted first-order point cloud data, and converts the generated model data into a second-order point cloud. By converting the three-dimensional data into data, the three-dimensional data is converted into the point cloud data. The point cloud forming unit performs filter processing on the converted point cloud data. In the first mode, the monitoring device may detect that the image data converted by the imaging section, the point cloud data converted by the point cloud forming section, or the photographic data is image data. The image forming apparatus includes a labeling section that adds a predictive model label to other image data at a certain time, and outputs the data to which the predictive model label has been added to the predictive model creating section.

In the present disclosure, the monitoring system according to Mode 2 and its modified modes can be appropriately selected and combined.

The monitoring system according to Mode 2 includes a monitoring camera that photographs a base, and a monitoring device according to Mode 1.

As a modified mode of the monitoring system according to Mode 2, an automatic driving vehicle is provided which is equipped with the monitoring camera, is also equipped with a satellite positioning device, and has a communication unit that enables communication with the monitoring device. The self-driving vehicle includes positioning information measured by the satellite positioning device in the photographic data taken by the monitoring camera, and transmits the data to the monitoring device via the communication unit.

In the present disclosure, as a monitoring method according to mode 3, the step of converting the video data into image data when the photographed data from the surveillance camera photographing the base is video data, and the step of converting the video data into image data, and the photographed data is three-dimensional data. Sometimes, the step of converting the three-dimensional data into point cloud data, and in the first mode, the converted image data, the converted point group data, or the photographic data is image data. a step of creating a predictive model by machine learning using other image data at the time; and a step of creating a predictive model by machine learning using other image data at the time, and at the time of a second mode different from the first mode, the converted image data or the converted point. By comparing the group data or the other image data when the photographed data is image data and the created prediction model, the image data, the point group data, or the other image data determining a change in the monitored object included in the second mode; and measuring a predetermined form of the monitored object included in the converted point cloud data in the second mode, and a measured value; and a step of determining a change in the monitored object by comparing with a preset reference value.

In the present disclosure, as a program related to mode 4, when the photographed data from a surveillance camera photographing a base is video data, the video data is converted into image data, and when the photographic data is three-dimensional data, the process converts the video data into image data. a process of converting the three-dimensional data into point cloud data, and when in a first mode, the converted image data, the converted point group data, or the photographic data is image data. A process of creating a predictive model by machine learning using other image data, and the converted image data or the converted point cloud when in a second mode different from the first mode. data, or the other image data when the photographed data is image data, and the created prediction model, the image data, the point cloud data, or the other image data. a process of determining a change in the included monitoring target; and, in the second mode, measuring a predetermined form of the monitoring target included in the converted point cloud data; A hardware resource is caused to execute a process of determining a change in the monitored object by comparing the set reference value with the set reference value.

Hereinafter, embodiments will be described with reference to the drawings. Note that when drawing reference symbols are used in this application, they are solely for the purpose of aiding understanding, and are not intended to limit the embodiments to the illustrated embodiments. Furthermore, the embodiments described below are merely illustrative and do not limit the present invention. Furthermore, connection lines between blocks in the drawings and the like referred to in the following description include both bidirectional and unidirectional connections. The unidirectional arrows schematically indicate the main signal (data) flow, and do not exclude bidirectionality. Furthermore, in the circuit diagrams, block diagrams, internal configuration diagrams, connection diagrams, etc. shown in the present disclosure, although not explicitly stated, an input port and an output port are present at the input end and output end of each connection line, respectively. The same applies to the input/output interface.

[Embodiment 1]
A monitoring system according to Embodiment 1 will be explained using the drawings. FIG. 1 is a block diagram schematically showing the configuration of a monitoring system according to a first embodiment. FIG. 2 is a block diagram schematically showing the detailed configuration of the monitoring device in the monitoring system according to the first embodiment. 3 to 8 are diagrams schematically showing processing paths of various data of the monitoring device in the monitoring system according to the first embodiment.

The monitoring system 1 is a system that monitors (inspects) objects to be monitored (hereinafter sometimes referred to as "objects") existing at bases 50a to 50n (for example, raw material yards, warehouses, factories, transportation routes, blast furnaces, etc.). be. The monitoring system 1 includes a monitoring device 10, monitoring cameras 30a to 30n, and a network 40.

Here, the objects to be monitored may be, for example, raw materials in a raw material yard or warehouse, a factory, a blast furnace, or fallen coal on a transport route.

The monitoring device 10 is a device that monitors objects existing at bases 50a to 50n using monitoring cameras 30a to 30n (see FIG. 1). The monitoring device 10 is communicably connected to the monitoring cameras 30a to 30n via a network 40. The monitoring device 10 includes a data acquisition unit 11, a route control unit 12, an imaging unit 13, a point cloud generation unit 14, a route control unit 15, a labeling unit 16, a predictive model creation unit 17, and a model. It has a determination section 18, a reference value determination section 19, and an output section 20 (see FIG. 2). For the monitoring device 10, for example, hardware resources (for example, an information processing device, a computer) including a processor, memory, network interface, etc. can be used. In this case, the hardware resources include a virtual data acquisition unit 11, a route control unit 12, an imaging unit 13, and a point cloud generation unit by executing the program in a processor while using a memory that stores the program. unit 14, route control unit 15, labeling unit 16, predictive model creation unit 17, model determination unit 18, reference value determination unit 19, and output unit 20 may be implemented.

The data acquisition unit 11 is a functional unit that acquires photographic data captured by a surveillance camera (30a to 30n in FIG. 1) via a network (40 in FIG. 1) (see FIG. 2). The data acquisition unit 11 is communicably connected to a surveillance camera (30a to 30n in FIG. 1) via a network (40 in FIG. 1). The data acquisition unit 11 outputs the acquired imaging data to the route control unit 12 (see FIGS. 3 to 8). The photographic data acquired by the data acquisition unit 11 is photographed by surveillance cameras of various photographic formats (30a to 30n in FIG. 1; infrared camera, hyperspectral camera, RGB (Red Green Blue) camera, three-dimensional sensor, depth sensor, etc.). It is possible to use captured photographic data (video data, image data, three-dimensional data, etc.).

Here, the photographed data is data photographed by the monitoring cameras 30a to 30n, and is any one of moving image data, image data, and three-dimensional data.

The route control unit 12 is a functional unit (first route control unit) that controls the output route of the imaging data acquired by the data acquisition unit 11 (see FIG. 2). The route control unit 12 determines the type of photographic data (video data, image data, three-dimensional data). When the photographed data is video data (video data taken with an infrared camera, hyperspectral camera, RGB camera, etc.), the route control unit 12 outputs the video data to the imaging unit 13 (FIG. 3, (See Figure 6). When the photographed data is image data (still image data taken with an infrared camera, hyperspectral camera, RGB camera, etc.), the route control unit 12 outputs the image data to the route control unit 15 (FIG. 4). , see Figure 7). When the photographed data is three-dimensional data (for example, three-dimensional data photographed with a three-dimensional camera, depth sensor, etc.), the path control section 12 outputs the three-dimensional data to the point group forming section 14 (see FIG. 5, see Figure 8).

The imaging unit 13 is a functional unit that converts the video data from the route control unit 12 into image data (see FIG. 2). The imaging unit 13 cuts out image data from the video data at a preset sampling period, and subjects the cut out image data to predetermined image processing (for example, refinement, high resolution, noise removal, low resolution, opening processing). , morphological conversion, etc.) to convert video data to image data. Note that image processing can be omitted if necessary. The imaging unit 13 outputs the converted image data to the route control unit 15 (see FIGS. 3 and 6).

The point cloud forming unit 14 is a functional unit that converts the three-dimensional data from the path control unit 12 into point cloud data (see FIG. 2). The point cloud generation unit 14 converts three-dimensional data into first-order point cloud data, generates model data (for example, CAD (Computer Aided Design) data) based on the converted first-order point cloud data, and generates model data (for example, CAD (Computer Aided Design) data). The three-dimensional data is converted to point cloud data by converting the model data into quadratic point cloud data. When converting into point cloud data, the point cloud forming unit 14 may perform filter processing on the point cloud data, remove outlier points, or perform downsampling, as necessary. The creation unit 17 may remove the point cloud other than the part to be subjected to machine learning. Note that generation of model data and conversion of model data to point cloud data can be omitted as necessary. The point cloud generation unit 14 outputs the converted point cloud data (secondary point cloud data when model data is generated) to the path control unit 15 (see FIGS. 5 and 8). Here, the reason why the 3D data is converted into point cloud data in the point cloud generation unit 14 is because the format of 3D data captured by a 3D camera or depth sensor is often manufacturer-specific and not standardized. This is because while it is difficult to perform uniform processing, point cloud data has a unified format and is easy to perform uniform processing.

The route control unit 15 is a functional unit (second route) that controls the output route of the image data from the imaging unit 13, the image data from the route control unit 12, and the point cloud data from the point cloud generation unit 14. (control unit) (see Figure 2). In the learning mode, the route control unit 15 sends the image data from the imaging unit 13, the image data from the route control unit 12, or the point cloud data from the point cloud formation unit 14 to the labeling unit 16. (See Figures 3, 4, and 5). In the operation mode, the route control unit 15 outputs image data from the imaging unit 13 or image data from the route control unit 12 to the model determination unit 18 (see FIGS. 6 and 7). In the operation mode, the route control unit 15 outputs the point cloud data from the point cloud formation unit 14 to the model determination unit 18 and the reference value determination unit 19 (see FIG. 8).

Here, the learning mode is a mode (first mode) when performing machine learning (extracting features such as useful rules, knowledge expressions, and judgment criteria) in the predictive model creation unit 17. The operation mode is a mode (second mode) in which the model determination unit 18 and the reference value determination unit 19 perform normal operation (determination processing).

The labeling unit 16 is a functional unit that adds a predictive model label to the data (image data, point cloud data) from the route control unit 15 in the learning mode (see FIG. 2). The labeling unit 16 outputs data to which a predictive model label has been added (labeled data) to the predictive model creating unit 17 .

The predictive model creation unit 17 is a functional unit that creates a predictive model by performing machine learning (extracting features such as useful rules, knowledge expressions, and judgment criteria) using the labeled data from the labeling unit 16. . The prediction model is a model (data) related to a target object (object) having features extracted by machine learning (features of the target object to be predicted), and serves as a determination standard in the model determination unit 18. The predictive model creation unit 17 outputs the created predictive model to the model determination unit 18.

The model determination unit 18 compares the data (image data, point cloud data) from the route control unit 15 with the corresponding prediction model created by the prediction model creation unit 17, and determines the targets included in the data. This is a functional unit that determines changes in objects (see FIGS. 2 and 6 to 8). The model determination unit 18 holds the latest prediction model created by the prediction model creation unit 17. Judgments include temperature (if the data is image data derived from an infrared camera), component (if the data is image data derived from a hyperspectral camera), appearance (if the data is image data derived from an RGB camera), and point cloud (data is point cloud data derived from a three-dimensional camera). In determining a change in the target object, the determination can be made using the certainty factor (%) that the object exists and the certainty factor (%) that the object does not exist. For example, the determination can be made such that if the certainty level of the determination that the object is present is 85% or higher, it is finally determined that the object is present. The confidence level can be tuned depending on the situation to be determined. In addition, when determining changes in objects, it is possible to determine how much (for example, by what percentage) the objects included in the data have changed relative to the prediction model, and if the change is within the upper and lower limits. It can be determined whether or not there is. The model determination unit 18 outputs a result (determination result) related to determination of a change in the object to the output unit 20 . The determination result can include time information, position information, camera information, object information, and the like.

The reference value determination unit 19 uses a measured value related to a predetermined form of the object included in the point cloud data from the path control unit 15 (for example, a measured value of a form such as volume, shape size, area, etc.) and a preset value. This is a functional unit that determines a change in the target object by comparing it with the reference value determined (see FIGS. 2 and 8). The reference value determination unit 19 extracts a point group related to the object from the point group data from the route control unit 15. The reference value determination unit 19 measures a predetermined form of the target object based on the extracted point group related to the target object. In the learning mode, the reference value determination unit 19 holds a measured value of a predetermined form related to the measured object as a reference value. The reference value is set in advance according to a predetermined form of the object to be measured, and may be, for example, volume, shape size, area, etc. In the operation mode, the reference value determination unit 19 determines a change in the object by comparing a predetermined measurement value of the measured object with a preset reference value. In determining changes in an object, it is possible to determine how much (for example, by what percentage) a measured value of a predetermined form of the object has changed with respect to a reference value, and whether the degree of change is within the upper and lower limits. It is possible to determine whether it is within the range. The reference value determination unit 19 outputs a result (determination result) related to determination of a change in the object to the output unit 20 . The determination result can include time information, position information, camera information, object information, and the like.

The output unit 20 is a functional unit that outputs (displays, transmits, outputs audio, prints, etc.) the determination results from the model determination unit 18 or the reference value determination unit 19 (see FIG. 2). The output unit 20 outputs the determination result to the data acquisition unit 11, and the data acquisition unit 11 checks the change in the object based on the determination result, and monitors the surveillance cameras 30a to 30a according to the degree of change. The time interval for acquiring photographic data from 30n may be adjusted. For example, if the change in the object is large, the time interval for acquiring photographic data can be shortened, and if the change in the object is small, the time interval for acquiring photographic data can be lengthened. By shortening the time interval for acquiring photographic data, changes in the object can be grasped quickly and accurately, and by lengthening the time interval for acquiring photographic data, the power consumption of the system can be reduced. . As the output unit 20, for example, a display for displaying images, an interface for data transfer, a speaker for outputting audio, a printer for printing, or other output means can be used, and a wired or An information terminal or the like having an output section connected to enable wireless communication may also be used.

The surveillance cameras 30a to 30n are cameras (including sensors) that photograph objects to be monitored (see FIG. 1). The surveillance cameras 30a to 30n generate photographic data (any of video data, image data, or three-dimensional data) regarding the photographed object, and send the generated photographic data to the data acquisition unit of the monitoring device 10 (Figs. 2 to 3). The output is directed to 11) in FIG. The surveillance cameras 30a to 30n may output photographed data according to the time interval at which the data acquisition unit (11 in FIGS. 2 to 8) acquires photographed data. Various cameras can be used as the surveillance cameras 30a to 30n. The photographic data can include time information, position information, camera information, object information, and the like.

Surveillance cameras 30a to 30n are installed at a large number of bases 50a to 50n. Various cameras are selected from the surveillance cameras 30a to 30n according to the customer's requests. Not only one but two or more surveillance cameras 30a to 30n may be installed at each base 50a to 50n, and a plurality of cameras of different types may be installed at one base. To confirm a customer's request regarding what purpose, what, and range they want to monitor when monitoring an object. This makes it possible to flexibly combine service deployments according to customer needs.

For example, an infrared camera, a hyperspectral camera, an RGB camera, a three-dimensional camera, a depth sensor, etc. can be used as the surveillance cameras 30a to 30n.

For example, when the target object is fallen coal, there is a risk that the fallen coal may catch fire, so infrared cameras capable of detecting temperature can be used as the monitoring cameras 30a to 30n. According to an infrared camera, temperature changes in fallen coal can be extracted from temperature information. This allows determination at temperatures that are invisible to the human eye.

In addition, since there is not only fallen coal but also deposits of soil and other components on the factory site, the surveillance cameras 30a to 30n are equipped with hyperspectral cameras that can detect the components. Can be used. According to the hyperspectral camera, the components of fallen coal can be extracted using wavelength information. This allows determination based on components that are invisible to the human eye.

Further, when detecting the "appearance" characteristics of the object, RGB cameras can be used as the surveillance cameras 30a to 30n. According to the RGB camera, it is possible to extract fallen coal based on visual texture information and the like. This makes it possible to make a judgment similar to that made by human appearance.

Further, when detecting the size or volume of the object, a three-dimensional camera or a depth sensor can be used as the monitoring cameras 30a to 30n. By using 3D cameras and depth sensors, it is possible to accurately detect the size and volume of irregularly shaped objects (raw materials, fallen coal) by utilizing 3D data that cannot be measured with the human eye. It is possible to manage the inventory of raw materials and automatically order raw materials that are in short supply, and it is also possible to manage the accumulation of fallen coal and narrow down the location and timing of cleaning. There are many types of 3D cameras, such as 3D-LiDAR (Light Detection And Ranging), ToF (Time of Flight) cameras, and stereo cameras, depending on indoor/outdoor shooting, shooting distance, and shooting system. Although the format of the three-dimensional data is different, the point cloud generator 14 can perform uniform processing, so any type of three-dimensional camera can be selected according to the customer's application.

Note that if only one type of surveillance camera 30a to 30n is installed at one base 50a to 50n, it may be difficult to determine changes in objects; ~30n is installed to obtain and fuse a plurality of types of determination results, it is possible to effectively determine changes in the object.

The network 40 is an information communication network that connects the monitoring device 10 and the monitoring cameras 30a to 30n in a wired or wirelessly communicable manner (see FIG. 1). The network 40 includes, for example, a LAN (Local Area Network), a PAN (Personal Area Network), a CAN (Campus Area Network), a MAN (Metropolitan Area Network), a WAN (Wide Area Network), a GAN (Global Area Network), etc. Can be used.

Next, the operation of the imaging unit of the monitoring device in the monitoring system according to the first embodiment will be explained using the drawings. FIG. 9 is a flowchart diagram schematically showing the operation of the imaging unit of the monitoring device in the monitoring system according to the first embodiment. Note that for the components of the monitoring system 1 and the monitoring device 10, please refer to FIGS. 1 and 2.

First, the imaging unit 13 acquires video data (video data captured by the surveillance cameras 30a to 30n (infrared camera, hyperspectral camera, RGB camera, etc.)) from the route control unit 12 (step A1).

Next, the imaging unit 13 cuts out image data at a preset sampling period from the video data acquired in step A1 (step A2).

Next, the imaging unit 13 refines (for example, increases resolution and removes noise) the image data cut out in step A2 (step A3).

Finally, the imaging unit 13 outputs the image data refined in step A3 to the path control unit 15 (step A4), and then ends the process.

Next, the operation of the point cloud forming section of the monitoring device in the monitoring system according to the first embodiment will be explained using the drawings. FIG. 10 is a flowchart diagram schematically showing the operation of the point cloud forming section of the monitoring device in the monitoring system according to the first embodiment. Note that for the components of the monitoring system 1 and the monitoring device 10, please refer to FIGS. 1 and 2.

First, the point group forming unit 14 acquires three-dimensional data from the path control unit 12 (step B1).

Next, the point cloud forming unit 14 converts the three-dimensional data acquired in step B1 into first-order point cloud data (step B2).

Next, the point group forming unit 14 generates model data (for example, CAD data) based on the primary point group data converted in step B2 (step B3).

Next, the point group forming unit 14 converts the model data generated in step B3 into secondary point group data (step B4).

Finally, the point group forming unit 14 outputs the secondary point group data converted in step B4 to the path control unit 15 (step B5), and then ends the process.

Next, the operation of the model determination unit of the monitoring device in the monitoring system according to the first embodiment will be explained using the drawings. FIG. 11 is a flowchart diagram schematically showing the operation of the model determination unit of the monitoring device in the monitoring system according to the first embodiment. Note that for the components of the monitoring system 1 and the monitoring device 10, please refer to FIGS. 1 and 2. Here, it is assumed that a predictive model is created in advance by the predictive model creating section 17, and the created predictive model is held in the model determining section 18.

First, the model determination unit 18 acquires data (image data, point cloud data (secondary point cloud data if model data is generated)) from the route control unit 15 (step C1).

Next, the model determination unit 18 compares the data acquired in step C1 and the corresponding prediction model created by the prediction model creation unit 17 (step C2).

Next, the model determination unit 18 determines a change in the object included in the data acquired in step C1 based on the comparison in step C2 (step C3).

Finally, the model determination unit 18 outputs the result (determination result) related to the determination in step C3 to the output unit 20 (step C4), and then ends the process.

Next, the operation of the reference value determining section of the monitoring device in the monitoring system according to the first embodiment will be described using the drawings. FIG. 12 is a flowchart diagram schematically showing the operation of the reference value determination unit of the monitoring device in the monitoring system according to the first embodiment. Note that for the components of the monitoring system 1 and the monitoring device 10, please refer to FIGS. 1 and 2.

First, the reference value determination unit 19 acquires point cloud data from the route control unit 15 (step D1).

Next, the reference value determination unit 19 extracts a point group related to the object from the point group data acquired in step D1 (step D2).

Next, the reference value determination unit 19 measures a predetermined form of the object based on the point group related to the object extracted in step D2 (step D3).

Next, the reference value determination unit 19 uses a predetermined type of measurement value (for example, a measurement value of a volume, shape size, etc.) of the object measured in step D3 and a preset reference value. , are compared (step D4).

Next, the reference value determination unit 19 determines a change in the object included in the point cloud data acquired in step D1 by the comparison in step D4 (step D5).

Finally, the reference value determination unit 19 outputs the result (determination result) related to the determination of the change in the object in step D5 to the output unit 20 (step D6), and then ends the process.

The above-mentioned monitoring system 1 can be used to manage raw materials in the smart factory field, mine-fall coal management in the mining field, raw material management in the food manufacturing industry, waste management in the waste treatment industry, and various other systems. It can be used to manage toilet cleaning in facilities, etc.

According to the first embodiment, the following effects are achieved.

The first effect is that it is possible to select the optimal type of surveillance cameras 30a to 30n according to the customer's requests, which can contribute to monitoring various situations at the site.

The second effect is that it is possible to detect the shape, size, volume, temperature, composition, etc. of the monitored object from the photographed data, so it is possible to monitor elements that cannot be determined by the worker's appearance alone. can do.

The third effect is that by installing the surveillance cameras 30a to 30n at the location you want to monitor, workers can remotely monitor the shape, size, volume, temperature, components, etc. of the monitored object without having to go to the site. This makes monitoring work more efficient.

The fourth effect is that an abnormality in the monitored object can be detected by setting a reference value according to the detected content of the monitored object.

The fifth effect is that various surveillance cameras 30a to 30n can be fused and utilized in one surveillance system 1, and the quality of the surveillance work itself can be improved.

The sixth effect is that the power consumption of the monitoring system 1 is reduced because not only is monitoring performed regularly, but also the time interval for monitoring can be adjusted according to changes in the status of the monitored object. be able to.

The seventh effect is that by using three-dimensional cameras as the surveillance cameras 30a to 30n, it becomes possible to acquire positions, narrow down areas to be cleaned and items to be ordered, and perform wide-ranging inspections and monitoring. It can be realized.

The eighth effect is that in raw material inventory management, it is possible to automatically order items that are in short supply, and to manage the period up to a stable state after inventory replenishment as a life cycle. In addition, in managing the accumulation of fallen coal, it is possible to notify the timing of cleaning.

[Embodiment 2]
A monitoring system according to a second embodiment will be explained using the drawings. FIG. 13 is a block diagram schematically showing the configuration of a monitoring system according to the second embodiment.

Embodiment 2 is a modification of Embodiment 1, and instead of installing a surveillance camera at each base 50a to 50n, a surveillance camera 30 (one surveillance camera or multiple types of surveillance cameras may be used) and a communication unit 80 are installed. Using an autonomous vehicle 60 equipped with a wireless communication unit (wireless communication unit), the monitoring device 10 can acquire photographic data via the communication unit 80 and the network 40 by patrolling the bases 50a to 50n where objects to be monitored exist. This is how it was done. The automatic driving vehicle 60 is equipped with a satellite positioning device 70 (eg, GPS; Global Positioning System). The positioning information measured by the satellite positioning device 70 is included in the photographic data and transmitted to the monitoring device 10 via the communication unit 80 and the network 40. The monitoring device 10 can specify the photographing position of the photographic data using the positioning information. The other configurations are the same as in the first embodiment.

According to the second embodiment, similarly to the first embodiment, it is possible to contribute to monitoring various situations at the site.

[Embodiment 3]
A monitoring device according to Embodiment 3 will be explained using the drawings. FIG. 14 is a block diagram schematically showing the configuration of a monitoring device according to the third embodiment.

The monitoring device 10 is a device that uses a monitoring camera 30 to monitor objects to be monitored at the base 50 (see FIG. 14). The monitoring device 10 includes an imaging section 13 , a point group forming section 14 , a predictive model creating section 17 , a model determining section 18 , and a reference value determining section 19 .

The imaging unit 13 converts the video data into image data when the photographic data from the surveillance camera 30 photographing the base 50 is video data (see FIG. 14).

The point group forming unit 14 converts the three-dimensional data into point group data when the imaging data is three-dimensional data (see FIG. 14).

When in the first mode, the predictive model creation unit 17 generates image data when the image data converted by the imaging unit 13, point cloud data converted by the point cloud formation unit 14, or photographic data is image data. A predictive model is created by machine learning using other image data (see FIG. 14).

The model determination unit 18 determines whether the image data converted by the imaging unit 13, the point cloud data converted by the point cloud formation unit 14, or the photographic data is in a second mode different from the first mode. By comparing other image data when it is image data and the prediction model created by the prediction model creation unit 17, changes in the monitoring target included in the image data, point cloud data, or other image data can be detected. Determine.

The reference value determination unit 19 measures a predetermined form of the monitoring target included in the point cloud data converted by the point cloud formation unit 14 in the second mode, and calculates the measured value and a preset value. Changes in the monitored object are determined by comparing with the reference value.

According to the third embodiment, similarly to the first embodiment, it is possible to contribute to monitoring various situations at the site.

Note that the monitoring devices according to Embodiments 1 to 3 can be configured by so-called hardware resources (information processing devices, computers), and those having the configuration illustrated in FIG. 15 can be used. For example, the hardware resource 100 includes a processor 101, a memory 102, a network interface 103, etc., which are interconnected by an internal bus 104.

Note that the configuration shown in FIG. 15 is not intended to limit the hardware configuration of the hardware resource 100. The hardware resource 100 may include hardware (for example, an input/output interface) that is not shown. Alternatively, the number of units such as the processors 101 included in the device is not limited to the example shown in FIG. 15; for example, a plurality of processors 101 may be included in the hardware resource 100. For example, a CPU (Central Processing Unit), an MPU (Micro Processor Unit), a GPU (Graphics Processing Unit), etc. can be used as the processor 101.

As the memory 102, for example, RAM (Random Access Memory), ROM (Read Only Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), etc. can be used.

For the network interface 103, for example, a LAN (Local Area Network) card, a network adapter, a network interface card, etc. can be used.

The functions of the hardware resource 100 are realized by the processing modules described above. The processing module is realized, for example, by the processor 101 executing a program stored in the memory 102. Further, the program can be updated via a network or by using a storage medium storing the program. Furthermore, the processing module may be realized by a semiconductor chip. That is, the functions performed by the processing module need only be realized by executing software on some kind of hardware.

Some or all of the above embodiments may be described as in the following supplementary notes, but are not limited to the following.

[Additional note 1]
In the present invention, the configuration of the monitoring device according to the first viewpoint is possible, and is as follows.
an imaging unit that converts the video data into image data when the shooting data from a surveillance camera that photographs the base is video data;
a point group forming unit that converts the three-dimensional data into point cloud data when the photographic data is three-dimensional data;
In the first mode, the image data converted by the imaging section, the point cloud data converted by the point group forming section, or another image when the photographic data is image data. A predictive model creation unit that uses data to perform machine learning to create a predictive model;
When in a second mode different from the first mode, the image data converted by the imaging section, the point cloud data converted by the point group forming section, or the photographed data is image data. By comparing the other image data when a model determination unit that determines changes in objects;
When in the second mode, a predetermined form of the monitoring target included in the point cloud data converted by the point cloud forming unit is measured, and a measured value and a preset reference value are obtained; a reference value determination unit that determines a change in the monitored object by comparing the
Equipped with
Monitoring equipment.

[Additional note 2]
a first route control unit that controls an output route of data from the surveillance camera;
a second route control unit that controls an output route of data from the imaging unit, the point grouping unit, or the first route control unit;
Furthermore,
The first route control unit includes:
When the photographic data is video data, outputting the video data to the imaging unit,
when the photographic data is three-dimensional data, outputting the three-dimensional data to the point group forming section;
outputting the other image data when the photographing data is image data to a second path control unit;
The second route control section includes:
When in the first mode, the image data from the imaging section, the point cloud data from the point group forming section, or the other image data from the first path control section are used as the model. Output to the creation department,
When in the second mode, the image data from the imaging section or the other image data from the first path control section is output to the model determination section, or the point grouping is performed. outputting the point cloud data from the unit to the model determining unit and the reference value determining unit;
The monitoring device described in Appendix 1.

[Additional note 3]
a data acquisition unit that acquires the photographic data from the surveillance camera and outputs it to the first route control unit;
an output unit that outputs a determination result from at least one of the model determination unit and the reference value determination unit;
Furthermore,
The output unit outputs the determination result to the data acquisition unit,
The data acquisition unit checks a change in the monitored object based on the determination result from the output unit, and adjusts a time interval for acquiring the photographed data from the surveillance camera according to the degree of change.
Monitoring device described in Appendix 2.

[Additional note 4]
The photographic data includes at least one of time information, position information, camera information, and object information,
The determination result includes at least one of the time information, the position information, the camera information, and the object information.
Monitoring device described in Appendix 3.

[Additional note 5]
The imaging unit cuts out the image data from the video data at a preset sampling period, and converts the video data into the image data by performing predetermined image processing on the cut out image data.
The monitoring device according to any one of Supplementary Notes 1 to 4.

[Additional note 6]
The predetermined image processing is at least one of refinement, resolution enhancement, noise removal, resolution reduction, opening processing, and morphological conversion.
The monitoring device described in Appendix 5.

[Additional note 7]
The point cloud generation unit converts the three-dimensional data into first-order point cloud data, generates model data based on the converted first-order point cloud data, and converts the generated model data into a second-order point cloud. converting the three-dimensional data into the point cloud data by converting it into data;
The monitoring device according to any one of Supplementary Notes 1 to 6.

[Additional note 8]
The point cloud forming unit performs filter processing on the converted point cloud data.
Monitoring device described in Appendix 7.

[Additional note 9]
When in the first mode, the image data converted by the imaging section, the point cloud data converted by the point group forming section, or other when the photographic data is image data. further comprising a labeling unit that adds a predictive model label to the image data and outputs the data to which the predictive model label is added to the predictive model creating unit;
The monitoring device according to any one of Supplementary Notes 1 to 6.

[Additional note 10]
In the present invention, the form of the monitoring system according to the second viewpoint is possible, and is as follows.
Surveillance cameras that photograph the base,
A monitoring device according to any one of Supplementary Notes 1 to 9,
Equipped with
Monitoring system.

[Additional note 11]
Further comprising an automatic driving vehicle equipped with the surveillance camera, a satellite positioning device, and a communication unit capable of communicating with the surveillance device,
The self-driving vehicle includes positioning information obtained by the satellite positioning device in the photographic data taken by the monitoring camera, and transmits the data to the monitoring device via the communication unit.
The monitoring system described in Appendix 10.

[Additional note 12]
In the present invention, the monitoring method according to the third viewpoint is possible, and is as follows.
When the photographed data from a surveillance camera photographing the base is video data, converting the video data into image data;
converting the three-dimensional data into point cloud data when the photographic data is three-dimensional data;
In the first mode, a predictive model is modeled by machine learning using the converted image data, the converted point cloud data, or other image data when the shooting data is image data. and the steps to create
When in a second mode different from the first mode, the converted image data, the converted point group data, or the other image data when the photographic data is image data; determining a change in the monitored object included in the image data, the point cloud data, or the other image data by comparing with the created prediction model;
In the second mode, by measuring a predetermined form of the monitored object included in the converted point cloud data and comparing the measured value with a preset reference value, determining a change in the monitored object;
including,
Monitoring method.

[Additional note 13]
In the present invention, the form of the program according to the fourth aspect is possible, and is as follows.
When the photographed data from a surveillance camera photographing the base is video data, a process of converting the video data into image data;
a process of converting the three-dimensional data into point cloud data when the photographic data is three-dimensional data;
In the first mode, a predictive model is modeled by machine learning using the converted image data, the converted point cloud data, or other image data when the shooting data is image data. The process of creating
When in a second mode different from the first mode, the converted image data, the converted point group data, or the other image data when the photographic data is image data; A process of determining a change in the monitored object included in the image data, the point cloud data, or the other image data by comparing the created prediction model with the created prediction model;
In the second mode, by measuring a predetermined form of the monitored object included in the converted point cloud data and comparing the measured value with a preset reference value, a process of determining a change in the monitored object;
to be executed by hardware resources,
program.

Furthermore, the disclosures of the above patent documents are incorporated into this book by reference. Within the framework of the entire disclosure of the present invention (including claims and drawings), changes and adjustments to the embodiments and examples are possible based on the basic technical idea thereof. Furthermore, various combinations or selections (as necessary) of various disclosed elements (including each element of each claim, each element of each embodiment or example, each element of each drawing, etc.) within the framework of the entire disclosure of the present invention. (not selected) is possible. That is, it goes without saying that the present invention includes the entire disclosure including the claims and drawings, as well as various modifications and modifications that a person skilled in the art would be able to make in accordance with the technical idea. Furthermore, with respect to the numerical values and numerical ranges described in this application, any intermediate values, lower numerical values, and small ranges thereof are deemed to be included even if not explicitly stated.

1 Monitoring system 10 Monitoring device 11 Data acquisition unit 12 Route control unit 13 Imaging unit 14 Point grouping unit 15 Route control unit 16 Labeling unit 17 Prediction model creation unit 18 Model determination unit 19 Reference value determination unit 20 Output unit 30, 30a to 30n Surveillance camera 40 Network 50, 50a to 50n Base 60 Automatic driving vehicle 70 Satellite positioning device 80 Communication unit 100 Hardware resources 101 Processor 102 Memory 103 Network interface 104 Internal bus

Claims (10)

an imaging unit that converts the video data into image data when the shooting data from a surveillance camera that photographs the base is video data;
a point group forming unit that converts the three-dimensional data into point cloud data when the photographic data is three-dimensional data;
In the first mode, the image data converted by the imaging section, the point cloud data converted by the point group forming section, or the still image data when the photographic data is still image data. a predictive model creation unit that performs machine learning to create a predictive model using the image data;
When in a second mode different from the first mode, the image data converted by the imaging section, the point cloud data converted by the point cloud forming section, or the photographed data is a still image. By comparing the still image data when it is data with the prediction model created by the prediction model creation unit, the monitoring target included in the image data, the point cloud data, or the still image data is detected. a model determination unit that determines a change in
When in the second mode, a predetermined form of the monitoring target included in the point cloud data converted by the point cloud forming unit is measured, and a measured value and a preset reference value are obtained; a reference value determination unit that determines a change in the monitored object by comparing the
Equipped with
The type of the image data, the point cloud data, or the still image data used to create the prediction model corresponds to the type of the image data, the point cloud data, or the still image data to be compared in the second mode. and
The prediction model is data related to an object having features extracted by the machine learning,
Monitoring equipment. a first route control unit that controls an output route of data from the surveillance camera;
a second route control unit that controls an output route of data from the imaging unit, the point grouping unit, or the first route control unit;
Furthermore,
The first route control unit includes:
When the photographic data is video data, outputting the video data to the imaging unit,
when the photographic data is three-dimensional data, outputting the three-dimensional data to the point group forming section;
outputting the still image data when the photographing data is still image data to a second path control unit;
The second route control section includes:
When in the first mode, the image data from the imaging unit, the point cloud data from the point cloud forming unit, or the still image data from the first path control unit are used in the prediction model. Output to the creation department,
When in the second mode, the image data from the imaging section or the still image data from the first path control section is output to the model determination section, or the point cloud formation section outputting the point cloud data from to the model determination unit and the reference value determination unit;
The monitoring device according to claim 1. a data acquisition unit that acquires the photographic data from the surveillance camera and outputs it to the first route control unit;
an output unit that outputs a determination result from at least one of the model determination unit and the reference value determination unit;
Furthermore,
The output unit outputs the determination result to the data acquisition unit,
The data acquisition unit checks a change in the monitored object based on the determination result from the output unit, and adjusts a time interval for acquiring the photographed data from the surveillance camera according to the degree of change.
The monitoring device according to claim 2. The imaging unit cuts out the image data from the video data at a preset sampling period, and converts the video data into the image data by performing predetermined image processing on the cut out image data.
A monitoring device according to any one of claims 1 to 3. The point cloud generation unit converts the three-dimensional data into first-order point cloud data, generates model data based on the converted first-order point cloud data, and converts the generated model data into a second-order point cloud. converting the three-dimensional data into the point cloud data by converting it into data;
A monitoring device according to any one of claims 1 to 4. When in the first mode, the image data converted by the imaging section, the point cloud data converted by the point group forming section, or the photographed data is still image data. Further comprising a labeling unit that adds a predictive model label to the still image data and outputs the data to which the predictive model label is added to the predictive model creation unit.
A monitoring device according to any one of claims 1 to 5. Surveillance cameras that photograph the base,
A monitoring device according to any one of claims 1 to 6,
Equipped with
Monitoring system. Further comprising an automatic driving vehicle equipped with the surveillance camera, a satellite positioning device, and a communication unit capable of communicating with the surveillance device,
The self-driving vehicle includes positioning information obtained by the satellite positioning device in the photographic data taken by the monitoring camera, and transmits the data to the monitoring device via the communication unit.
The monitoring system according to claim 7. When the photographed data from a surveillance camera photographing the base is video data, converting the video data into image data;
converting the three-dimensional data into point cloud data when the photographic data is three-dimensional data;
In the first mode, prediction is performed by machine learning using the converted image data, the converted point cloud data, or the still image data when the photographed data is still image data. a step of creating a model;
When in a second mode different from the first mode, the converted image data, the converted point group data, or the still image data when the photographic data is still image data; determining a change in the monitored object included in the image data, the point cloud data, or the still image data by comparing with the created prediction model;
In the second mode, by measuring a predetermined form of the monitored object included in the converted point cloud data and comparing the measured value with a preset reference value, determining a change in the monitored object;
including;
The type of the image data, the point cloud data, or the still image data used to create the prediction model corresponds to the type of the image data, the point cloud data, or the still image data to be compared in the second mode. and
The prediction model is data related to an object having features extracted by the machine learning,
Monitoring method. When the photographed data from a surveillance camera photographing the base is video data, a process of converting the video data into image data;
a process of converting the three-dimensional data into point cloud data when the photographic data is three-dimensional data;
In the first mode, prediction is performed by machine learning using the converted image data, the converted point cloud data, or the still image data when the photographed data is still image data. The process of creating a model,
When in a second mode different from the first mode, the converted image data, the converted point group data, or the still image data when the photographic data is still image data; A process of determining a change in a monitoring target included in the image data, the point cloud data, or the still image data by comparing the created prediction model with the created prediction model;
In the second mode, by measuring a predetermined form of the monitored object included in the converted point cloud data and comparing the measured value with a preset reference value, a process of determining a change in the monitored object;
to be executed by hardware resources ,
The type of the image data, the point cloud data, or the still image data used to create the prediction model corresponds to the type of the image data, the point cloud data, or the still image data to be compared in the second mode. and
The prediction model is data related to an object having features extracted by the machine learning,
program.

Images