JP2020087036A - Monitoring apparatus, monitoring system, monitoring method, and program - Google Patents

Abstract

To provide a monitoring apparatus and so on which can contribute to monitoring under various states of a site.SOLUTION: A monitoring apparatus according to the present invention has an imaging unit for converting motion picture data into image data if captured data from a monitoring camera is motion picture data, a point group making unit for converting three-dimensional data into point group data if the captured data is the three-dimensional data, a predication model generating unit for generating, in a first mode, a prediction model through machine learning with using the image data or the point group data, a model determining unit for determining, in a second mode, change of a monitoring object included in the data by comparing the image data or the point group data with the prediction model, and a reference value determining unit for measuring a predetermined mode of the monitoring object included in the point group data in the second mode and for determining the change of the monitoring object by comparing the measured value with a preset reference value.SELECTED DRAWING: Figure 14

Description

The present invention relates to a monitoring device, a monitoring system, a monitoring method, and a program.

In factories such as glass factories and steel mills, raw materials such as silica sand which is a raw material of glass and iron ore which is a raw material of steel are stored in a raw material yard installed outdoors or indoors of the factory. Raw materials stored in the raw material yard are transferred to a blast furnace or the like by a yard machine such as a stacker or a reclaimer. When monitoring the site of raw material yard, blast furnace, etc. (patrol, patrol inspection, environmental maintenance, cleaning, maintenance inspection, etc.), workers go to the site regularly on foot or on a moving body (car etc.) The operator monitors the site by visually checking it or taking a picture with a camera and checking the taken image.

There are several problems with the onsite visit and monitoring method. For example, manual monitoring of a site takes a long time because it covers a wide area. In addition, since there are various forms of raw materials, there is a possibility that the confirmation of the operator becomes sensuous, and the monitoring standard varies depending on the experience of the operator. Furthermore, since there are various types of raw materials, it is necessary to understand the properties of the raw materials as well as simple monitoring.

In order to solve such a problem, there is a method capable of automatically and continuously monitoring the site by using a visible camera and an infrared camera installed on the site (for example, refer to Patent Document 1).

JP-A-7-298248

The following analysis is given by the inventor of the present application.

However, as the monitoring means, not only the visible camera and the infrared camera described in Patent Document 1, but also various types of cameras can be used to monitor various situations in the field according to the form and type of raw materials and the customer's request. Is expected to be.

A main object of the present invention is to provide a monitoring device, a monitoring system, a monitoring method, and a program that can contribute to monitoring various situations in the field.

The monitoring device according to the first aspect is an imaging unit that converts the moving image data into image data when the shooting data from the monitoring camera that shoots the base is moving image data, and the shooting data is three-dimensional data. Sometimes a point grouping unit for converting the three-dimensional data into point group data, and in the first mode, the image data converted by the image forming unit, or the image data converted by the point grouping unit. A prediction model creation unit that creates a prediction model by machine learning using point cloud data or other image data when the shooting data is image data, and a second mode different from the first mode. Sometimes, the image data converted by the imaging unit, or the point group data converted by the point grouping unit, or the other image data when the shooting data is image data, By comparing the prediction model created by the prediction model creation unit, a model determination unit that determines a change in the monitoring target included in the image data or the point cloud data or the other image data, In the second mode, the predetermined form of the monitoring target included in the point cloud data converted by the point cloud forming unit is measured, and the measured value and a preset reference value are measured. And a reference value determination unit that determines a change in the monitored object.

The monitoring system according to the second viewpoint includes a monitoring camera that captures an image of the site and a monitoring device according to the first viewpoint.

The monitoring method according to the third aspect includes a step of converting the moving image data into image data when the shooting data from the surveillance camera that shoots the base is moving image data, and when the shooting data is three-dimensional data. Converting the three-dimensional data into point cloud data; and in the first mode, the converted image data, the converted point cloud data, or the captured data is image data. Machine learning is used to create a prediction model using other image data, and the converted image data or the converted point cloud data in the second mode different from the first mode. Or, the image data is included in the image data, the point cloud data, or the other image data by comparing the other image data when the image data is image data with the created prediction model. A step of determining a change of the monitored object, the predetermined mode of the monitored object included in the converted point cloud data is measured in the second mode, and the measured value is set in advance. Determining a change in the monitored object by comparing the measured reference value with the determined reference value.

The program according to the fourth aspect is a process of converting the moving image data into image data when the image data from the surveillance camera that shoots the base is moving image data, and the process when the image data is three-dimensional data. The process of converting three-dimensional data into point cloud data, and the process of converting the image data converted in the first mode, or the converted point cloud data, or the photographing data being image data Process of creating a prediction model by machine learning using the image data of, and in the second mode different from the first mode, the converted image data, or the converted point cloud data, Alternatively, the image data is included in the image data, the point cloud data, or the other image data by comparing the other image data when the captured data is image data with the created prediction model. The process of determining the change of the monitoring target, and measuring the predetermined form of the monitoring target included in the converted point cloud data in the second mode, and measuring the measured value, preset The hardware resource is caused to execute a process of determining a change in the monitored object by comparing the measured value with the reference value.

The program can be recorded in a computer-readable storage medium. The storage medium may be a non-transient one such as a semiconductor memory, a hard disk, a magnetic recording medium, an optical recording medium, or the like. Further, the present disclosure may be embodied as a computer program product.

According to the first to fourth aspects, it is possible to contribute to monitoring various situations at the site.

1 is a block diagram schematically showing a configuration of a monitoring system according to a first exemplary embodiment. 1 is a block diagram schematically showing a detailed configuration of a monitoring device in the monitoring system according to the first exemplary embodiment. FIG. 3 is a diagram schematically showing a processing path of moving image data in a learning mode of the monitoring device in the monitoring system according to the first embodiment. FIG. 3 is a diagram schematically showing a processing path of image data in a learning mode of the monitoring device in the monitoring system according to the first embodiment. FIG. 3 is a diagram schematically showing a processing path of three-dimensional data in a learning mode of the monitoring device in the monitoring system according to the first exemplary embodiment. FIG. 3 is a diagram schematically showing a processing path of moving image data in an operation mode of the monitoring device in the monitoring system according to the first embodiment. FIG. 3 is a diagram schematically illustrating a processing path of image data in an operation mode of a monitoring device in the monitoring system according to the first exemplary embodiment. FIG. 3 is a diagram schematically showing a processing path of three-dimensional data in the operation mode of the monitoring device in the monitoring system according to the first embodiment. 3 is a flow chart diagram schematically showing the operation of the imaging unit of the monitoring device in the monitoring system according to the first embodiment. FIG. FIG. 3 is a flow chart diagram schematically showing the operation of the point cloud unit of the monitoring device in the monitoring system according to the first exemplary embodiment. FIG. 3 is a flowchart diagram schematically showing the operation of a model determination unit of the monitoring device in the monitoring system according to the first exemplary embodiment. FIG. 3 is a flow chart diagram schematically showing the operation of the reference value determination unit of the monitoring device in the monitoring system according to the first embodiment. It is a block diagram which showed typically the structure of the monitoring system which concerns on Embodiment 2. 9 is a block diagram schematically showing the configuration of a monitoring device according to a third embodiment. FIG. It is the block diagram which showed the structure of the hardware resource typically.

In the present disclosure described below, it is possible to appropriately select and combine the monitoring device according to Mode 1 and its modification mode.

The monitoring device according to the mode 1 includes an imaging unit that converts the moving image data into image data when the shooting data from the monitoring camera that shoots the site is moving image data. The monitoring device includes a point cloud conversion unit that converts the three-dimensional data into point cloud data when the captured data is three-dimensional data. In the monitoring device, in the first mode, the image data converted by the imaging unit, the point group data converted by the point grouping unit, or the imaging data is image data. A prediction model creation unit that creates a prediction model by machine learning using other image data at this time is provided. The monitoring device, in a second mode different from the first mode, the image data converted by the imaging unit, or the point cloud data converted by the point cloud unit, or the By comparing the other image data when the shooting data is image data and the prediction model created by the prediction model creation unit, the image data or the point cloud data or the other image data And a model determination unit that determines a change in the monitored object included in. In the second mode, the monitoring device measures a predetermined form of the monitoring target object included in the point cloud data converted by the point cloud forming unit, and sets a measured value and a preset value. And a reference value determination unit that determines a change in the monitored object by comparing the reference value with the reference value.

As a modification mode of the monitoring device according to the mode 1, a first path control unit that controls an output path of data from the monitoring camera is provided. The monitoring device includes a second route control unit that controls an output route of data from the imaging unit, the point grouping unit, or the first route control unit. The first path control unit outputs the moving image data to the imaging unit when the captured data is moving image data. The first path control unit outputs the three-dimensional data toward the point grouping unit when the captured data is three-dimensional data. The first route control unit outputs the other image data when the captured data is image data to the second route control unit. The second route control unit, in the first mode, the image data from the imaging unit, the point cloud data from the point grouping unit, or the first route control unit. The other image data is output to the model creating unit. The second route control unit outputs the image data from the imaging unit or the other image data from the first route control unit to the model determination unit in the second mode. Alternatively, the point cloud data from the point cloud forming unit is output to the model determining unit and the reference value determining unit. The monitoring device includes a data acquisition unit that acquires the shooting data from the monitoring camera and outputs the shooting data to the first route control unit. The monitoring device includes an output unit that outputs a determination result from at least one of the model determination unit and the reference value determination unit. The output unit outputs the determination result to the data acquisition unit. The data acquisition unit confirms a change in the monitored object based on the determination result from the output unit, and adjusts a time interval for acquiring the imaging data from the monitoring camera according to the degree of the change. The shooting data includes at least one of time information, position information, camera information, and object information. The determination result includes at least one of the time information, the position information, the camera information, and the object information. The imaging unit converts the moving image data into the image data by cutting out the image data from the moving image data at a preset sampling cycle and subjecting the cut out image data to predetermined image processing. The predetermined image processing is at least one of finer definition, higher resolution, noise removal, lower resolution, opening processing, and morphological conversion. The point cloud conversion unit converts the three-dimensional data into primary point cloud data, generates model data based on the converted primary point cloud data, and uses the generated model data as a secondary point cloud. By converting it into data, the three-dimensional data is converted into the point cloud data. The point grouping unit performs a filtering process on the converted point group data. In the monitoring device, in the first mode, the image data converted by the imaging unit, the point group data converted by the point grouping unit, or the imaging data is image data. A label assigning unit is provided which assigns a prediction model label to other image data at a certain time and outputs the data to which the prediction model label is assigned, to the prediction model creating unit.

In the present disclosure, the monitoring system according to the mode 2 and its modification mode can be appropriately selected and combined.

The monitoring system according to the mode 2 includes a monitoring camera that captures an image of a site and a monitoring device according to the mode 1.

As a modified mode of the surveillance system according to the mode 2, the self-driving vehicle is equipped with the surveillance camera, a satellite positioning device, and a communication unit capable of communicating with the surveillance device. The self-driving vehicle includes the positioning information measured by the satellite positioning device in the image data captured by the surveillance camera, and transmits the information to the monitoring device via the communication unit.

In the present disclosure, as a monitoring method according to Mode 3, a step of converting the moving image data into image data when the imaged data from the surveillance camera that shoots the site is moving image data, and the captured data is three-dimensional data. Sometimes the step of converting the three-dimensional data into point cloud data, and in the first mode, the converted image data, or the converted point cloud data or the photographing data is image data. When the second mode different from the first mode and the step of creating a prediction model by machine learning using the other image data of time, the converted image data or the converted point is used. Group data, or by comparing the other image data when the shooting data is image data with the created prediction model, the image data or the point cloud data or the other image data A step of determining a change of the monitoring target included in the second mode, measuring a predetermined form of the monitoring target included in the converted point cloud data in the second mode, and a measured value; Determining a change in the monitored object by comparing it with a preset reference value.

In the present disclosure, as a program related to mode 4, when the shooting data from the surveillance camera that shoots the base is moving image data, a process of converting the moving image data into image data, and when the shooting data is three-dimensional data A process of converting the three-dimensional data into point cloud data, and in the first mode, the converted image data, the converted point cloud data, or the photographing data is image data. Process of creating a prediction model by machine learning using another image data of the above, and the converted image data or the converted point group in the second mode different from the first mode. By comparing the data, or the other image data when the photographing data is image data, and the created prediction model, to the image data or the point cloud data or the other image data. A process of determining a change in the included monitoring target, measuring a predetermined form of the monitoring target included in the converted point cloud data in the second mode, and measuring the measured value in advance; The hardware resource is caused to execute a process of determining a change in the monitored object by comparing the set reference value.

Hereinafter, embodiments will be described with reference to the drawings. Note that, in the present application, when reference numerals are attached to the drawings, they are for the purpose of helping understanding only, and are not intended to be limited to the illustrated modes. The following embodiments are merely examples and do not limit the present invention. Further, the connecting lines between blocks in the drawings and the like referred to in the following description include both bidirectional and unidirectional. The unidirectional arrows schematically show the flow of main signals (data), and do not exclude bidirectionality. Further, although not explicitly shown in the circuit diagram, block diagram, internal configuration diagram, connection diagram, etc. disclosed in the present disclosure, an input port and an output port exist at each of the input end and the output end of each connection line. The input/output interface is similar.

[Embodiment 1]
The monitoring system according to the first embodiment will be described with reference to the drawings. FIG. 1 is a block diagram schematically showing the configuration of the monitoring system according to the first embodiment. FIG. 2 is a block diagram schematically showing a detailed configuration of the monitoring device in the monitoring system according to the first embodiment. 3 to 8 are diagrams schematically showing processing paths of various data of the monitoring device in the monitoring system according to the first embodiment.

The monitoring system 1 is a system for monitoring (inspecting) an object to be monitored (hereinafter may be referred to as “object”) existing in the bases 50a to 50n (eg, raw material yard, warehouse, factory, transfer route, blast furnace, etc.). is there. The monitoring system 1 includes a monitoring device 10, monitoring cameras 30a to 30n, and a network 40.

Here, the monitored object may be, for example, a raw material in a raw material yard or a warehouse, a factory, a blast furnace, or a fallen mine coal in a transfer route.

The monitoring device 10 is a device that monitors the objects existing at the bases 50a to 50n using the monitoring cameras 30a to 30n (see FIG. 1). The monitoring device 10 is communicatively connected to the monitoring cameras 30a to 30n via a network 40. The monitoring device 10 includes a data acquisition unit 11, a route control unit 12, an imaging unit 13, a point grouping unit 14, a route control unit 15, a labeling unit 16, a prediction model creation unit 17, and a model. It has the judgment part 18, the reference value judgment part 19, and the output part 20 (refer FIG. 2). For the monitoring device 10, for example, hardware resources including a processor, a memory, a network interface, etc. (for example, an information processing device, a computer) can be used. In this case, the hardware resource is a virtual data acquisition unit 11, a route control unit 12, an imaging unit 13, a point cloud, by executing the program in a processor while using a memory that stores the program. You may make it implement|achieve the part 14, the path|route control part 15, the label provision part 16, the prediction model preparation part 17, the model determination part 18, the reference value determination part 19, and the output part 20.

The data acquisition unit 11 is a functional unit that acquires imaging data captured by the surveillance camera (30a to 30n in FIG. 1) via the network (40 in FIG. 1) (see FIG. 2). The data acquisition unit 11 is communicatively connected to the surveillance cameras (30a to 30n in FIG. 1) via the network (40 in FIG. 1). The data acquisition unit 11 outputs the acquired shooting data to the route control unit 12 (see FIGS. 3 to 8). The image data acquired by the data acquisition unit 11 is imaged by monitoring cameras of various imaging formats (30a to 30n in FIG. 1; infrared camera, hyperspectral camera, RGB (Red Green Blue) camera, three-dimensional sensor, depth sensor, etc.). The captured image data (moving image data, image data, three-dimensional data, etc.) can be used.

Here, the shooting data is data shot by the surveillance cameras 30a to 30n, and is any of moving image data, image data, and three-dimensional data.

The route control unit 12 is a functional unit (first route control unit) that controls the output route of the imaging data acquired by the data acquisition unit 11 (see FIG. 2 ). The route control unit 12 determines the type of moving image data (moving image data, image data, 3D data). When the captured data is moving image data (moving image data captured by an infrared camera, a hyperspectral camera, an RGB camera, etc.), the path control unit 12 outputs the moving image data to the imaging unit 13 (FIG. 3, FIG. (See FIG. 6). When the captured data is image data (still image data captured by an infrared camera, a hyperspectral camera, an RGB camera, etc.), the route control unit 12 outputs the image data to the route control unit 15 (FIG. 4). , See FIG. 7). When the captured data is three-dimensional data (for example, three-dimensional data captured by a three-dimensional camera, a depth sensor, etc.), the path control unit 12 outputs the three-dimensional data to the point grouping unit 14 (Fig. 5, see FIG. 8).

The imaging unit 13 is a functional unit that converts the moving image data from the route control unit 12 into image data (see FIG. 2). The imaging unit 13 cuts out image data from the moving image data at a preset sampling cycle, and performs predetermined image processing on the cut-out image data (for example, finer definition, higher resolution, noise removal, lower resolution, opening processing). , Morphological conversion, etc.) to convert the moving image data into image data. The image processing can be omitted if necessary. The imaging unit 13 outputs the converted image data to the route control unit 15 (see FIGS. 3 and 6).

The point grouping unit 14 is a functional unit that converts the three-dimensional data from the route control unit 12 into point group data (see FIG. 2). The point cloud conversion unit 14 converts three-dimensional data into primary point cloud data, generates model data (for example, CAD (Computer Aided Design) data) based on the converted primary point cloud data, and is generated. The three-dimensional data is converted into the point cloud data by converting the model data into the secondary point cloud data. When converted into point cloud data, the point cloud conversion unit 14 may perform filter processing on the point cloud data as necessary to remove the outlying point cloud or perform downsampling. You may make it remove the point cloud other than the part which machine-learns in the preparation part 17. The generation of model data and the conversion of model data into point cloud data can be omitted if necessary. The point cloud forming unit 14 outputs the converted point cloud data (secondary point cloud data when model data is generated) to the route control unit 15 (see FIGS. 5 and 8). Here, the point cloud conversion unit 14 converts the three-dimensional data into the point cloud data because the format of the three-dimensional data captured by the three-dimensional camera or the depth sensor is not unique to each manufacturer in many cases. This is because the uniform processing is difficult to perform, but the format of the point cloud data is unified, and uniform processing is easy to perform.

The route control unit 15 controls the output route of the image data from the imaging unit 13, the image data from the route control unit 12, and the point cloud data from the point cloud unit 14 (second route). Control unit) (see FIG. 2). In the learning mode, the route control unit 15 sends the image data from the imaging unit 13, the image data from the route control unit 12, or the point cloud data from the point cloud unit 14 to the labeling unit 16. Output (see FIGS. 3, 4, and 5). In the operation mode, the route control unit 15 outputs the image data from the imaging unit 13 or the image data from the route control unit 12 to the model determination unit 18 (see FIGS. 6 and 7). In the operation mode, the route control unit 15 outputs the point cloud data from the point cloud forming unit 14 to the model determining unit 18 and the reference value determining unit 19 (see FIG. 8).

Here, the learning mode is a mode (first mode) for machine learning (extracting features such as useful rules, rules, knowledge representations, judgment criteria) in the prediction model creation unit 17. The operation mode is a mode (second mode) for normal operation (judgment processing) in the model judgment unit 18 and the reference value judgment unit 19.

The label assigning unit 16 is a functional unit that assigns a prediction model label to the data (image data, point cloud data) from the route control unit 15 in the learning mode (see FIG. 2 ). The labeling unit 16 outputs the data (labeled data) to which the prediction model label is attached to the prediction model creation unit 17.

The prediction model creation unit 17 is a functional unit that creates a prediction model by machine learning (extracting features such as useful rules, rules, knowledge representations, judgment criteria) using labeled data from the labeling unit 16. .. The prediction model is a model (data) related to an object (object) having a feature (feature of the object to be predicted) extracted by machine learning, and serves as a criterion for the model determination unit 18. The prediction model creation unit 17 outputs the created prediction model to the model determination unit 18.

The model determination unit 18 compares the data (image data, point cloud data) from the route control unit 15 with the corresponding prediction model created by the prediction model creation unit 17 to determine the target included in the data. It is a functional unit that determines a change in an object (see FIGS. 2 and 6 to 8 ). The model determination unit 18 holds the latest prediction model created by the prediction model creation unit 17. In the judgment, temperature (when the data is image data from an infrared camera), component (when the data is image data from a hyperspectral camera), appearance (when the data is image data from an RGB camera), point cloud (data) Is a point cloud data derived from a three-dimensional camera). In the determination of the change of the target object, the determination can be performed using the certainty factor (%) with the target object and the certainty factor (%) with no target object. For example, it is possible to make a final determination that an object has a certainty factor of 85% or more that is determined to have an object. The certainty factor can be tuned according to the situation to be determined. Further, in the determination of the change of the target object, it is possible to determine how much (for example, what percentage) the target object included in the data is changing with respect to the prediction model, and the change is within the upper limit and the lower limit. It can be determined whether or not there is. The model determination unit 18 outputs the result (determination result) related to the determination of the change of the object to the output unit 20. The determination result can include time information, position information, camera information, object information, and the like.

The reference value determination unit 19 presets a measurement value (for example, a measurement value of a shape such as volume, shape size, or area) related to a predetermined shape of the object included in the point cloud data from the route control unit 15. It is a functional unit that determines a change in the target object by comparing the calculated reference value (see FIGS. 2 and 8). The reference value determination unit 19 extracts the point cloud related to the object from the point cloud data from the route control unit 15. The reference value determination unit 19 measures the predetermined form of the target object based on the extracted point group of the target object. In the learning mode, the reference value determination unit 19 holds the measured value of the predetermined form of the measured object as the reference value. The reference value is set in advance according to the predetermined form of the object to be measured, and can be, for example, the volume, the size of the shape, the area, or the like. In the operation mode, the reference value determination unit 19 determines a change in the target object by comparing the measured value of the measured target object in a predetermined form with a preset reference value. In the determination of the change of the object, it is possible to determine how much (for example,%) the measured value of the predetermined form related to the object is changing with respect to the reference value, and the degree of change is the upper limit and the lower limit. It can be determined whether or not it is within the range. The reference value determination unit 19 outputs the result (determination result) related to the determination of the change of the target object to the output unit 20. The determination result can include time information, position information, camera information, object information, and the like.

The output unit 20 is a functional unit that outputs (display, transmission, voice output, print, etc.) the determination result from the model determination unit 18 or the reference value determination unit 19 (see FIG. 2). The output unit 20 outputs the determination result to the data acquisition unit 11, the data acquisition unit 11 confirms the change of the target object based on the determination result, and the monitoring cameras 30a to 30a... You may make it adjust the time interval which acquires the imaging|photography data from 30n. For example, if the change of the object is large, the time interval for acquiring the shooting data can be shortened, and if the change of the object is small, the time interval for acquiring the shooting data can be lengthened. By shortening the time interval for capturing image data, changes in the target object can be grasped quickly and accurately, and by increasing the time interval for capturing image data, the power consumption of the system can be suppressed. .. As the output unit 20, for example, a display for displaying, an interface for transferring data, a speaker for outputting voice, a printer for printing, or other output means can be used. Wired or via a communication unit and a network (not shown). You may use the information terminal etc. which have an output part connected so that communication by wireless may be possible.

The monitoring cameras 30a to 30n are cameras (including sensors) that capture a monitoring target (target) (see FIG. 1). The monitoring cameras 30a to 30n generate shooting data (any of moving image data, image data, and three-dimensional data) related to a shot target object, and generate the shooting data in the data acquisition unit (FIG. 2 to FIG. 2) of the monitoring device 10. It outputs to 11) of FIG. The monitoring cameras 30a to 30n may output the shooting data according to the time intervals at which the data acquisition unit (11 in FIGS. 2 to 8) acquires the shooting data. Various cameras can be used as the monitoring cameras 30a to 30n. The shooting data can include time information, position information, camera information, object information, and the like.

The surveillance cameras 30a to 30n are installed at a number of bases 50a to 50n. Various cameras are selected as the monitoring cameras 30a to 30n according to the customer's request. The monitoring cameras 30a to 30n may be installed not only one for each of the bases 50a to 50n but also two or more, and a plurality of cameras of different types may be installed in one base. When monitoring the object to the customer, confirm the request for what purpose and what range to monitor. As a result, it is possible to flexibly combine the service development according to the customer's request.

An infrared camera, a hyperspectral camera, an RGB camera, a three-dimensional camera, a depth sensor, or the like can be used as the monitoring cameras 30a to 30n.

For example, when a fallen mine coal is used as an object, since the fallen mine coal may be ignited, infrared cameras capable of detecting temperature can be used as the monitoring cameras 30a to 30n. According to the infrared camera, it is possible to extract the temperature change of the fallen mine coal from the temperature information. As a result, it is possible to make a determination at a temperature invisible to the human eye.

In addition, since not only lignite and lignite but also deposits of soil and other components are present on the site of the factory, the monitoring cameras 30a to 30n are hyperspectral cameras capable of detecting the components. Can be used. With the hyperspectral camera, it is possible to extract the components of the fallen mine coal from the wavelength information. As a result, it is possible to make a determination with a component that is not visible to the human eye.

Further, when detecting the "visual appearance" of the object, RGB cameras can be used as the monitoring cameras 30a to 30n. According to the RGB camera, it is possible to extract fallen mine coal based on the appearance texture information and the like. As a result, it is possible to make the same determination as the human visual determination.

Further, when detecting the size and volume of the shape of the object, a three-dimensional camera or a depth sensor can be used as the monitoring cameras 30a to 30n. If a 3D camera or depth sensor is used, it is possible to detect the exact size and volume of irregularly shaped objects (raw materials, fallen coal) using 3D data that cannot be measured by the human eye. It is possible to manage the stock of raw materials and automatically place an order for the shortage of raw materials. Also, it is possible to manage the deposition of lignite lignite and narrow down the place and timing for cleaning. There are many types of 3D cameras, such as 3D-LiDAR (Light Detection And Ranging), ToF (Time of Flight) cameras, and stereo cameras, depending on indoor/outdoor shooting, shooting distance, and shooting system. Although the format of the three-dimensional data is different, uniform processing can be performed by the point grouping unit 14, so any type of three-dimensional camera can be selected according to the customer's application.

If only one type of surveillance camera 30a to 30n is installed in one base 50a to 50n, it may be difficult to determine the change of the object, but one base 50a to 50n has a plurality of types of surveillance cameras 30a. It is possible to effectively determine the change of the target object by installing ˜30n and obtaining a plurality of types of determination results to perform fusion.

The network 40 is an information communication network that connects the monitoring device 10 and the monitoring cameras 30a to 30n so that they can communicate with each other in a wired or wireless manner (see FIG. 1). The network 40 includes, for example, a LAN (Local Area Network), a PAN (Personal Area Network), a CAN (Campus Area Network), a MAN (Metropolitan Area Network), a WAN (Wide Area Network), and a GAN (Global Area Network). Can be used.

Next, the operation of the imaging unit of the surveillance device in the surveillance system according to the first embodiment will be described with reference to the drawings. FIG. 9 is a flowchart diagram schematically showing the operation of the imaging unit of the monitoring device in the monitoring system according to the first embodiment. Please refer to FIG. 1 and FIG. 2 for the components of the monitoring system 1 and the monitoring device 10.

First, the imaging unit 13 acquires moving image data (moving image data captured by the monitoring cameras 30a to 30n (infrared cameras, hyperspectral cameras, RGB cameras, etc.)) from the route control unit 12 (step A1).

Next, the imaging unit 13 cuts out image data from the moving image data acquired in step A1 at a preset sampling cycle (step A2).

Next, the imaging unit 13 refines the image data cut out in step A2 (for example, increases resolution and removes noise) (step A3).

Finally, the imaging unit 13 outputs the image data refined in step A3 to the route control unit 15 (step A4), and then ends.

Next, the operation of the point grouping unit of the monitoring device in the monitoring system according to the first embodiment will be described with reference to the drawings. FIG. 10 is a flowchart diagram schematically showing the operation of the point grouping unit of the monitoring device in the monitoring system according to the first embodiment. Please refer to FIG. 1 and FIG. 2 for the components of the monitoring system 1 and the monitoring device 10.

First, the point grouping unit 14 acquires the three-dimensional data from the route control unit 12 (step B1).

Next, the point cloud forming unit 14 converts the three-dimensional data acquired in step B1 into primary point cloud data (step B2).

Next, the point cloud formation part 14 produces|generates model data (for example, CAD data) based on the primary point cloud data converted by step B2 (step B3).

Next, the point cloud forming unit 14 converts the model data generated in step B3 into secondary point cloud data (step B4).

Finally, the point cloud forming unit 14 outputs the secondary point cloud data converted in step B4 to the route control unit 15 (step B5), and thereafter ends.

Next, the operation of the model determination unit of the monitoring device in the monitoring system according to the first embodiment will be described with reference to the drawings. FIG. 11 is a flow chart diagram schematically showing the operation of the model determination unit of the monitoring device in the monitoring system according to the first embodiment. Please refer to FIG. 1 and FIG. 2 for the components of the monitoring system 1 and the monitoring device 10. Here, it is assumed that the prediction model creation unit 17 creates a prediction model in advance, and the created prediction model is held in the model determination unit 18.

First, the model determination unit 18 acquires data (image data, point cloud data (secondary point cloud data when model data is generated)) from the route control unit 15 (step C1).

Next, the model determination unit 18 compares the data acquired in step C1 with the corresponding prediction model created by the prediction model creation unit 17 (step C2).

Next, the model determination unit 18 determines the change in the object included in the data acquired in step C1 by the comparison in step C2 (step C3).

Finally, the model determination unit 18 outputs the result (determination result) related to the determination in step C3 to the output unit 20 (step C4), and then ends.

Next, the operation of the reference value determination unit of the monitoring device in the monitoring system according to the first embodiment will be described with reference to the drawings. FIG. 12 is a flowchart diagram schematically showing the operation of the reference value determination unit of the monitoring device in the monitoring system according to the first embodiment. Please refer to FIG. 1 and FIG. 2 for the components of the monitoring system 1 and the monitoring device 10.

First, the reference value determination unit 19 acquires the point cloud data from the route control unit 15 (step D1).

Next, the reference value determination unit 19 extracts the point cloud relating to the object from the point cloud data acquired in step D1 (step D2).

Next, the reference value determination unit 19 measures the predetermined form of the target object based on the point cloud related to the target object extracted in step D2 (step D3).

Next, the reference value determination unit 19 sets a predetermined measurement value (for example, a measurement value of a shape such as a volume or a shape size) of the object measured in step D3, and a preset reference value. , Are compared (step D4).

Next, the reference value determination unit 19 determines the change in the object included in the point cloud data acquired in step D1 by the comparison in step D4 (step D5).

Finally, the reference value determination unit 19 outputs the result (determination result) regarding the determination of the change of the target object in step D5 to the output unit 20 (step D6), and then ends.

The monitoring system 1 as described above is used for the management of raw materials in the field of smart factories, the management of fallen coal and coal in the field of mining, the management of raw materials in the field of food manufacturing, the management of waste in the field of waste treatment, and various types of management. It can be used for managing toilet cleaning at the facility.

According to the first embodiment, the following effects are achieved.

The first effect is that it is possible to select the most suitable type of monitoring cameras 30a to 30n according to the customer's request, which can contribute to monitoring various situations in the field.

The second effect is that it is possible to detect the shape, size, volume, temperature, component, etc. of the monitored object from the captured image data, so that the operator can also monitor the elements that cannot be discriminated from the appearance alone. can do.

The third effect is that by installing the monitoring cameras 30a to 30n at the place to be monitored, the worker can monitor the shape, size, volume, temperature, composition, etc. of the monitored object at a remote place without going to the site. It is possible to improve the efficiency of the monitoring work.

A fourth effect is that it is possible to detect an abnormality of the monitored object by setting the reference value according to the detected content of the monitored object.

The fifth effect is that various monitoring cameras 30a to 30n are fused and utilized in one monitoring system 1 to improve the quality of the monitoring operation itself.

The sixth effect is that not only is the monitoring constantly performed, but also the time interval for monitoring can be adjusted according to the change in the status of the monitored object, so the power consumption of the monitoring system 1 is suppressed. be able to.

The seventh effect is that by utilizing a three-dimensional camera as the monitoring cameras 30a to 30n, the position can be acquired, the place to be cleaned and the object to be ordered can be narrowed down, and a wide range of inspection and monitoring can be performed. Can be realized.

The eighth effect is that in inventory management of raw materials, it is possible to automatically place an order for a shortage of objects or manage a stable state after inventory replenishment as a life cycle. In addition, the timing of cleaning can be notified in the management of sedimentation of fallen coal.

[Embodiment 2]
A monitoring system according to the second embodiment will be described with reference to the drawings. FIG. 13 is a block diagram schematically showing the configuration of the monitoring system according to the second embodiment.

The second embodiment is a modified example of the first embodiment, in which the monitoring cameras are not installed for each of the bases 50a to 50n, and the monitoring cameras 30 (one monitoring camera, a plurality of types of monitoring cameras may be used) and the communication unit 80. Using the self-driving vehicle 60 equipped with the (wireless communication unit), the monitoring apparatus 10 can be acquired through the communication unit 80 and the network 40 by patrolling the bases 50a to 50n where the monitored object exists. It was done like this. The autonomous driving vehicle 60 is equipped with a satellite positioning device 70 (for example, GPS; Global Positioning System). The positioning information measured by the satellite positioning device 70 is included in the captured data and transmitted to the monitoring device 10 via the communication unit 80 and the network 40. The monitoring device 10 can specify the shooting position of the shooting data based on the positioning information. Other configurations are similar to those of the first embodiment.

According to the second embodiment, similarly to the first embodiment, it is possible to contribute to monitoring various situations at the site.

[Third Embodiment]
A monitoring device according to the third embodiment will be described with reference to the drawings. FIG. 14 is a block diagram schematically showing the configuration of the monitoring device according to the third embodiment.

The monitoring device 10 is a device that monitors a monitoring target existing in the base 50 using the monitoring camera 30 (see FIG. 14 ). The monitoring device 10 includes an imaging unit 13, a point grouping unit 14, a prediction model creation unit 17, a model determination unit 18, and a reference value determination unit 19.

The imaging unit 13 converts the moving image data into image data when the shooting data from the surveillance camera 30 that shoots the site 50 is moving image data (see FIG. 14 ).

The point cloud forming unit 14 converts the three-dimensional data into point cloud data when the captured data is three-dimensional data (see FIG. 14).

When the prediction model creation unit 17 is in the first mode, when the image data converted by the imaging unit 13 or the point cloud data converted by the point cloud unit 14 or the shooting data is image data Machine learning is used to create a prediction model using the other image data (see FIG. 14).

In the second mode different from the first mode, the model determination unit 18 determines whether the image data converted by the imaging unit 13 or the point group data converted by the point grouping unit 14 or the imaging data is acquired. By comparing the other image data that is image data with the prediction model created by the prediction model creation unit 17, the change of the monitoring target included in the image data, the point cloud data, or the other image data. To judge.

In the reference value determination unit 19 and the second mode, the predetermined form of the monitoring target included in the point cloud data converted by the point cloud formation unit 14 is measured, and the measured value and the preset value are set. The change of the monitored object is determined by comparing with the reference value.

According to the third embodiment, similarly to the first embodiment, it is possible to contribute to monitoring various situations at the site.

The monitoring device according to the first to third embodiments can be configured by so-called hardware resources (information processing device, computer), and the monitoring device having the configuration illustrated in FIG. 15 can be used. For example, the hardware resource 100 includes a processor 101, a memory 102, a network interface 103, etc., which are interconnected by an internal bus 104.

The configuration illustrated in FIG. 15 is not intended to limit the hardware configuration of the hardware resource 100. The hardware resource 100 may include hardware (not shown) (for example, an input/output interface). Alternatively, the number of units such as the processor 101 included in the device is not limited to the example illustrated in FIG. 15, and a plurality of processors 101 may be included in the hardware resource 100, for example. As the processor 101, for example, a CPU (Central Processing Unit), MPU (Micro Processor Unit), GPU (Graphics Processing Unit), or the like can be used.

As the memory 102, for example, RAM (Random Access Memory), ROM (Read Only Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), or the like can be used.

As the network interface 103, for example, a LAN (Local Area Network) card, a network adapter, a network interface card, or the like can be used.

The function of the hardware resource 100 is realized by the processing module described above. The processing module is realized by the processor 101 executing a program stored in the memory 102, for example. In addition, the program can be downloaded via a network or updated using a storage medium storing the program. Further, the processing module may be realized by a semiconductor chip. That is, the function performed by the processing module may be realized by executing software on some hardware.

The whole or part of the exemplary embodiments disclosed above can be described as, but not limited to, the following supplementary notes.

[Appendix 1]
In the present invention, the form of the monitoring device according to the first aspect is possible, and is as follows.
An imaging unit that converts the moving image data into image data when the shooting data from the surveillance camera that shoots the base is moving image data,
A point cloud conversion unit that converts the three-dimensional data into point cloud data when the captured data is three-dimensional data;
In the first mode, the image data converted by the imaging unit, the point group data converted by the point grouping unit, or another image when the shooting data is image data A predictive model creation unit that creates a predictive model by machine learning using data,
In the second mode different from the first mode, the image data converted by the imaging unit, the point group data converted by the point grouping unit, or the photographing data is image data. When the other image data and the prediction model created by the prediction model creation unit are compared, a monitoring target included in the image data, the point cloud data, or the other image data. A model determination unit that determines changes in objects,
In the second mode, the predetermined form of the monitoring target included in the point cloud data converted by the point cloud forming unit is measured, and the measured value and a preset reference value are measured. By comparing the reference value determination unit for determining the change of the monitored object,
With
Monitoring equipment.

[Appendix 2]
A first path control unit for controlling an output path of data from the surveillance camera;
A second route control unit that controls an output route of data from the imaging unit, the point grouping unit, or the first route control unit;
Further equipped with,
The first route control unit,
When the shooting data is moving image data, the moving image data is output toward the imaging unit,
When the photographed data is three-dimensional data, the three-dimensional data is output toward the point grouping unit,
Outputting the other image data when the photographing data is image data, toward the second path control unit,
The second route control unit,
In the first mode, the image data from the imaging unit, the point cloud data from the point clouding unit, or the other image data from the first path control unit is used as the model. Output to the creation section,
In the second mode, the image data from the imaging unit or the other image data from the first path control unit is output toward the model determination unit, or the point cloud formation is performed. Outputting the point cloud data from the unit toward the model determination unit and the reference value determination unit,
The monitoring device according to attachment 1.

[Appendix 3]
A data acquisition unit that acquires the shooting data from the surveillance camera and outputs the shooting data to the first route control unit;
An output unit that outputs a determination result from at least one of the model determination unit and the reference value determination unit;
Further equipped with,
The output unit outputs the determination result toward the data acquisition unit,
The data acquisition unit confirms a change in the monitoring target object based on the determination result from the output unit, and adjusts a time interval for acquiring the imaging data from the monitoring camera according to the degree of the change,
The monitoring device according to attachment 2.

[Appendix 4]
The photographing data includes at least one of time information, position information, camera information, and object information,
The determination result includes at least one of the time information, the position information, the camera information, and the object information,
The monitoring device according to attachment 3.

[Appendix 5]
The imaging unit converts the moving image data into the image data by cutting out the image data from the moving image data at a preset sampling cycle, and subjecting the cut out image data to predetermined image processing,
The monitoring device according to any one of appendices 1 to 4.

[Appendix 6]
The predetermined image processing is at least one of finer definition, higher resolution, noise removal, lower resolution, opening processing, and morphology conversion.
The monitoring device according to attachment 5.

[Appendix 7]
The point cloud conversion unit converts the three-dimensional data into primary point cloud data, generates model data based on the converted primary point cloud data, and uses the generated model data as a secondary point cloud. Converting the three-dimensional data into the point cloud data by converting into data
7. The monitoring device according to any one of appendices 1 to 6.

[Appendix 8]
The point grouping unit performs a filter process on the converted point group data,
The monitoring device according to attachment 7.

[Appendix 9]
In the first mode, when the image data converted by the imaging unit, the point cloud data converted by the point cloud unit, or the shooting data is image data, A label for a prediction model is attached to the image data, and the label attachment unit that outputs the data attached with the label for the prediction model toward the prediction model creation unit is further provided.
7. The monitoring device according to any one of appendices 1 to 6.

[Appendix 10]
In the present invention, the form of the monitoring system according to the second aspect is possible, and is as follows.
A surveillance camera that shoots the base,
A monitoring device according to any one of appendices 1 to 9;
With
Monitoring system.

[Appendix 11]
In addition to mounting the surveillance camera, a satellite positioning device is mounted, and further comprising an autonomous vehicle having a communication unit capable of communicating with the monitoring device,
The self-driving vehicle includes the positioning information measured by the satellite positioning device in the captured data captured by the surveillance camera, and transmits the information to the monitoring device via the communication unit.
The monitoring system according to attachment 10.

[Appendix 12]
In the present invention, the form of the monitoring method according to the third aspect is possible, and is as follows.
A step of converting the moving image data into image data when the shooting data from the surveillance camera shooting the base is moving image data,
Converting the three-dimensional data into point cloud data when the imaging data is three-dimensional data;
In the first mode, machine learning is performed to perform a prediction model using the converted image data, the converted point cloud data, or other image data when the imaging data is image data. To create
In the second mode different from the first mode, the converted image data, or the converted point cloud data, or the other image data when the shooting data is image data, By comparing the created prediction model, by determining the change of the monitoring target included in the image data or the point cloud data or the other image data,
In the second mode, by measuring a predetermined form of the monitored object included in the converted point cloud data, by comparing the measured value and a preset reference value, Determining a change in the monitored object,
including,
Monitoring method.

[Appendix 13]
In the present invention, the form of the program according to the fourth aspect is possible, and is as follows.
A process of converting the video data into image data when the shooting data from the surveillance camera shooting the base is video data;
A process of converting the three-dimensional data into point cloud data when the photographing data is three-dimensional data;
In the first mode, machine learning is performed to perform a prediction model using the converted image data, the converted point cloud data, or other image data when the imaging data is image data. And the process of creating
In the second mode different from the first mode, the converted image data, or the converted point cloud data, or the other image data when the shooting data is image data, By comparing the created prediction model, the process of determining the change of the monitoring target included in the image data or the point cloud data or the other image data,
In the second mode, by measuring a predetermined form of the monitored object included in the converted point cloud data, by comparing the measured value and a preset reference value, A process of determining a change in the monitored object,
Causes a hardware resource to execute,
program.

The disclosures of the above patent documents are incorporated herein by reference. Modifications and adjustments of the exemplary embodiments and examples are possible within the scope of the entire disclosure (including claims and drawings) of the present invention and based on the basic technical concept of the invention. In addition, various combinations or selections of various disclosed elements (including each element of each claim, each element of each embodiment or example, each element of each drawing, etc.) within the scope of the entire disclosure of the present invention (necessary) Can be selected). That is, it goes without saying that the present invention includes various variations and modifications that can be made by those skilled in the art according to the entire disclosure including the claims and the drawings and the technical idea. Regarding the numerical values and numerical ranges described in the present application, it is considered that arbitrary intermediate values, lower numerical values, and small ranges are described even if not explicitly stated.

DESCRIPTION OF SYMBOLS 1 monitoring system 10 monitoring device 11 data acquisition unit 12 route control unit 13 imaging unit 14 point grouping unit 15 route control unit 16 label assigning unit 17 prediction model creating unit 18 model determining unit 19 reference value determining unit 20 output unit 30, 30a to 30n Surveillance camera 40 Network 50, 50a to 50n Location 60 Self-driving vehicle 70 Satellite positioning device 80 Communication unit 100 Hardware resource 101 Processor 102 Memory 103 Network interface 104 Internal bus

Claims (10)

An imaging unit that converts the moving image data into image data when the shooting data from the surveillance camera that shoots the base is moving image data,
A point cloud conversion unit that converts the three-dimensional data into point cloud data when the captured data is three-dimensional data;
In the first mode, the image data converted by the imaging unit, the point group data converted by the point grouping unit, or another image when the shooting data is image data A predictive model creation unit that creates a predictive model by machine learning using data,
In the second mode different from the first mode, the image data converted by the imaging unit, the point group data converted by the point grouping unit, or the photographing data is image data. When the other image data and the prediction model created by the prediction model creation unit are compared, a monitoring target included in the image data, the point cloud data, or the other image data. A model determination unit that determines changes in objects,
In the second mode, the predetermined form of the monitoring target included in the point cloud data converted by the point cloud forming unit is measured, and the measured value and a preset reference value are measured. By comparing the reference value determination unit for determining the change of the monitored object,
With
Monitoring equipment. A first path control unit for controlling an output path of data from the surveillance camera;
A second route control unit that controls an output route of data from the imaging unit, the point grouping unit, or the first route control unit;
Further equipped with,
The first route control unit,
When the shooting data is moving image data, the moving image data is output toward the imaging unit,
When the photographed data is three-dimensional data, the three-dimensional data is output toward the point grouping unit,
Outputting the other image data when the photographing data is image data, toward the second path control unit,
The second route control unit,
In the first mode, the image data from the imaging unit, the point cloud data from the point cloud unit, or the other image data from the first route control unit is predicted. Output to the model creation section,
In the second mode, the image data from the imaging unit or the other image data from the first path control unit is output toward the model determination unit, or the point cloud formation is performed. Outputting the point cloud data from the unit toward the model determination unit and the reference value determination unit,
The monitoring device according to claim 1. A data acquisition unit that acquires the shooting data from the surveillance camera and outputs the shooting data to the first route control unit;
An output unit that outputs a determination result from at least one of the model determination unit and the reference value determination unit;
Further equipped with,
The output unit outputs the determination result toward the data acquisition unit,
The data acquisition unit confirms a change in the monitoring target object based on the determination result from the output unit, and adjusts a time interval for acquiring the imaging data from the monitoring camera according to the degree of the change,
The monitoring device according to claim 2. The imaging unit converts the moving image data into the image data by cutting out the image data from the moving image data at a preset sampling cycle, and subjecting the cut out image data to predetermined image processing,
The monitoring device according to claim 1. The point cloud conversion unit converts the three-dimensional data into primary point cloud data, generates model data based on the converted primary point cloud data, and uses the generated model data as a secondary point cloud. Converting the three-dimensional data into the point cloud data by converting into data
The monitoring device according to claim 1. In the first mode, when the image data converted by the imaging unit, the point cloud data converted by the point cloud unit, or the shooting data is image data, A label for a prediction model is attached to the image data, and the label attachment unit that outputs the data attached with the label for the prediction model toward the prediction model creation unit is further provided.
The monitoring device according to claim 1. A surveillance camera that shoots the base,
A monitoring device according to any one of claims 1 to 6,
With
Monitoring system. In addition to mounting the surveillance camera, a satellite positioning device is mounted, and further comprising an autonomous vehicle having a communication unit capable of communicating with the monitoring device,
The self-driving vehicle includes the positioning information measured by the satellite positioning device in the captured data captured by the surveillance camera, and transmits the information to the monitoring device via the communication unit.
The monitoring system according to claim 7. A step of converting the moving image data into image data when the shooting data from the surveillance camera shooting the base is moving image data,
Converting the three-dimensional data into point cloud data when the imaging data is three-dimensional data;
In the first mode, machine learning is performed to perform a prediction model using the converted image data, the converted point cloud data, or other image data when the imaging data is image data. To create
In the second mode different from the first mode, the converted image data, or the converted point cloud data, or the other image data when the shooting data is image data, By comparing the created prediction model, by determining the change of the monitoring target included in the image data or the point cloud data or the other image data,
In the second mode, by measuring a predetermined form of the monitored object included in the converted point cloud data, by comparing the measured value and a preset reference value, Determining a change in the monitored object,
including,
Monitoring method. A process of converting the video data into image data when the shooting data from the surveillance camera shooting the base is video data;
A process of converting the three-dimensional data into point cloud data when the photographing data is three-dimensional data;
In the first mode, machine learning is performed to perform a prediction model using the converted image data, the converted point cloud data, or other image data when the imaging data is image data. And the process of creating
In the second mode different from the first mode, the converted image data, or the converted point cloud data, or the other image data when the shooting data is image data, By comparing the created prediction model, the process of determining the change of the monitoring target included in the image data or the point cloud data or the other image data,
In the second mode, by measuring a predetermined form of the monitored object included in the converted point cloud data, by comparing the measured value and a preset reference value, A process of determining a change in the monitored object,
Causes a hardware resource to execute,
program.

Images