JP2020021111A - Type determination program, type determination apparatus and type determination method - Google Patents

Abstract

To rapidly and correctly determine a type of an object when the number of pixels related the object included in an input image is small.SOLUTION: A type determination program detects, when a plurality of input images A including an object A2 is input, an object A1 included in each of the input images A (S11), and acquires movement history information B of the object A1 (S12). Then, if the movement history information B is input in a trained model 50 (S13), a probability for a type of object is output from the trained model 50 (S14). The trained model 50 is a trained model in which a deep learning is performed by applying the movement history information of the object constituting teacher data and a type of the object to a predetermined multilayer neural network CNN). The type determination program determines the type of the object A1 included in the input image A based on the probability for the type of object (S15).SELECTED DRAWING: Figure 1

Description

  The present invention relates to a type determination program, a type determination device, and a type determination method that can quickly and accurately determine the type of an object when the number of pixels related to the object included in an input image is small.

  2. Description of the Related Art Conventionally, on a test course of a vehicle, it is necessary to detect an animal or the like that intrudes into the test course over an outer fence. This is to prevent a trouble that an animal or the like entering the test course collides with a test vehicle running on the test course. For this reason, a plurality of surveillance cameras for monitoring the inside of the test course are provided on the test course of the vehicle. Then, the observer visually recognizes the video imaged by the monitoring camera and detects an abnormality.

  However, if such a visual check by a supervisor is adopted, there is a possibility that the supervisor may omit monitoring. For this reason, when monitoring is automated, animals are learned using image processing techniques such as template matching and machine learning such as deep learning (see Patent Literature 1 and Patent Literature 2). It is determined whether or not it is included.

JP 2015-210824 A JP 2017-187954 A

  However, in order to detect the presence of an animal by the above-described template matching, the image of the animal must be sufficiently included in the input image. Similarly, when detecting the presence of an animal by using machine learning, the image of the animal must be sufficiently included in the input image.

  However, in many cases, the actual image captured by the surveillance camera does not sufficiently include the figure of the animal. This is because the number of surveillance cameras that can be installed on the test course is limited, and in many cases, the test course must be photographed from a long distance. As a result, when the number of pixels of the animal included in the input image is small, there is a problem that the animal cannot be detected even if template matching or machine learning is used.

  For these reasons, when the number of pixels related to an object included in an input image is small, how to quickly and accurately determine the type of the object has become an important issue. This problem is not only a problem on the test course, but also occurs in a security area or various facilities.

  The present invention has been made to solve the above-mentioned problems (problems) of the related art, and when the number of pixels related to an object included in an input image is small, the type of the object is quickly and accurately determined. It is an object of the present invention to provide a type determination program, a type determination device, and a type determination method that can perform the determination.

  In order to solve the above-described problems, the present invention provides a detection procedure for detecting an object included in an image captured by a predetermined imaging device, and tracking of the object detected by the detection procedure to record movement history information of the object. And a type determination step of determining a type of the object detected by the detection procedure based on at least the movement history information.

  Also, the present invention is characterized in that, in the above invention, the detecting step detects an object included in an image captured by the predetermined imaging device based on a movement vector of pixels between a plurality of images. I do.

  Further, in the present invention according to the above invention, the moving history information acquiring step specifies a rectangular area including the object based on a moving vector of a pixel forming the object detected by the detecting step and a predetermined likelihood. Then, information relating to the movement history of the specified rectangular area is obtained as movement history information.

  In addition, in the present invention according to the above invention, the type determination step is performed based on the movement history information of an object forming teacher data and a learned model in which the type of the object is applied to a predetermined multilayer neural network. The type of the object detected is determined.

  Further, in the present invention according to the above invention, the type determination step includes, in the learned model, a horizontal movement amount and a vertical movement amount of the rectangular area included in the movement history information for each predetermined time. The type of the object detected by the detection procedure is determined based on the probability of each type input and output from the learned model.

  In addition, in the present invention according to the above invention, the type determination step includes a step of moving the rectangular area in a horizontal direction, the width of the rectangular area, the height of the rectangular area, and the movement of the rectangular area at predetermined time intervals included in the movement history information. The amount, the vertical movement amount of the rectangular area, the width difference from the previous rectangular area, and the height difference from the previous rectangular area are input to the trained model, and the probability for each type output from the trained model is calculated. And determining the type of the object detected by the detection procedure based on the detection procedure.

  In addition, in the present invention according to the above invention, the type determination step includes a step of moving the rectangular area in a horizontal direction, the width of the rectangular area, the height of the rectangular area, and the movement of the rectangular area at predetermined time intervals included in the movement history information. Input the amount, the vertical movement amount of the rectangular area, the width difference from the previous rectangular area, the height difference from the previous rectangular area, and the partial image of the rectangular area to the trained model, And determining the type of the object detected by the detection procedure on the basis of the probability of each type output from.

  Further, in the present invention according to the above invention, the type determination step includes a parameter image obtained by imaging a horizontal movement amount and a vertical movement amount of the rectangular area for each predetermined time included in the movement history information. Is input to the learned model, and the type of the object detected by the detection procedure is determined based on the probability of each type output from the learned model.

  Further, in the present invention according to the above invention, the type determination step includes a parameter image obtained by imaging a horizontal movement amount and a vertical movement amount of the rectangular area for each predetermined time included in the movement history information. And a partial image of the rectangular area are input to the learned model, and the type of the object detected by the detection procedure is determined based on the probability of each type output from the learned model. And

  Further, in the present invention according to the above invention, the learned model outputs a probability that the object is an animal, a person, or a vehicle when at least movement history information related to an animal, a person, or a vehicle is input. It is characterized by.

  Further, according to the present invention, in the above invention, if the type of the object is determined to be an animal by the type determination procedure, control is performed so as to issue a warning by a predetermined warning light, and the type of the object is human or If it is determined that the vehicle is a vehicle, the computer is further caused to execute an alarm control procedure for controlling not to issue an alarm by the predetermined alarm light.

  Further, the present invention provides a detection unit for detecting an object included in an image captured by a predetermined imaging device, and movement history information for tracking the object detected by the detection unit and acquiring the movement history information of the object. An acquisition unit and a type determination unit that determines the type of the object detected by the detection unit based on at least the movement history information.

  Further, the present invention provides a detecting step of detecting an object included in an image captured by a predetermined imaging device, and moving history information of tracking the object detected in the detecting step and acquiring moving history information of the object. The method includes an acquisition step and a type determination step of determining a type of the object detected in the detection step based on at least the movement history information.

  According to the present invention, it is possible to quickly and accurately determine the type of an object even when the number of pixels related to the object included in the input image is small.

FIG. 1 is an explanatory diagram for explaining an overview of the type determination program according to the first embodiment. FIG. 2 is a diagram illustrating an example of the input image illustrated in FIG. FIG. 3 is a diagram showing a system configuration of an alarm system installed on a test course for performing a running test of a vehicle. FIG. 4 is a functional block diagram showing the configuration of the alarm control device shown in FIG. FIG. 5 is a diagram illustrating an example of type determination using the learned model by the type determination unit illustrated in FIG. 4. FIG. 6 is a diagram illustrating an example of movement history information as learning data. FIG. 7 is a diagram illustrating an example of a layer structure of the learned model illustrated in FIG. FIG. 8 is a flowchart showing a processing procedure of the alarm control device shown in FIG. FIG. 9 is a flowchart showing a processing procedure relating to object detection and acquisition of movement history information shown in step S102 of FIG. FIG. 10 is a diagram illustrating an example of the type determination using the learned model according to the second embodiment. FIG. 11 is a diagram illustrating an example of a layer structure of the learned model illustrated in FIG. FIG. 12 is a diagram illustrating an example of the type determination using the learned model according to the third embodiment. FIG. 13 is an explanatory diagram for explaining the parameter image shown in FIG. FIG. 14 is a diagram illustrating a modification of the type determination using the learned model. FIG. 15 is a diagram illustrating an example of a hardware configuration of the alarm control device illustrated in FIG.

  Hereinafter, a type determination program, a type determination device, and a type determination method according to the present embodiment will be described with reference to the accompanying drawings. In this embodiment, a case where the present invention is applied to a test course for performing a running test of a vehicle will be described. Hereinafter, a description will be given mainly of a case where a learned model of a CNN (Convolutional Neural Network) including a program is used.

<Overview of type determination program>
First, an outline of the type determination program according to the first embodiment will be described. FIG. 1 is an explanatory diagram for explaining an overview of the type determination program according to the first embodiment, and FIG. 2 is a diagram illustrating an example of the input image A illustrated in FIG.

  The type determination program shown in FIG. 1 is a program installed in the alarm control device 40. As will be described in detail later, when the alarm control device 40 receives a plurality of images from the monitoring camera 10, the alarm control device 40 cuts out an area where an object is present from each image to obtain an input image A, which is included in the input image A. This is a device for determining the type of the object A1 to be performed.

  Here, like the object A2 shown in FIG. 2, when the number of pixels forming the object A2 is large and it is clear that the object A2 is a vehicle, existing image processing such as template matching or machine processing The type of the object A2 can be determined using learning. On the other hand, as in the case of the object A1 shown in FIG. 2, when the number of pixels forming the object A1 is small and the type of the object A1 is not clear, existing image processing such as template matching or machine learning is performed. It is difficult to determine the type of the object A2 using the information.

  Therefore, in the type determination program according to the first embodiment, the movement history information B of the object A1 included in each image received from the monitoring camera 10 is input to the learned model 50. Then, the type of the object A1 is determined based on the probability (probability of being an animal, probability of being a vehicle, probability of being a person) for each type of the object A1 output from the learned model 50 and a predetermined threshold. Thus, even when the number of pixels forming the object A1 is small, the type of the object A1 can be determined.

  Next, specific processing contents of the type determination program shown in FIG. 1 will be described. As shown in FIG. 1, when the type determination program detects an object A1 included in each image received from the surveillance camera 10, the partial image including the object A1 is set as an input image A, and An object A1 included in A is detected (S11). Specifically, an object A1 included in each input image A is detected using a movement vector of a pixel between the plurality of input images A.

  Then, the movement history information B of the object A1 is obtained (S12). Specifically, a rectangular area including the object A1 is specified based on the detected movement vector of the pixel forming the object A1 and a predetermined likelihood, and information related to the movement history of the specified rectangular area is referred to as movement history information. Obtained as B. Note that the movement history information B includes the width of the rectangular area, the height of the rectangular area, the amount of horizontal movement of the rectangular area, the amount of vertical movement of the rectangular area, and the width difference from the shaped area at predetermined time intervals. And the height difference from the shape region.

  Thereafter, if the movement history information B is input to the learned model 50 (S13), the probability for each type of object is output from the learned model 50 (S14). The trained model 50 is a trained model in which deep learning has been performed by applying the movement history information of the object forming the teacher data and the type of the object to a predetermined multilayer neural network (CNN). Here, the case where the learned model 50 is a program is shown, but the learned model 50 may be constituted by a logic circuit such as an FPGA (Field-Programmable Gate Array).

  The learned model 50 outputs the probability for each type of object on condition that the movement history information B is input. For example, as shown in FIG. 1, the probability that the type of the object is “vehicle” is “0.1 (10%)”, the probability that the type of the object is “person” is “0.1 (10%)”, The probability that the type of the object is “animal” is “0.8 (80%)”.

  The type determination program determines the type of the object A1 included in the input image A based on the probability of each type of the object (S15). For example, if any one of the probabilities for each type of object exceeds a predetermined threshold (for example, 0.8 (80%)), the type corresponding to the probability of exceeding the threshold is the type of object. judge.

  As described above, the type determination program according to the first embodiment detects the object A1 included in the image captured by the monitoring camera, tracks the detected object A1, and acquires the movement history information B of the object A1. Then, the program causes the computer to execute a process of determining the type of the detected object based on the movement history information B.

  When the type of the object A1 is determined to be “animal”, the type determination program controls to issue an alarm by a predetermined warning light, and the type of the object A1 is “human” or “vehicle”. If it is determined that the above condition is satisfied, control is performed so as not to issue a warning by a predetermined warning light. Thus, when an animal enters the test course, the running test of the test vehicle is stopped, and a collision accident between the test vehicle and the animal can be avoided.

<Configuration of alarm system>
Next, a system configuration of an alarm system installed on a test course T for performing a running test of a vehicle will be described. FIG. 3 is a diagram showing a system configuration of an alarm system installed on a test course T for performing a running test of a vehicle. As shown in the figure, the alarm system has a configuration in which a monitoring camera 10, an alarm lamp 20, and a speaker 30 are connected to be able to communicate with an alarm control device 40, respectively. Here, the case where each device is wirelessly connected is shown. The above-described type determination program is installed in the alarm control device 40.

  The monitoring camera 10 is an imaging device that captures an image of the object A1 that enters the test course T from outside the test course T, and is disposed at regular intervals (for example, every several hundred meters) around the outer fence of the test course T. Have been. The monitoring camera 10 transmits an image to the alarm control device 40 via an IP network such as a wireless LAN (Local Area Network). Although the case where an image is transmitted from the monitoring camera 10 to the alarm control device 40 has been described here, a moving image can be transmitted from the monitoring camera 10 to the alarm control device 40. When transmitting a moving image, the alarm control device 40 performs a process of cutting out an image from the moving image for each frame.

  The warning light 20 is a display device for visually notifying a driver who drives the test vehicle and a supervisor who monitors the test course T of the occurrence of an abnormality in the test course T. The warning lights 20 are provided at regular intervals (for example, every several hundred meters) so that the driver of the test vehicle traveling on the test course T can visually recognize the warning lights 20, and the display is controlled by the warning control device 40. For example, the warning light 20 is displayed in green during the running test of the test vehicle, and the warning light 20 is flashed in red if an event to stop the running test occurs. For this reason, for example, if the driver of the test vehicle visually recognizes the flashing of the red warning light 20 while traveling on the road, it is immediately determined that the test is to be stopped and the test vehicle is moved to a predetermined position.

  The speaker 30 is a notification device for acoustically notifying a driver driving the test vehicle, a supervisor monitoring the test course T, and the like of a test situation on the test course T. When an event to stop the running test occurs, a predetermined alarm sound is output according to the instruction of the alarm control device 40. For this reason, for example, if the driver of the test vehicle confirms the warning sound while traveling on the traveling road, it is immediately determined that the test has been stopped and the vehicle moves to a predetermined position.

  The alarm control device 40 is a type of computer, and performs the type determination process according to the present invention by installing the type determination program described with reference to FIG. Specifically, the type of the object A1 included in the image is determined, and when the type of the object A1 is determined to be “animal”, the display control of the warning light 20 and the output control of the warning sound from the speaker 30 are performed. To warn the driver of the test vehicle. When it is determined that the type of the object A1 is “person” or “vehicle”, the display control of the warning light 20 and the output control of the warning sound from the speaker 30 are not performed. The detailed description of the alarm control device 40 will be described later.

<Configuration of alarm control device 40>
Next, the configuration of the alarm control device 40 shown in FIG. 3 will be described. FIG. 4 is a functional block diagram showing the configuration of the alarm control device 40 shown in FIG. As shown in FIG. 1, the alarm control device 40 includes an input unit 41, a display unit 42, a communication interface unit (hereinafter, simply referred to as a “communication I / F unit”) 43, a storage unit 44, and a control unit 45. .

  The input unit 41 is an input device such as a keyboard or a mouse, and the display unit 42 is a display device such as a liquid crystal panel or a display device. The communication I / F unit 43 is an interface unit for communicating with the monitoring camera 10, the alarm lamp 20, and the speaker 30 via a wireless LAN or the like.

  The storage unit 44 is a storage device such as a nonvolatile memory or a hard disk device, and stores the learning data 44a, the movement history information 44b, and the like. The learning data 44a is learning image data used when the CNN performs supervised learning. The movement history information 44b is information indicating the movement history of the object A1.

  The control unit 45 is a control unit that performs overall control of the alarm control device 40, and includes a detection unit 45a, a movement history information acquisition unit 45b, a type determination unit 45c, and an alarm instruction unit 45d. Although a detailed description will be given later, the CPU (Central Processing Unit) of the alarm control device 40 stores the type determination program according to the present invention from a non-volatile memory or the like on a RAM (Random Access Memory) as a main storage device. By loading and executing, processes corresponding to the detection unit 45a, the movement history information acquisition unit 45b, the type determination unit 45c, and the alarm instruction unit 45d are formed.

  The detection unit 45a is a processing unit that detects an object A1 included in an image captured by the monitoring camera 10 based on a movement vector of pixels between images of a plurality of continuous frames. Specifically, if there are moving pixels in a plurality of images, the object A1 is detected based on the moving pixels. Note that the pixel movement vector is a vector (for example, an optical flow) between pixels indicating the same location of the object A1.

  The movement history information acquisition unit 45b specifies a rectangular area including the object A1 based on a movement vector of a pixel forming the object A1 detected by the detection unit 45a and a predetermined likelihood, and specifies a movement history of the specified rectangular area. Is a processing unit that acquires the information according to (1) as movement history information. Specifically, an optical flow and likelihood are calculated for each pixel forming an image, a past object extraction result is moved based on the optical flow to extract a moving pixel, and a rectangular area as a particle is displayed on the image. Particles that move based on the optical flow at the position of the object extraction result within the rectangular area, and are resampled using the likelihood calculated for each rectangular area based on the object extraction results inside and outside the rectangular area The object A1 is detected using the filter, and the object A1 is tracked using the particle filter resampled using the likelihood calculated for each rectangular area based on the color histogram of the position of the object extraction result inside and outside the rectangular area. To obtain the movement history information B. The specific processing is similar to that of Japanese Patent Application No. 2018-046582.

  The type determination unit 45c is a processing unit that determines the type of the object A1 detected by the detection unit 45a, and outputs a probability for each type of the object A1 using the learned model 50 of the CNN. Then, when there is a probability exceeding a predetermined threshold (for example, 80%) among the probabilities for each type of the object A1, it is determined that the type corresponding to the probability is the type of the object A1. As this CNN, “caffe”, “TensorFlow”, “Chainer”, “CNTK”, or the like provided as an open source product can be used.

  The alarm instruction unit 45d is a processing unit that issues an alarm instruction to the alarm lamp 20 and the speaker 30 when the type of the object A1 is determined to be “animal”. When it is determined that the type of the object A1 is “vehicle” or “person”, the warning instruction to the warning light 20 and the speaker 30 is not performed. The warning light 20 previously stores a display mode of the warning light in association with the data pattern of the warning instruction, and performs display in a display mode corresponding to the data pattern included in the warning instruction of the warning instruction unit 45d. Similarly, an output sound pattern is stored in the speaker 30 in advance in association with the data pattern of the warning instruction, and outputs an output sound corresponding to the data pattern included in the warning instruction of the warning instruction section 45d. The speaker 30 can output an output sound included in the warning instruction output from the warning instruction unit 45d.

  Here, the type determination using the learned model 50 of the CNN will be further described. FIG. 5 is a diagram illustrating an example of type determination using the learned model 50 by the type determination unit 45c illustrated in FIG. FIG. 6 is a diagram illustrating an example of movement history information as learning data. FIG. 7 is a diagram illustrating an example of a layer structure of the learned model 50 illustrated in FIG.

  When the CNN performs the supervised learning, the movement history information and the correct answer data as the learning data shown in FIG. 6 are provided to the CNN, and the supervised learning is performed by back propagation or the like. Since back propagation is a well-known technique related to deep learning, its description is omitted here. For example, when the CNN learns “animal”, the movement history information shown in FIG. 6A is input to the CNN, and correct data (P1, P2, P3) = (1, 0, 0) is added. , Backpropagation. When the CNN is to learn "vehicle", the movement history information shown in FIG. 6B is input to the CNN, and correct data (P1, P2, P3) = (0, 1, 0) is added. , Backpropagation. Further, when making the CNN learn “person”, the movement history information shown in FIG. 6C is input to the CNN, and correct data (P1, P2, P3) = (0, 0, 1) is added. , Backpropagation. In this way, parameters such as the weight of the path between the neurons of the learned model 50 are determined.

  Here, each of the movement history information shown in FIGS. 6A to 6C includes a rectangular area width (first row) and a rectangular area height (second row) at predetermined time intervals. ), The amount of horizontal movement of the rectangular area (third line), the amount of vertical movement of the rectangular area (fourth line), the width difference from the previous rectangular area (fifth line), (The sixth row). It is also possible to input only the horizontal movement amount (third line) and the vertical movement amount (fourth line) of the rectangular area at predetermined time intervals. Here, an example of the movement history information as learning data has been described. However, also when performing the type determination of the object A1, the movement history information having the same structure is input to the learned model 50.

  Next, the layer structure of the learned model 50 will be described. The learned model 50 shown in FIG. 7 includes a convolution layer (Convolution) 51, a convolution layer (Convolution) 52, an average pooling layer (Average Pooling) 53, a convolution layer (Convolution) 54, and an average pooling layer (Average Pooling). 55, a fully connected layer (Fully Connect) 56, a fully connected layer (Fully Connect) 57, and an output layer (Softmax) 58.

  The convolution layers 51, 52, and 54 generate a feature map by convolving a filter with a nearby node in the previous layer in order to extract local features. The average pooling layers 53 and 55 further reduce the feature map output from the convolution layer, which is the previous layer, into a new feature map in order to collect local features. As described above, the hidden layer of the CNN is formed by the convolution layer and the average pooling layer. The fully connected layers 56 and 57 combine the feature map from which the feature portions have been extracted into one node, and output a value converted by a predetermined activation function. For this activation function, a well-known technique such as ReLU (Rectified Linear Unit) can be used. The output layer 58 converts the outputs (feature variables) from the fully connected layer 57 into probabilities using a softmax function, and outputs the probabilities of being correctly classified. Note that a dropout layer can be added to avoid overfitting. Since the basic structure of the CNN is a known technology, a detailed description thereof is omitted here.

<Processing procedure of alarm control device 40>
Next, a processing procedure of the alarm control device 40 shown in FIG. 4 will be described. FIG. 8 is a flowchart showing a processing procedure of the alarm control device 40 shown in FIG. As shown in the figure, the alarm control device 40 receives a plurality of images from the monitoring camera 10 (Step S101). The plurality of images are images of frames that are continuous in time series. Note that a moving image may be input from the monitoring camera 10 and an image corresponding to each frame may be cut out from the moving image.

  Thereafter, the alarm control device 40 performs object detection using a plurality of images and obtains movement history information (step S102), and determines the type of the object A1 in each image (step S103). In this type determination, if the movement history information B is input to the learned model 50, the learned model 50 has a probability P1 that the type of the object A1 is “animal” and a probability that the type of the object A1 is “vehicle”. P2 and the probability P3 that the type of the object A1 is "person" are output. Here, when any one of the probabilities exceeds a predetermined threshold (for example, “0.8”), the type corresponding to the probability is determined to be the type of the object A1.

  Thereafter, when it is determined that the type of the object A1 is “animal” (Step S104; Yes), the alarm control device 40 issues an alarm instruction to the alarm lamp 20 and the speaker 30 (Step S105). On the other hand, if it is determined that the type of the object A1 is not “animal” (Step S104; No), no warning instruction is issued to the warning light 20 and the speaker 30. Here, for convenience of explanation, it is determined that the warning instruction is not issued when the type of the object A1 is not “animal”. However, when it is determined that the type of the object A1 is “vehicle” or “person”, If the type of the object A1 is determined to be "UNKNOWN (unknown)" without issuing a warning instruction, a warning instruction can be given.

  Next, a procedure related to object detection and acquisition of movement history information shown in step S102 of FIG. 8 will be described. FIG. 9 is a flowchart showing a procedure relating to object detection and acquisition of movement history information shown in step S102 of FIG.

  As shown in FIG. 9, the alarm control device 40 calculates an optical flow and a likelihood for each pixel of an image (step S201), and extracts a moving pixel based on the optical flow (step S202). Thereafter, a rectangle as a particle is arranged on the image (step S203), and the rectangle is moved based on the optical flow (step S204). Thereafter, the history of the movement vector of the center of the rectangle is stored in the storage unit 44 as the movement history information 44b (step S205). Thus, the movement history information 44b can be input to the learned model 50 to make a type determination.

  As described above, in the first embodiment, the object A1 included in the image captured by the monitoring camera 10 is detected, the detected object A1 is tracked, and the movement history information B of the object A1 is acquired. Since the type of the object A1 is determined based on the movement history information B, even when the number of pixels forming the object A1 included in the image is small, the movement history information B of the object A1 is used while utilizing the movement history information B. The type of the object A1 can be determined.

  By the way, in the first embodiment, the case where only the movement history information B of the object A1 is input to the learned model 50 when performing the type determination using the learned model 50 has been described. It is not limited. In the second embodiment, a case will be described in which the movement history information B of the object A1 and the input image A including the object A1 are input to the learned model 60 and the type determination is performed. Note that the system configuration and the configuration of the alarm control device are almost the same as those in the first embodiment, and thus description thereof is omitted here.

  FIG. 10 is a diagram illustrating an example of type determination using the learned model 60 according to the second embodiment. As shown in FIG. 10, in the second embodiment, the movement history information B of the object A1 and the image C of the object are input to the learned model 60. This image C is preferably an image such as the input image A including the object A1 shown in FIG. Note that the movement history information B is the data sequence described with reference to FIG.

  FIG. 11 is a diagram showing an example of the layer structure of the learned model 60 shown in FIG. As shown in FIG. 11, the learned model 60 includes a convolution layer (Convolution) 61a, a convolution layer (Convolution) 61b, a max pooling layer (Max Pooling) 61c, and a convolution layer ( Convolution) 61d, a Max Pooling layer 61e, and a Fully Connect layer 61f.

  Further, in order to process the movement history information B, a convolution layer (Convolution) 62a, a convolution layer (Convolution) 62b, an average pooling layer (Average Pooling) 62c, a convolution layer (Convolution) 62d, and an average pooling layer (Average pool) Pooling) 62e and a fully connected layer (Fully Connect) 62f. Further, it has a fully connected layer (Fully Connect) 63, a fully connected layer (Fully Connect) 64, and an output layer (Softmax) 65. The description of each layer is omitted.

  As described above, when the image C including the object A1 and the movement history information B of the object A1 are input, the learned model 60 has a probability P1 that the type of the object A1 is “animal” and a type P of the object A1. Is output as a probability P2 that is a "vehicle", and a probability P3 that the type of the object A1 is "a person".

  As described above, in the second embodiment, the image C including the object A1 and the movement history information B of the object A1 are input to the learned model 60, and the probability of each type of the object A1 is output. The type of the object A1 can be accurately determined from both the characteristics of the image of the object A1 and the characteristics of the movement history.

  By the way, in the first embodiment, the case where the movement history information B shown in FIG. 6 is input to the learned model 50 has been described. However, since deep learning using CNN has a high affinity with an image, this movement is performed. The history information B can be handled as a kind of parameter image.

  FIG. 12 is a diagram illustrating an example of type determination using the learned model 70 according to the third embodiment. As shown in FIG. 12, in the third embodiment, a parameter image D generated from the movement history information B of the object A1 is input to the learned model 70. Then, the learned model 70 outputs a probability P1 that the type of the object A1 is “animal”, a probability P2 that the type of the object A1 is “vehicle”, and a probability P3 that the type of the object A1 is “person”. Note that the layer structure of the learned model 70 can be the same as the learned model 50 shown in FIG.

  FIG. 13 is an explanatory diagram for explaining the parameter image D shown in FIG. As shown in FIG. 13A, the parameter image D is a pseudo image in which the horizontal axis is time, the plus side of the vertical axis is x moving distance, and the minus side of the vertical axis is y moving distance.

  FIG. 13B is a parameter image of a vehicle, FIG. 13C is a parameter image of an animal, and FIG. 13D is a parameter image of a person. The parameter image of the animal shown in FIG. 13C has a larger x-moving distance and a larger y-moving distance than the vehicle parameter image or the human parameter image. Therefore, when this parameter image is input to the learned model 70, the probability that the object A1 is an animal can be obtained with high accuracy.

  As shown in FIG. 14, a parameter image D generated from the movement history information B of the object A1 and an image C including the object A1 are input to the learned model 80, and the learned model 70 It is also possible to output a probability P1 that the type is "animal", a probability P2 that the type of the object A1 is "vehicle", and a probability P3 that the type of the object A1 is "person". The layer structure of the trained model 80 can be the same as the trained model 60 shown in FIG.

<Relationship with hardware>
Next, the correspondence between the alarm control devices 40 according to the first to third embodiments and the main hardware configuration of the computer will be described. FIG. 15 is a diagram illustrating an example of a hardware configuration.

  A general computer has a configuration in which a CPU 91, a ROM (Read Only Memory) 92, a RAM 93, a nonvolatile memory 94, and the like are connected by a bus 95. A hard disk device may be provided instead of the nonvolatile memory 94. For convenience of explanation, only a basic hardware configuration is shown.

  Here, the ROM 92 or the non-volatile memory 94 stores a program or the like necessary for starting an operating system (hereinafter, simply referred to as “OS”). The OS program is read from the memory 94 and executed.

  On the other hand, various application programs executed on the OS are stored in the non-volatile memory 94. When the CPU 91 executes the application program while using the RAM 93 as a main memory, a process corresponding to the application is executed. You.

  The type determination program of the alarm control device 40 according to the first to third embodiments is also stored in the nonvolatile memory 94 or the like, like the other application programs, and the CPU 91 loads and executes the type determination program. Will be. In the case of the alarm control device 40 according to the first to third embodiments, a type determination program including a routine corresponding to the detection unit 45a, the movement history information acquisition unit 45b, the type determination unit 45c, and the alarm instruction unit 45d illustrated in FIG. Is stored in the nonvolatile memory 94 or the like. When the type determination program is loaded and executed by the CPU 91, processes corresponding to the detection unit 45a, the movement history information acquisition unit 45b, the type determination unit 45c, and the alarm instruction unit 45d are generated. The learning data 44a and the setting data are stored in the nonvolatile memory 94 in advance.

  Further, in each of the first to third embodiments, the case where the operation is executed on the stand-alone device has been described. However, the present invention is not limited to this, and is applied to the case where the operation is performed in the cloud edge server device. You can also. Also, the present invention can be applied to a case where distributed processing is performed by a plurality of computers.

  Further, in each of Embodiments 1 to 3, the case where the learned model of the deep learning is used has been described, but the present invention is not limited to this, and the learned model of the machine learning other than the deep learning is used. You can also. In each of the first to third embodiments, the case where a learned model is used as a program has been described. However, the present invention is not limited to this, and may be realized by a hardware circuit such as an FPGA.

  In each of the first to third embodiments described above, the case in which animals are set as alarm targets is described. However, the present invention is not limited to this, and suspicious persons who are not allowed to enter the test course, The present invention can also be applied to a case where various objects such as objects and drones are set as alarm targets.

  In addition, each configuration illustrated in the above-described first to third embodiments is a schematic function and does not necessarily need to be physically configured as illustrated. In other words, the form of distribution / integration of each device is not limited to the one shown in the drawing, and all or a part thereof is functionally or physically distributed / integrated in arbitrary units according to various loads and usage conditions. Can be configured.

  INDUSTRIAL APPLICABILITY The type determination program, type determination device, and type determination method of the present invention are useful for quickly and accurately determining the type of an object in a situation where the number of pixels related to the object included in the input image is small.

A input image A1 object A2 object B movement history information C image D parameter image 10 surveillance camera 20 warning light 30 speaker 40 alarm control device 41 input unit 42 display unit 43 communication I / F unit 44 storage unit 44a learning data 44b movement history information 45 control unit 45a detection unit 45b movement history information acquisition unit 45c type determination unit 45d alarm processing unit 50, 60, 70, 80 learned model 91 CPU
92 ROM
93 RAM
94 Non-volatile memory 95 Bus

In order to solve the above-described problems, the present invention provides a detection procedure for detecting an object included in an image captured by a predetermined imaging device, and information relating to a movement history of a rectangular area including the object detected by the detection procedure. a movement history information acquisition procedure for acquiring a movement history information of the object, based on at least the travel history information, that to execute a type determination procedure for determining the type of the detected object by the detection procedure in the computer Features.

In addition, in the present invention according to the above invention, the type determination step includes a step of moving the rectangular area in a horizontal direction, the width of the rectangular area, the height of the rectangular area, and the movement of the rectangular area at predetermined time intervals included in the movement history information. the amount, the movement amount in the vertical direction of the rectangular area, a height difference between the width difference and the rectangular region of the rectangular region input to the learned model, the probability of each type output from the learned model And determining the type of the object detected by the detection procedure based on the detection procedure.

In addition, in the present invention according to the above invention, the type determination step includes a step of moving the rectangular area in a horizontal direction, the width of the rectangular area, the height of the rectangular area, and the movement of the rectangular area at predetermined time intervals included in the movement history information. the amount, the movement amount in the vertical direction of the rectangular area, enter the height difference between the width difference and the rectangular region of the rectangular region, and a partial image of the rectangular region in the learned model, the learned model And determining the type of the object detected by the detection procedure on the basis of the probability of each type output from.

Also, the present invention provides a detecting unit for detecting an object included in an image captured by a predetermined imaging device, and information relating to a moving history of a rectangular area including the object detected by the detecting unit. A moving history information acquiring unit that acquires the information as information ; and a type determining unit that determines a type of the object detected by the detecting unit based on at least the moving history information.

Further, the present invention provides a detecting step of detecting an object included in an image captured by a predetermined imaging device, and information relating to a moving history of a rectangular area including the object detected by the detecting step. A moving history information acquiring step of acquiring as information ; and a type determining step of determining a type of the object detected in the detecting step based on at least the moving history information.

Claims (13)

A detection procedure for detecting an object included in an image captured by a predetermined imaging device,
A movement history information obtaining step of tracking an object detected by the detection step and obtaining movement history information of the object;
And a type determining step of determining a type of the object detected by the detecting step based on at least the movement history information. The detection procedure includes:
The computer-readable storage medium according to claim 1, wherein an object included in an image captured by the predetermined imaging device is detected based on a movement vector of pixels between the plurality of images. The moving history information acquisition procedure includes:
A rectangular area including the object is specified based on a movement vector of a pixel forming the object detected by the detection procedure and a predetermined likelihood, and information on a movement history of the specified rectangular area is obtained as movement history information. The type determination program according to claim 1 or 2, wherein: The type determination procedure includes:
The type of the object detected by the detection procedure is determined based on the movement history information of the object forming the teacher data and a learned model in which the type of the object is applied to a predetermined multilayer neural network. 3. The type determination program according to 3. The type determination procedure includes:
The movement amount in the horizontal direction and the movement amount in the vertical direction of the rectangular area for each predetermined time included in the movement history information are input to the trained model, and based on the probability for each type output from the trained model. The program according to claim 4, wherein the type of the object detected by the detection procedure is determined. The type determination procedure includes:
The width of the rectangular area, the height of the rectangular area, the amount of horizontal movement of the rectangular area, the amount of vertical movement of the rectangular area, A width difference and a height difference from the previous rectangular area are input to the learned model, and a type of the object detected by the detection procedure is determined based on a probability for each type output from the learned model. The type determination program according to claim 4, characterized in that: The type determination procedure includes:
The width of the rectangular area, the height of the rectangular area, the amount of horizontal movement of the rectangular area, the amount of vertical movement of the rectangular area, The width difference and the height difference from the previous rectangular area and the partial image of the rectangular area are input to the learned model, and the detection is performed by the detection procedure based on the probability for each type output from the learned model. The type determination program according to claim 4, wherein the type of the determined object is determined. The type determination procedure includes:
A parameter image obtained by imaging a horizontal movement amount and a vertical movement amount of the rectangular area for each predetermined time included in the movement history information is input to the learned model, and is output from the learned model. The program according to claim 4, wherein the type of the object detected by the detection procedure is determined based on the probability of each type. The type determination procedure includes:
A parameter image obtained by imaging a horizontal movement amount and a vertical movement amount of the rectangular area for each predetermined time included in the movement history information, and a partial image of the rectangular area are input to the learned model. The type determination program according to claim 4, wherein the type of the object detected by the detection procedure is determined based on the probability of each type output from the learned model. The trained model is
The method according to any one of claims 4 to 9, wherein when at least movement history information relating to an animal, a person, or a vehicle is input, a probability that the object is an animal, a person, or a vehicle is output. Type determination program.   If the type of the object is determined to be an animal by the type determination procedure, control is performed to issue a warning by a predetermined warning light, and if the type of the object is determined to be a person or a vehicle, the 11. The program according to claim 10, further causing the computer to execute an alarm control procedure for controlling not to issue an alarm by a predetermined alarm light. A detection unit that detects an object included in an image captured by a predetermined imaging device;
A movement history information acquisition unit that acquires the movement history information of the object by tracking the object detected by the detection unit;
A type determining unit configured to determine a type of the object detected by the detecting unit based on at least the movement history information. A detection step of detecting an object included in an image captured by a predetermined imaging device,
A movement history information acquisition step of tracking an object detected by the detection step and acquiring movement history information of the object;
A type determining step of determining a type of the object detected in the detecting step based on at least the movement history information.

Images