JP2020140236A - Image analysis system and image analysis method - Google Patents

Abstract

To provide an image analysis technique for determining propriety of oil supply.SOLUTION: An image analysis system comprises: a video acquisition section for acquiring videos obtained by imaging an oil supply space; and an image processing server. The image processing server includes: a detection section for detecting a fraudulent act concerning a video acquired by the video acquisition section with the use of learning models obtained by learning fraudulent acts by machine learning concerning a video group of an oil supply space; and a determination section for generating information related to propriety of oil supply based on a detection result of a fraudulent act.SELECTED DRAWING: Figure 2

Description

The present invention relates to an image analysis system and an image analysis method.

Conventionally, gas stations that refuel by self-service are known. At such gas stations, observers who are qualified as hazardous materials handlers are stationed by law. The observer remotely controls whether or not refueling is possible while checking the refueling operation of the refueling person through a live image of the refueling space from the office of the gas station.

Further, in Patent Document 1, "the color tone of the refueling nozzle is identified by imaging the vicinity of the refueling port of the vehicle, and the oil type of the refueling nozzle matches the oil type of the vehicle based on the identified color tone of the refueling nozzle. If it is determined that the oil type is not suitable, a warning for inappropriate oil type is given. "

JP-A-2009-179103

Patent Document 1 does not disclose any warnings other than "inappropriate oil type based on the color tone of the refueling nozzle".

However, there are other fraudulent acts (acts that deviate from the norms of safety or error prevention) that the observer should judge when deciding whether or not to refuel.
In general, observers must keep an eye on the live footage of the refueling space and pay attention to various frauds at the same time. Therefore, there is a problem that the burden on the observer is heavy.

Therefore, the present invention provides an image analysis technique for determining whether or not refueling is possible.

In order to solve the above problem, one of the typical image analysis systems of the present invention is a video acquisition unit that acquires an image of the refueling space and a learning model that machine-learns fraudulent activity about the video group of the refueling space. It is provided with a detection unit that detects fraudulent activity in the video acquired by the image acquisition unit and a determination unit that generates information on whether or not refueling is possible based on the detection result of the fraudulent activity.

INDUSTRIAL APPLICABILITY According to the present invention, image analysis for determining whether or not refueling is possible becomes possible.
Issues, configurations and effects other than those described above will be clarified by the description of the following embodiments.

It is a figure which shows the self-service type gas station GS. It is a block diagram of an image analysis system 100. It is a figure which shows the basic structure of a learning model. It is a figure explaining the preprocessing example of video information. It is a figure which shows the learning data set of a refueling position. It is a figure which shows the data set for learning of the static electricity removal behavior. It is a figure which shows the learning data set of a nozzle insertion. It is a figure which shows the learning data set of the number of staying people. It is a figure which shows the learning data set of refueling to other than a vehicle. It is a figure which shows the learning data set of the attention-requiring situation. It is a figure which shows the learning data set for the area division of a car number. It is a flow chart (1/2) which shows the operation of the image analysis system 100. It is a flow chart (2/2) which shows the operation of the image analysis system 100. It is a figure explaining the detection example of the refueling position of a vehicle. It is a figure explaining the detection example of the static electricity removal behavior. It is a figure explaining the detection example of the insertion state of the refueling nozzle. It is a figure explaining the detection example of the number of staying people. It is a figure explaining the detection example such as refueling outside the vehicle, dangerous / unsafe act. It is a figure which shows the display screen of a monitoring personal computer and a handheld terminal. It is a figure explaining image analysis after refueling permission. It is a figure which shows the main part of the image analysis system 500. It is a flow chart which shows the operation of the image analysis system 500.

Hereinafter, examples of the present invention will be described with reference to the drawings.

[Structure Description of Example 1]
FIG. 1 is a diagram showing a self-service gas station GS.
In the figure, a plurality of refueling lanes (No1 to NoX) are installed at the gas station GS. A refueling device 10 is arranged in each of these refueling lanes. The refueling device 10 is provided with three systems of refueling nozzles corresponding to three oil types (regular, high-octane, and light oil). These three types of refueling nozzles are color-coded so that the refueler can easily distinguish the oil type.

For example, a refueling nozzle that refuels regular gasoline is color coded "red." Refueling nozzles that refuel high octane gasoline are color coded "yellow". The refueling nozzle that refuels light oil is color-coded into "green".

On the other hand, in each of the refueling lanes, an image acquisition unit 11 is installed so as to acquire an image of the refueling space with as few blind spots as possible. For example, the image acquisition unit 11 includes a camera A1 (for behavior analysis) that captures the behavior of the refueling person from a bird's-eye view position or a bird's-eye view position, and a camera B1 (for vehicle number analysis) that captures the license plate of the vehicle from an oblique lateral direction. Consists of.

FIG. 2 is a block diagram of the image analysis system 100 for refueling monitoring.
In the figure, the image analysis system 100 includes an image acquisition unit 11, a detection unit 12 (including learning models 200 to 260), a determination unit 13, a refueling history 14, a monitoring personal computer 15, an alarm device 16, an alarm device 16a, and a handheld terminal. It is configured to include 17, a wireless LAN 18, an interface unit 19, a refueling control device 20, and a network switch 21.

Here, the detection unit 12, the determination unit 13, and the refueling history 14 are realized by the functions of the image processing server 22 with the AI learning function.

The image processing server 22 is configured as a computer equipped with a CPU (Central Processing Unit), a memory, and the like as hardware. Part or all of this hardware may be replaced with a DSP (Digital Signal Processor), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit), or the like. In addition, part or all of the hardware may be centralized or distributed to servers on the network and placed in the cloud.

Next, the signal flow in the image analysis system 100 will be described.
The video acquisition unit 11 is connected to the network switch 21 via a LAN cable (PoE power supply). The video acquisition unit 11 receives power from the network switch 21 side via a LAN cable. Further, the video information output from the video acquisition unit 11 is supplied to the network switch 21 via the LAN cable.

The network switch 21 switches the video information (live video) supplied from the video acquisition unit 11 for each refueling lane for each refueling lane and outputs it to the detection unit 12.

The detection unit 12 analyzes the video information using the learning models 200 to 260, and detects fraudulent activity in the refueling space. The determination unit 13 generates information on whether or not refueling is possible based on the detection result of fraudulent activity.

The information regarding whether or not refueling is possible, which is generated by the determination unit 13, is transmitted to the monitoring personal computer 15 in the office room of the gas station GS via the network switch 21. Based on the information on whether or not refueling is possible, the monitoring personal computer 15 alerts the resident observer through the monitor screen and the alarm device 16.

An alarm 16a is also installed in each of the refueling lanes. These alarms 16a are connected to the monitoring personal computer 15 via the network switch 21. The monitoring personal computer 15 provides voice guidance and alerts to the refueling person via the alarm device 16a.

Further, the monitoring personal computer 15 is connected to the refueling control device 20 via the network switch 21 and the interface unit 19.

The refueling control device 20 controls the refueling stop, refueling amount, and discharge amount of the refueling device 10 by controlling the refueling valve and the refueling pressure in the refueling device 10 for each refueling lane.

The refueling device 10 also communicates with the monitoring personal computer 15 and the image processing server 22 via the network switch 21, and transmits information such as billing settlement at the time of refueling and the selected oil type and refueling amount. The refueling history 14 records information on the history related to refueling.

The observer can perform refueling monitoring and refueling control similar to those of the monitoring personal computer 15 via a handheld terminal 17 such as a tablet. A plurality of wireless LANs 18 are distributed and arranged so that the handheld terminal 17 covers communication within the premises of the gas station GS.

The learning models 200 to 260 included in the detection unit 12 are used for image analysis such as the following fraudulent activity and vehicle number recognition.
・ Learning model 200: Cheating that does not stop properly at the refueling position ・ Learning model 210: Cheating that does not remove static electricity ・ Learning model 220: Cheating that does not insert the refueling nozzle correctly ・ Learning model 230: Unnecessary number of people stay Cheating / learning model 240: Cheating / learning model 250 for refueling other than vehicles: Cheating / learning model 260 for situations requiring attention: Car number recognition The machine learning and use of these learning models 200 to 260 will be described later. To do.

[Basic configuration of learning models 200 to 260]
First, the basic configuration common to the learning models 200 to 260 will be described.

FIG. 3 is a diagram showing a basic configuration of learning models 200 to 260.
In the figure, the video information of the refueling space is input to the preprocessing unit 31. The pre-processing unit 31 performs pre-processing on the video information according to the analysis content of the image analysis.

For example, the preprocessing unit 31 can set an analysis area (crop, resize, bird's-eye view conversion, viewpoint conversion, distortion correction, padding before convolution, etc.), subtraction comparison (background subtraction, contour extraction, etc.), and vector analysis (motion vector). Detection, motion tracking) and signal processing (feature extraction, shape extraction, contour enhancement, gradation correction, dynamic range scaling, level normalization, noise suppression, moire removal, white balance adjustment, color processing, color reduction processing, monochrome And other pretreatments.

For example, in the learning model that performs "image analysis of the presence or absence of an object" shown in FIG. 4 [A], the preprocessing unit 31 performs difference comparison, object shape extraction, object feature extraction, and an object area in which an object may exist. Perform image processing such as cropping.

Further, in the learning model for performing "image analysis on the refueling nozzle, refueling destination, etc." shown in FIG. 4 [B], the pretreatment unit 31 determines the oil type of the refueling type from the color of the refueling nozzle and the front storage position of the refueling device 10. To judge. Further, the pretreatment unit 31 detects the target area of the refueling nozzle and the refueling destination by vector-analyzing the movement of the refueling nozzle (or the hand of the refueling person), and crops the target area.

Further, in the learning model for performing "image analysis on vehicle number" shown in FIG. 4 [C], the preprocessing unit 31 detects the target area of the license plate by difference comparison or shape extraction of the license plate, and determines the target area. Crop. Further, if necessary, keystone correction or the like may be added to the license plate, or color reduction processing for removing the ground color of the license plate may be performed.

Further, in the learning model of the type that performs "image analysis on human behavior" shown in FIG. 4 [D], the preprocessing unit 31 uses the mobile device or during smoking or telephone call by comparing the movement, shape, and difference of the hand. Detects a target area such as inside and crops the target area.

The video information preprocessed by the preprocessing unit 31 is input to the input layer 32 in the learning models 200 to 260. The video information of the input layer 32 is processed via the convolution layer / activation function 33 and the pooling layer 34, and is converted into a feature map composed of neurons that are activated according to the local features of the image space. The convolution layer / activation function 33 and the pooling layer 34 are multi-layered by the number of layers suitable for image analysis.

The neuron values of the feature map via the multi-layered convolution layer / activation function 33 and the pooling layer 34 are serialized and then input to the neural network 35. The neural network 35 is configured by connecting neurons (for example, fully connected) with a number of layers suitable for image analysis. A value indicating the result of image analysis (likelihood, etc.) is output from the output layer 36 at the final end of the neural network 35.

[Machine learning of learning models 200-260]
The above-mentioned convolution layer / activation function 33 and the neurons in the neural network 35 have a weight array w and a bias b. These weight arrays w and bias b are sequentially updated by repeating machine learning (error back propagation method, etc.) using the learning data set, and the determination result according to the learned teacher data is output.

This machine learning is performed as a preparation before the system starts operation. In addition, after the system starts operation, machine learning is performed in line with reality by automatically collecting the actual surveillance image as a training image and using the judgment of the observer for the surveillance image as teacher data.

5 to 11 are diagrams showing examples of learning data sets (sets of training images and teacher data) used for such machine learning by type of image analysis. Hereinafter, each of these training data sets will be described.

-Machine learning of "fraud that does not stop correctly at the refueling position" for the learning model 200 Fig. 5 shows a learning data set for performing machine learning about the correct stop position of the vehicle at the time of refueling (hereinafter referred to as "refueling position"). 200A-C are shown. The training images of the training data sets 200A to C are preprocessed by the preprocessing unit 31 of the learning model 200 (processing of cropping the vicinity of the refueling position as the target area, preferably processing of converting the stop frame into a rectangle by bird's-eye view conversion). Is the video information that has already been implemented.

In the figure, the group of training images of the learning data set 200A is video information in a state where the vehicle is stopped without protruding from the stop frame of the refueling space and the refueling port of the vehicle is in the direction of the refueling device 10. .. Teacher data of "refueling position OK" is given to the group of these training images.

On the other hand, the group of training images of the learning data set 200B is video information in a state where the vehicle is stopped outside the stop frame of the refueling space. Teacher data of "refueling position NG" is given to the group of these training images.

Further, the group of training images of the learning data set 200C is video information in a state where the refueling port of the vehicle is on the opposite side of the refueling device 10. Teacher data of "refueling position NG (or refueling port reverse NG)" is given to the group of these training images.

By machine learning whether or not the vehicle is stopped at the refueling position by using the learning data sets 200A to C shown in FIG. 5, fraudulent acts in which the vehicle does not stop at the refueling position (furthermore, the refueling port is the refueling device 10). A learning model 200 that detects fraudulent activity on the opposite side is obtained.

-Machine learning of "cheating without static electricity removal" for the learning model 210 FIG. 6 shows learning data sets 210A to C for performing machine learning about cheating without static electricity removal. The training images of the learning data sets 210A to C are video information that has been preprocessed by the preprocessing unit 31 of the learning model 210 (processing of cropping the vicinity of the static electricity removing sheet as a target area).

In the figure, the group of training images of the learning data set 210A is video information in a state where the refueling person is in contact with the static electricity removing sheet with bare hands (without wearing gloves or the like). Teacher data of "static electricity removal OK" is given to the group of these training images.

On the other hand, the group of training images of the learning data set 210B is video information in a state where the refueling person is not in contact with the static electricity removing sheet. Teacher data of "static electricity removal NG" is given to these groups of training images.

Further, the group of training images of the learning data set 210C is video information in a state where the refueling person contacts the static electricity removing sheet through gloves, a handkerchief, or the like (insufficient discharge). Teacher data of "static electricity removal NG (or glove wearing NG)" is given to these groups of training images.

By using the learning data sets 210A to 210C shown in FIG. 6 to machine-learn whether or not the refueler has performed static electricity removal behavior, the refueler does not perform static electricity removal behavior (further, wearing gloves or the like). A learning model 210 that detects (a fraudulent act of removing static electricity) is obtained.

-Machine learning of "fraud of not inserting the refueling nozzle correctly" with respect to the learning model 220 FIG. 7 shows learning data sets 220A to C for machine learning of fraudulent acts related to the insertion of the refueling nozzle. The training images of the learning data sets 220A to C are video information in which the preprocessing of the preprocessing unit 31 of the learning model 220 (processing of cropping the vicinity of the refueling port detected by the movement vector of the refueling nozzle as the target area) has been performed. Is.

In the figure, the group of training images of the learning data set 220A is video information in a state where the refueling nozzle is correctly inserted in the refueling port. Teacher data of "nozzle insertion OK" is given to the group of these training images.

On the other hand, the group of training images of the learning data set 220B is video information in a state where the refueling nozzle is not inserted in the refueling port. Teacher data of "nozzle insertion NG" is given to these groups of training images.

Further, the group of training images of the learning data set 220C is video information in a state where the refueling nozzle is floated or tilted without being inserted all the way to the back of the refueling port. Teacher data of "nozzle insertion NG" is given to these groups of training images.

A learning model that detects fraudulent activities in which the refueling nozzle is not correctly inserted into the refueling port by machine learning whether or not the refueling nozzle is correctly inserted into the refueling port using the learning data sets 220A to C shown in FIG. 220 is obtained.

-Machine learning of "cheating behavior in which an unnecessary number of people stay" with respect to the learning model 230 FIG. 8 shows learning data sets 230A to C for machine learning of cheating behavior related to the number of staying people. The training images of the training data sets 230A to C are preprocessed by the preprocessing unit 31 of the learning model 230 (processing of cropping the periphery of the vehicle as a target area, preferably processing of converting the stop frame into a rectangle by bird's-eye view conversion). This is the video information that has already been implemented.

In the figure, the group of training images of the learning data set 230A is video information in which the number of people staying is the number of people required for refueling (for example, one refueling person). Teacher data of "number of people staying OK" is given to these groups of training images.

On the other hand, the group of training images of the learning data set 230B is also video information in which the number of people staying is the number of people required for refueling (for example, about two people, a refueler and an adviser). Teacher data of "number of people staying OK" is given to these groups of training images.

Further, the group of training images of the learning data set 230C is video information in a state where the number of people staying exceeds a predetermined number required for refueling. Teacher data of "number of people staying NG" is given to these groups of training images.

By machine learning the number of people staying in the refueling space using the learning data sets 230A to C shown in FIG. 8, a learning model 230 that detects fraudulent activity in which the number of people in the refueling space exceeds a predetermined number can be obtained.

-Machine learning of "fraud to refuel other than vehicle" for learning model 240 FIG. 9 shows learning data sets 240A to C for machine learning of fraudulent behavior to refuel other than vehicle. The training images of the learning data sets 240A to C are video information in which the preprocessing of the preprocessing unit 31 of the learning model 240 (processing of cropping the insertion destination of the refueling nozzle and the reachable range as the target area) has been performed. ..

In the figure, the group of training images of the learning data set 240A is video information in a state where the vehicle is refueled. Teacher data of "refueling to the vehicle is OK" is given to the group of these training images.

On the other hand, the group of training images of the learning data set 240B is video information in a state of refueling other than the vehicle (kerosene tank, plastic tank, carrying can, etc.). Teacher data of "refueling NG to other than the vehicle" is given to the group of these training images.

Further, the group of training images of the learning data set 240C is video information in a state of refueling (kerosene tank, plastic tank, carrying can, etc.) mounted on a vehicle trunk or the like. Teacher data of "refueling NG to other than the vehicle" is given to the group of these training images.

By machine learning whether or not the refueling destination is a vehicle by using the learning data sets 240A to C shown in FIG. 9, a learning model 240 for detecting a fraudulent act of refueling other than the vehicle can be obtained.

-Machine learning of "fraud related to a caution situation" with respect to the learning model 250 FIG. 10 shows learning data sets 250A to C for machine learning of a fraudulent act related to a caution situation. The training images of the learning data sets 250A to C are video information that has been preprocessed by the preprocessing unit 31 of the learning model 250 (processing of cropping a range in which a cautionary situation occurs as a target area).

In the figure, the group of training images of the learning data sets 250A and 250B is video information in a state where any one of the following types of caution situations (or a combination of a plurality of types of caution situations) has occurred.

(1) Use of fire, (2) Tobacco, (3) Lighter, (4) Match, (5) Smoking equipment, (6) Smoking, (7) Ignition, (8) Spark generation, (9) High temperature material, ( 10) Flammables, (11) Combustibles, (12) Explosives, (13) Dangerous goods, (14) Carrying cans, (15) Kerosene tanks, (16) Careless state of talking and screen gaze, (17) Carelessness due to mobile phones, (18) Carelessness due to mobile devices, (19) Presence of children outside the vehicle, (20) Looking away during refueling, (21) Conversation during refueling, (22) Falling of refuelers, etc. , (23) Conflict with refuelers, (24) Mayhem in refueling space, (25) Lost or forgotten refueling cap, (26) Drunk, collapsed, crouched, hidden something, etc. 27) Careless behavior, (28) Dangerous behavior Teacher data of "Caution required status NG" is given to these groups of training images.

On the other hand, the group of training images of the learning data set 250C is video information in a state in which the "need attention situation" is removed from the training images of the learning data sets 250A and 250B. For these groups of training images, teacher data of "No caution status OK" is given.

By machine learning the fraudulent activity related to the caution situation by using the learning data sets 250A to C shown in FIG. 10, a plurality of learning models 250 for detecting the fraudulent activity related to the caution situation can be obtained.

-Machine learning of "vehicle number recognition" for the learning model 260 FIG. 11 shows a learning data set 260A for area division of vehicle numbers. The training image of this training data set 260A is preprocessed by the preprocessing unit 31 of the training model 260 (processing of cropping the license plate area, preferably converting the license plate into a rectangle, and normalizing the image size of the license plate. This is the video information that has been processed.

Area labels (land code, vehicle type code, usage code, serial designation number code, etc.) for classifying the area of the training image according to the code type are assigned to these groups of training images. As the area label here, the area label in accordance with the license plate display law of the country or area of use of the image analysis system 100 is given.

By performing machine learning with such a learning data set 260A, a learning model 260 that divides the vehicle number of the license plate into an image area for each code type can be obtained.

Further, the learning model 260 reads each code of the vehicle number as characters, numbers, and symbols by processing the area-divided image area with the learning device for character recognition.

[Explanation of operation of image analysis system 100]
Subsequently, the operation of the image analysis system 100 performed by using the machine-learned learning models 200 to 260 will be described.

12 to 13 are flow charts showing the operation of the image analysis system 100.
14 to 18 are diagrams for explaining the state of image analysis.
FIG. 19 is a diagram showing a display screen of the monitoring personal computer 15 and the handheld terminal 17.
Hereinafter, the operation of image analysis will be described in the order of the step numbers shown in FIGS. 12 to 13.

Step S11: The cameras A1 and B1 (image acquisition unit 11) arranged for each refueling lane output video information (moving image) of the refueling space. This video information is input to the detection unit 12 in the image processing server 22 via the network switch 21.

Step S12: The detection unit 12 outputs video information to the learning model 200. The pre-processing unit 31 in the learning model 200 performs pre-processing (processing of cropping the vicinity of the refueling position of the vehicle as a target area, preferably processing of converting the stop frame into a rectangle by bird's-eye view conversion) on this video information. .. The preprocessed video information is processed via a learning model 200 that machine-learns whether or not the vehicle is correctly stopped at the refueling position. The learning model 200 outputs the likelihood of cheating that the vehicle does not stop at the refueling position. Alternatively, the likelihood of fraudulent activity in which the vehicle direction is opposite and the refueling port does not exist on the refueling device 10 side is output.

FIG. 14 is a diagram showing an example of detecting the refueling position of the vehicle.
The more the vehicle stops in the correct position and orientation, the less likely it is to cheat. Conversely, the more the vehicle deviates from the correct refueling position or the vehicle is turned in the opposite direction, the higher the likelihood of fraud.

Step S13: The determination unit 13 generates information on whether or not refueling is possible based on the likelihood of fraud output by the learning model 200. If it is determined that refueling is not possible here, the determination unit 13 shifts the operation to step S14. In other cases, the determination unit 13 shifts the operation to step S15.

Step S14: The determination unit 13 transfers the information regarding the inability to refuel to the monitoring personal computer 15, the handheld terminal 17, and the alarm device 16a. The monitoring personal computer 15, the handheld terminal 17, and the alarm 16 display a monitor display of "a fraudulent act in which the vehicle does not stop at the refueling position" or "a fraudulent act in which the vehicle direction is opposite and the refueling port does not exist on the refueling device 10 side". Notify the observer by voice notification or alarm lighting. The alarm 16a located in the refueling lane also gives a voice notification and lights an alarm, and tells the refueler that "the vehicle does not stop at the refueling position" or "the vehicle is facing in the opposite direction and the refueling port is on the refueling device 10 side." Notify "non-existent cheating". After these notification operations, the determination unit 13 returns the operation to step S12.

Step S15: The detection unit 12 outputs video information to the learning model 210. The pre-processing unit 31 in the learning model 210 performs pre-processing (processing of cropping the vicinity of the static electricity removing sheet as a target area) on this video information. The preprocessed video information is processed via a learning model 210 that has machine-learned whether or not static electricity has been removed. The learning model 210 outputs the likelihood of fraud without static electricity removal.

FIG. 15 is a diagram showing an example of detecting static electricity removal behavior.
If the refueling person touches the static electricity removal sheet even for a moment, the likelihood of fraud is reduced. However, if an inclusion such as a glove or a handkerchief with a color different from the skin color is detected, the likelihood of cheating increases.

Step S16: The determination unit 13 generates information on whether or not refueling is possible based on the likelihood of fraud output by the learning model 210. If it is determined that refueling is not possible here, the determination unit 13 shifts the operation to step S17. In other cases, the determination unit 13 shifts the operation to step S18.

Step S17: The determination unit 13 transfers the information regarding the inability to refuel to the monitoring personal computer 15, the handheld terminal 17, and the alarm device 16a. The monitoring personal computer 15, the handheld terminal 17, and the alarm device 16 notify the observer of "a fraudulent act that does not remove static electricity" by displaying a monitor, voice notification, or lighting an alarm. The alarm 16a arranged in the refueling lane also performs voice notification and alarm lighting to notify the refueler of "a fraudulent act that does not remove static electricity". After these notification operations, the determination unit 13 returns the operation to step S15.

Step S18: The detection unit 12 outputs video information to the learning model 220. The pre-processing unit 31 in the learning model 220 performs pre-processing (processing of cropping the vicinity of the refueling port detected by vector tracking of the refueling nozzle as a target area) on this video information. The preprocessed video information is processed via a learning model 220 that machine-learns whether or not the refueling nozzle is correctly inserted into the refueling port. The learning model 220 outputs the likelihood of fraudulent activity in which the refueling nozzle is not inserted correctly.

FIG. 16 is a diagram showing an example of detecting the insertion state of the refueling nozzle.
The more correctly the refueling nozzle is inserted into the refueling port, the lower the likelihood of fraud. The more the refueling nozzle floats or tilts, the higher the likelihood of fraud.

Step S19: The determination unit 13 generates information on whether or not refueling is possible based on the likelihood of fraud output by the learning model 220. If it is determined that refueling is not possible here, the determination unit 13 shifts the operation to step S20. In other cases, the determination unit 13 shifts the operation to step S21.

Step S20: The determination unit 13 transfers the information regarding the inability to refuel to the monitoring personal computer 15, the handheld terminal 17, and the alarm device 16a. The monitoring personal computer 15, the handheld terminal 17, and the alarm device 16 notify the observer of "a fraudulent act in which the refueling nozzle is not correctly inserted into the refueling port" by a monitor display, a voice notification, or an alarm lighting. The alarm 16a arranged in the refueling lane also gives a voice notification and lights an alarm to notify the refueler of "a fraudulent act in which the refueling nozzle is not correctly inserted into the refueling port". After these notification operations, the determination unit 13 returns the operation to step S18.

Step S21: The detection unit 12 outputs video information to the learning model 230. The pre-processing unit 31 in the learning model 230 performs pre-processing (processing of cropping the periphery of the vehicle as a target area, preferably processing of converting the stop frame into a rectangle by bird's-eye view conversion) on the video information. The preprocessed video information is processed via a learning model 230 in which the number of people unnecessary for refueling is machine-learned. The learning model 230 outputs the likelihood of fraudulent activity in which an unnecessary number of people stay in the refueling space when the number of people exceeds a predetermined number.

FIG. 17 is a diagram showing a detection example of a person other than the refueling person.
If the number of people required for refueling, the likelihood of cheating is low. The greater the number of non-fuelers, the higher the likelihood of fraud.

Step S22: The determination unit 13 generates information regarding whether or not refueling is possible based on the likelihood of fraud output by the learning model 230. If it is determined that refueling is not possible here, the determination unit 13 shifts the operation to step S23. In other cases, the determination unit 13 shifts the operation to step S24.

Step S23: The determination unit 13 transfers the information regarding the inability to refuel to the monitoring personal computer 15, the handheld terminal 17, and the alarm device 16a. The monitoring personal computer 15, the handheld terminal 17, and the alarm device 16 notify the observer of "a fraudulent act in which an unnecessary number of people stay for refueling" by displaying a monitor, voice notification, or lighting an alarm. The alarm 16a arranged in the refueling lane also gives a voice notification and lights an alarm to notify the refueler of "a fraudulent act in which an unnecessary number of people stay for refueling". After these notification operations, the determination unit 13 returns the operation to step S21.

Step S24: The detection unit 12 outputs video information to the learning model 240. The pre-processing unit 31 in the learning model 240 performs pre-processing (processing of cropping the reachable range of the refueling nozzle as a target area) on this video information. The preprocessed video information is processed via a learning model 240 that machine-learns whether or not the refueling destination is a vehicle (including a motorcycle). The learning model 240 outputs the likelihood of fraudulent activity with a refueling destination other than the vehicle.

FIG. 18 is a diagram showing a detection example of refueling to a vehicle other than the vehicle.

With proper refueling of the vehicle, the likelihood of fraud is low. If a refueling destination other than the vehicle is detected, the likelihood of fraud increases.

Step S25: The determination unit 13 generates information on whether or not refueling is possible based on the likelihood of fraud output by the learning model 240. If it is determined that refueling is not possible here, the determination unit 13 shifts the operation to step S26. In other cases, the determination unit 13 shifts the operation to step S27.

Step S26: The determination unit 13 transfers the information regarding the inability to refuel to the monitoring personal computer 15, the handheld terminal 17, and the alarm device 16a. The monitoring personal computer 15, the handheld terminal 17, and the alarm device 16 notify the observer of "a fraudulent act with a refueling destination other than the vehicle" by a monitor display, a voice notification, or an alarm lighting. The alarm 16a arranged in the refueling lane also performs voice notification and alarm lighting to notify the refueler of "a fraudulent act with a refueling destination other than the vehicle". After these notification operations, the determination unit 13 returns the operation to step S24.

Step S27: The detection unit 12 outputs video information to the learning model 250. The pre-processing unit 31 in the learning model 250 performs pre-processing (processing such as cropping a range in which a cautionary situation occurs as a target area) on the video information. The preprocessed video information is processed via a learning model 250 that has machine-learned about the situation requiring attention. The learning model 250 outputs the likelihood of cheating regarding the situation requiring attention.

Step S28: The determination unit 13 generates information on whether or not refueling is possible based on the likelihood of fraud output by the learning model 250. If it is determined that refueling is not possible here, the determination unit 13 shifts the operation to step S29. In other cases, the determination unit 13 shifts the operation to step S30.

Step S29: The determination unit 13 transfers the information regarding the inability to refuel to the monitoring personal computer 15, the handheld terminal 17, and the alarm device 16a. The monitoring personal computer 15, the handheld terminal 17, and the alarm device 16 notify the observer of "a fraudulent act related to a cautionary situation" by a monitor display, a voice notification, or an alarm lighting. The alarm 16a arranged in the refueling lane also performs voice notification and alarm lighting to notify the refueler of "misconduct regarding the situation requiring attention". After these notification operations, the determination unit 13 returns the operation to step S27.

Step S30: The detection unit 12 outputs video information to the learning model 260. The preprocessing unit 31 in the learning model 260 preprocesses this video information (processing of cropping the license plate area, preferably processing of converting the license plate into a rectangle and normalizing the image size of the license plate). To carry out. The preprocessed video information is processed via a learning model 260 that has machine-learned about reading the car number. The learning model 260 outputs the reading result of the car number.

Step S31: The detection unit 12 determines the color of the refueling nozzle used for refueling based on the image information of the refueling space, and acquires information on the oil type to be refueled. Information may be acquired from the refueling device 10 or the like as the oil type for refueling the oil type purchased and selected by the refueling device 10.

Step S32: The determination unit 13 collates the vehicle number read in step S30 with the refueling history 14. When the same vehicle number is found in the refueling history 14, the determination unit 13 acquires the oil type refueled in the past from the refueling history 14.

Step S33: When the oil type information acquired in step S31 is different from the oil type refueled to the same vehicle in the past, the determination unit 13 shifts the operation to step S34. On the other hand, when refueling the same oil type as in the past, or when the vehicle number is not found in the refueling history 14, the determination unit 13 shifts the operation to step S35.

Step S34: The determination unit 13 transfers the information regarding the unsuitability of the oil type to the monitoring personal computer 15, the handheld terminal 17, and the alarm device 16a. The monitoring personal computer 15, the handheld terminal 17, and the alarm device 16 notify the observer of "refueling is not possible due to improper oil type" by a monitor display, voice notification, or alarm lighting. The alarm 16a arranged in the refueling lane also gives a voice notification and lights an alarm to notify the refueler that "refueling is not possible due to inappropriate oil type". After these notification operations, the determination unit 13 returns the operation to step S31.

Step S35: The determination unit 13 refers to the vehicle number read in step S30 to a registration database such as the outside via a network such as the Internet, and acquires registration information of the oil type and the vehicle type. (If permission is obtained in advance for fixed customers, etc., if possible, obtain oil type and vehicle type registration information from the automobile inspection registration information service, etc.)

Step S36: If the oil type information acquired in step S31 does not match the registered information, the determination unit 13 shifts the operation to step S37. On the other hand, when refueling an oil type matching the registration information or when the vehicle number is not found in the registration database, the determination unit 13 shifts the operation to step S38.

Step S37: The determination unit 13 transfers the information regarding the unsuitability of the oil type to the monitoring personal computer 15, the handheld terminal 17, and the alarm device 16a. The monitoring personal computer 15, the handheld terminal 17, and the alarm device 16 notify the observer of "refueling is not possible due to improper oil type" by a monitor display, voice notification, or alarm lighting. The alarm 16a arranged in the refueling lane also gives a voice notification and lights an alarm to notify the refueler that "refueling is not possible due to inappropriate oil type". After these notification operations, the determination unit 13 returns the operation to step S31.

Step S38: The determination unit 13 determines whether or not the car number read in step S30 is the car number of a rental car (including car sharing) (for example, in Japan, the use code of the car number is "wa" or "re". It is the car number of the rental car).

Step S39: In the case of a rental car, since the refueler uses the vehicle temporarily, the probability that the refueler makes a mistake in the oil type increases. Therefore, when the determination unit 13 detects the car number of the rental car, it determines that "there is a risk of an incorrect oil type" and shifts the operation to step S40. If it is not the car number of the rental car, the determination unit 13 shifts the operation to step S41.

Step S40: The determination unit 13 transfers the possibility of an oil type error to the monitoring personal computer 15, the handheld terminal 17, or the alarm device 16a. The monitoring personal computer 15, the handheld terminal 17, and the alarm device 16 notify the observer of "there is a risk of oil type error" by displaying a monitor, voice notification, or lighting an alarm. The alarm 16a arranged in the refueling lane also performs voice notification and alarm lighting to notify the refueler of "there is a risk of oil type error". After these notification operations, the determination unit 13 returns the operation to step S41.

Step S41: In the case of a vehicle having a small refueling tank such as a motorcycle, the full tank detection (refueling stopper operation) by the refueling nozzle may be inaccurate because the nozzle insertion is tilted or shallow. Therefore, when the determination unit 13 detects the vehicle number of a vehicle having a small refueling tank such as a motorcycle, the determination unit 13 suppresses the refueling amount and the refueling amount to prevent the fuel from spilling.

Step S42: At the self-service gas station GS, a monitor who handles hazardous materials monitors the refueling performed by the refueler and permits refueling in accordance with laws and regulations.
FIG. 19 is a diagram showing a screen display of the monitoring personal computer 15 and the handheld terminal 17 used by the observer.
In the figure, the screen display is composed of a multi-window such as a live display area 301, an information field 302, an analysis display area 303, a confirmation required button 304, and an operation area 305.
In the live display area 301, live images for each refueling lane taken by the image acquisition unit 11 are simultaneously displayed on a multi-screen.
In the information column 302, the vehicle number detected in step S30, the registration information inquired in step S35, the data of the refueling history 14 inquired in step S32, the information on the customer's billing decision, the past in the vehicle and the refueling person. A list of fraudulent activities is displayed.

In the analysis display area 303, as the analysis result by the learning models 200 to 260 described above, an image of an important part related to refueling failure, a status explanation of the analysis content, and a countermeasure against fraudulent activity are displayed.
In this analysis display area 303, the analysis result may be displayed for the refueling lane selected by the observer by selecting the screen of the live display area 301.
Further, the analysis results for each of a plurality of refueling lanes may be automatically switched and displayed at predetermined intervals.
Further, the analysis result of the refueling lane in which the refueler exists may be displayed preferentially.
In addition, the analysis result of the refueling lane in which fraud is occurring may be automatically selected and displayed.

Among the displays in the analysis display area 303, the confirmation required button 304 lights up or blinks to prompt the observer to confirm the analysis result that may cause a security problem. The observer operates (clicks or taps) the confirmation required button 304 after confirming the status explanation of the analysis content. The operation result of the confirmation required button 304 is sequentially recorded in the monitoring log file in the monitoring personal computer 15 together with the content of the fraudulent activity, the live image, and the date and time information.

In the operation area 305, a refueling lane selection button and an operation button (OK button, NG button) for controlling whether or not the selected refueling lane can be refueled are displayed. The observer permits the refueler to refuel by operating the OK button. In addition, the observer prohibits (or suspends) the refueling of the refueling person by operating the NG button.

Step S43: The determination unit 13 records in the refueling history 14 information regarding the vehicle number, the type of refueled oil, the amount of refueling, the date and time, the content of fraudulent activity, and whether or not refueling is possible.

By repeating the above series of monitoring support as needed until the refueling permit, the observer can carefully and safely perform the refueling permit with a small burden.

[Operation after refueling permission of image analysis system 100]
FIG. 20 is a diagram showing items of fraudulent activity detected even after refueling permission.
Hereinafter, the operation after the refueling permission is described for each item shown in the figure.

・ "Cheating that does not insert the refueling nozzle correctly"
Immediately before refueling or during refueling, the act of floating the refueling nozzle from the refueling port to delay the detection of the liquid level by the refueling nozzle and refueling to the very limit of the refueling port (hereinafter referred to as "additional refueling act") is known. In this additional refueling action, the automatic refueling stop function does not function sufficiently, and there is a possibility that fuel may spill out from the refueling port.

Therefore, the image analysis system 100 continuously detects the insertion status of the refueling nozzle even after the refueling permission is granted. As a result, it becomes possible to detect an additional refueling action in which the refueling nozzle is floated from the refueling port immediately before or during refueling.

When an additional refueling action is detected immediately before refueling, the image analysis system 100 temporarily changes the refueling permission to non-refueling. In this state, the alarm 16a alerts the refueling person and the monitoring personal computer 15 or the like notifies the observer of the fraudulent activity. After confirming that the refueling nozzle has been inserted correctly, the observer operates the monitoring personal computer 15 or the like to switch the refueling impossible to the refueling permission.

Further, when an additional refueling action is detected during refueling, the image analysis system 100 temporarily changes the refueling permission to non-refueling and controls to suspend refueling for safety. In this state, the alarm 16a alerts the refueling person and the monitoring personal computer 15 or the like notifies the observer of the fraudulent activity. After confirming that the refueling nozzle has been inserted correctly, the observer operates the monitoring personal computer 15 or the like to switch the refueling impossibility to the refueling permission and cancel the suspension of the refueling.

・ "Cheating that does not stop properly at the refueling position"
If the driver is different from the refueling person, the driver may accidentally start the vehicle when refueling. In this case, not only can fuel not be refueled normally, but there is also the possibility of an unexpected situation caused by starting the engine during refueling, or fuel spilling from the refueling port due to shaking of the vehicle body.

Therefore, the image analysis system 100 continuously detects the refueling position of the vehicle even after the refueling permission is granted. As a result, it becomes possible to detect the movement of the vehicle immediately before or during refueling.

When the movement of the vehicle is detected immediately before refueling, the image analysis system 100 temporarily changes the refueling permission to non-refueling. In this state, the alarm 16a alerts the driver and the monitoring personal computer 15 or the like notifies the observer of fraudulent activity. After confirming that the vehicle has returned to the correct refueling position, the observer operates the monitoring personal computer 15 or the like to switch the refueling impossible to the refueling permission.

Further, when the movement of the vehicle is detected during refueling, the image analysis system 100 temporarily changes the refueling permission to non-refueling and controls to suspend refueling for safety. In this state, the alarm 16a alerts the driver and the monitoring personal computer 15 or the like notifies the observer of fraudulent activity. After confirming that the vehicle has returned to the correct refueling position, the observer operates the monitoring personal computer 15 or the like to switch the refueling disabled to the refueling permission, and cancels the suspension of refueling.

・ "Cheating about the situation requiring attention"
The following situations requiring caution are situations that should be noted in the same way even after refueling permission.

(1) Use of fire, (2) Tobacco, (3) Lighter, (4) Match, (5) Smoking equipment, (6) Smoking, (7) Ignition, (8) Spark generation, (9) High temperature material, ( 10) Flammables, (11) Combustibles, (12) Explosives, (13) Dangerous goods, (14) Carrying cans, (15) Kerosene tanks, (16) Careless state of talking and screen gaze, (17) Carelessness due to mobile phones, (18) Carelessness due to mobile devices, (19) Presence of children outside the vehicle, (20) Looking away during refueling, (21) Conversation during refueling, (22) Falling of refuelers, etc. , (23) Conflict with refuelers, (24) Mayhem in refueling space, (25) Lost or forgotten refueling cap, (26) Drunk, collapsed, crouched, hidden something, etc. 27) Careless behavior, (28) Dangerous behavior Therefore, the image analysis system 100 continuously detects the situation requiring attention even after the refueling permission.

When a caution status is detected immediately before refueling, the image analysis system 100 temporarily changes the refueling permission to non-refueling. In this state, the alarm device 16a alerts the refueling person, and the monitoring personal computer 15 or the like notifies the observer of the state requiring attention. After confirming that the caution status has been resolved, the observer operates the monitoring personal computer 15 or the like to switch the refueling disabled to the refueling permission.

Further, when a cautionary situation is detected during refueling, the image analysis system 100 temporarily changes the refueling permission to non-refueling and controls to suspend refueling for safety. In this state, the alarm 16a alerts the refueling person and the monitoring personal computer 15 or the like notifies the observer of the fraudulent activity. After confirming that the caution status has been resolved, the observer operates the monitoring personal computer 15 or the like to switch the refueling disabled to the refueling permission, and cancels the suspension of refueling.

・ "Cheating that unnecessary people stay"
The act of gathering an unnecessary number of people immediately before or during refueling can be an unexpected situation. Therefore, the image analysis system 100 detects the number of people staying in the refueling space even after the refueling permission is granted.

When "a fraudulent act in which an unnecessary number of people stay" is detected immediately before refueling, the image analysis system 100 temporarily changes the refueling permission to non-refueling. In this state, the alarm device 16a alerts the resident, and the monitoring personal computer 15 or the like notifies the observer of the state requiring attention. After confirming the safety of the situation, the observer operates the monitoring personal computer 15 or the like to switch the refueling impossible to the refueling permission.

In addition, when "a fraudulent act in which an unnecessary number of people stay" is detected during refueling, the image analysis system 100 temporarily changes the refueling permission to non-refueling and controls to suspend refueling for safety. Do. In this state, the alarm device 16a alerts the resident, and the monitoring personal computer 15 or the like notifies the observer of the fraudulent activity. After confirming the safety of the situation, the observer operates the monitoring personal computer 15 or the like to switch the refueling disabled to the refueling permission, and cancels the suspension of refueling.
[Effect of Example 1]

(1) In the first embodiment, a learning model in which cheating is machine-learned for a video group of a refueling space is used. Such machine learning provides a learning model that responds to fraudulent activity that the observer perceives as dangerous or incorrect. Therefore, it is possible to supplement the observer (or on behalf of the observer) to detect fraudulent activity in the refueling space.

(2) In the first embodiment, a learning model 200 is used in which machine learning is performed on whether or not the vehicle is stopped at the refueling position for the video group of the refueling space. Therefore, it is possible to supplement the observer (or on behalf of the observer) to detect fraudulent activity in which the vehicle does not stop at the refueling position.

(3) In the first embodiment, a learning model 210 is used in which the refueler is machine-learned whether or not the refueling person has performed static electricity removal behavior for the image group of the refueling space. Therefore, it is possible to supplement the observer (or on behalf of the observer) to detect fraudulent activity in which the refueler does not perform static electricity removal behavior.

(4) In the first embodiment, a learning model 220 is used in which it is machine-learned whether or not the refueling nozzle is correctly inserted in the refueling destination for the image group of the refueling space. Therefore, it is possible to supplement the observer (or on behalf of the observer) to detect fraudulent activity regarding the insertion of the refueling nozzle.

(5) In the first embodiment, a learning model 230 in which the number of people in the refueling space is machine-learned for the video group of the refueling space is used. Therefore, it is possible to supplement the observer (or on behalf of the observer) and detect fraudulent activities in which the number of people in the refueling space exceeds a predetermined number.

(6) In the first embodiment, a learning model 240 is used in which machine learning is performed on whether or not the refueling destination is a vehicle for the video group of the refueling space. Therefore, it is possible to supplement the observer (or on behalf of the observer) and detect fraudulent acts of refueling other than the vehicle.

(7) In Example 1, about the image group of the refueling space, fire use, cigarette, lighter, match, smoking equipment, smoking, ignition, spark generation, high temperature material, flammable material, combustible material, explosive material, dangerous material, carrying Cans, kerosene tanks, calls and screen gazes, mobile phones, mobile devices, children outside the car, looking away, talking, falling, fighting, mayhem, lost or forgetting to close the refueling cap, drunk, falling, crouching, hiding something, etc. A learning model 250 is used that is machine-learned for at least one or more attention-requiring situations in the group consisting of abnormal behavior, inattention behavior, and dangerous behavior. Therefore, it is possible to supplement (or replace the observer) the observer and detect these caution situations as fraudulent activity.

(8) In the first embodiment, the vehicle number is detected for the vehicle to be refueled. Since the vehicle number is different for all vehicles, it is possible to identify the individual vehicle. By collating the vehicle number with the past refueling history 14, it can be determined that refueling is not possible when the refueling is the second time or later and the refueling of the same oil type as the past is not selected.

(9) Further, in the first embodiment, the vehicle number of the vehicle to be refueled is inquired to the registration database in which the oil type (or the vehicle type whose oil type can be specified) is registered in association with the vehicle number. If the vehicle is a registered vehicle, it can be determined that refueling is not possible if an oil type that matches the inquiry result is not selected.

(10) In Example 1, if the car number to be refueled matches the car number of the rental car (including car sharing) (in Japan, the usage code is "wa" or "re"), "there is a risk of incorrect oil type". Yes ”is determined. In general, a rental car is a temporary use for a refueler, so it is easy for a refueler to make a mistake in the oil type. Therefore, by detecting the rental car and calling attention, it is possible to prevent "oil type error" with a high probability.

(11) In the first embodiment, on the screen of the monitoring personal computer 15 (see FIG. 19) in the office room, it is possible to centrally operate the detection of fraudulent activity and the determination of whether or not to refuel.

(12) In the first embodiment, the same unified operation as that of the monitoring personal computer 15 can be performed via the handheld terminal 17. Therefore, even if the observer leaves the office to head to the refueling lane, the handheld terminal 17 can be used to centrally operate the detection of fraudulent activity and the determination of whether or not to refuel.

Subsequently, the second embodiment will be described.
Conventionally, a company that operates a predetermined number of vehicles, such as a taxi company or a transportation company, may set up a dedicated gas station in a corner of a company-owned parking lot. At such a dedicated gas station, only vehicles such as company business vehicles and company vehicles (hereinafter referred to as "qualified vehicles") are refueled at the expense of the company.

Normally, a dedicated gas station determines whether or not a vehicle is qualified by lending an ID card indicating refueling qualification to the driver of the qualified vehicle. However, it has been difficult to prevent an unqualified vehicle from illegally using an ID card to refuel for free (hereinafter referred to as "unqualified refueling fraud").

[Structure Description of Example 2]
FIG. 21 is a diagram showing a main part of an image analysis system 500 suitable for refueling monitoring of a dedicated gas station.

In the figure, the image analysis system 500 and the refueling device 510 are installed in the dedicated gas station GS2.
The image analysis system 500 includes an integrated camera unit 520, a detection unit 530, a determination unit 540, a registration unit 550 in which the vehicle number of a qualified vehicle is registered in advance, and a monitoring personal computer 560.
Here, the detection unit 530 and the determination unit 540 are realized by the functions of the image processing server 570.

The integrated camera unit 520 includes a high-sensitivity camera 521 that shoots a number plate, an infrared lighting device 522 that illuminates a dark area, a heater glass device 523 that removes cloudiness, raindrops, and internal dew condensation, and an entrance gate of a parking lot. It is composed of an installation pillar 524 for installing a camera in a refueling space and an alarm 525 for warning of fraudulent activity.

In addition, the image analysis system 500 of the second embodiment has the same configuration as the image analysis system 100 of the first embodiment (see FIGS. 1 and 2).

[Explanation of operation of Example 2]
FIG. 22 is a diagram illustrating the operation of the image analysis system 500.
Hereinafter, description will be given according to the step numbers shown in the figure.

Step S111: The detection unit 530 acquires the image information of the license plate of the vehicle to be refueled from the high-sensitivity camera 521.

Step S112: The detection unit 530 detects the vehicle number from the image information of the license plate in the same manner as in the first embodiment.

Step S113: The determination unit 540 inquires the detected vehicle number to the registration unit 550 that has registered the vehicle number of the qualified vehicle having the refueling qualification.

Step S114: If the vehicle number is not registered in the registration unit 550, the determination unit 540 determines that the vehicle number is a fraudulent act of unqualified refueling, and shifts the operation to step S115. On the other hand, when the vehicle number is registered in the registration unit 550, the dedicated gas station GS2 determines that the vehicle is qualified for refueling, and shifts the operation to step S11 of the first embodiment (FIG. 12). After that, the same operation as in the first embodiment is performed.

Step S115: The determination unit 540 determines that the unqualified refueling is a fraudulent act, outputs a refueling prohibition signal to the refueling device 510, stops the refueling, and prevents the unqualified refueling.

Step S116: The determination unit 540 informs the monitoring personal computer 560 and the alarm device 525 of the fraudulent act of unqualified refueling. The monitoring personal computer 560 outputs an unqualified refueling alarm to the observer to alert the observer. The alarm 525 issues an alarm to the refueling person and explains the situation of unqualified refueling by voice guide. The monitoring personal computer 560 records and saves the license plate image and the vehicle number as an attempted unqualified refueling.

[Effect of Example 2]
In the second embodiment, in addition to the effects of the first embodiment, the following effects are further exerted.
(1) In the second embodiment, the vehicle number is detected for the vehicle to be refueled. Since the vehicle number is different for all vehicles, it is possible to identify the individual vehicle. By inquiring the vehicle number to the registration unit 550, it becomes possible to determine the fraudulent activity of unqualified refueling.

<Supplement to the embodiment>
In Examples 1 and 2 described above, the vehicle number is used to identify the vehicle. However, the present invention is not limited to this. Vehicle identification can also be performed by using vehicle body features instead of vehicle numbers.
For example, in the case of a commercial vehicle such as a taxi, it is possible to identify the vehicle of the taxi company (whether or not it has refueling qualification, etc.) by the painting pattern of the vehicle body and the roof light.
In addition, in the case of a vehicle of a transportation company, it is possible to identify the vehicle of the transportation company (whether or not it has refueling qualification, etc.) by the logo mark of the vehicle body, the company name, and the painting pattern.
Furthermore, even for ordinary cars, unique features such as differences in the position and arrangement of vehicle inspection stickers affixed to the car body, the combination of the car body and tile wheels, codes such as QR codes attached to the car body, and damage to the car body. Or, by combining two or more of them, it is possible to identify the individual vehicle.

Further, in the above-described first and second embodiments, the vehicle number is read from the video information. However, the present invention is not limited to this. The vehicle number may be acquired not only by reading from the video information but also by information communication through a wireless tag or a wireless device mounted on the vehicle.

Further, in the above-mentioned Examples 1 and 2, the observer makes the final judgment of the refueling permission. However, the present invention is not limited to this. If it does not violate the law, the determination unit may directly control whether or not the refueling device can be refueled.

Further, in Examples 1 and 2 described above, a convolutional neural network (see FIG. 3) was used as the learning models 200 to 260. However, the present invention is not limited to this. For example, such learning models include decision tree learning, correlation rule learning, neural networks, genetic programming, inductive logic programming, support vector machines, clustering, Basian networks, reinforcement learning, expression learning, principal component analysis, and extreme learning. A learning model based on at least one of the learning machines and other machine learning techniques may be adopted.

Further, in Examples 1 and 2 described above, refueling of gasoline or light oil has been described. However, the refueling of the present invention is not limited to this. For example, in the case of a device that refuels kerosene, refueling of a gasoline vehicle or a diesel vehicle may be detected as a fraudulent activity.

Further, in the above-mentioned Examples 1 and 2, the implementation is carried out in units of the gas station GS and the dedicated refueling station GS2. However, the present invention is not limited to this. By installing a part or all of the image analysis system on the cloud network and connecting it to each refueling device via a network, it is possible to implement the invention in units of a wide area such as a regional unit or a gasoline supplier unit. become.

Further, in Examples 1 and 2 described above, the refueling history is used for finding the unsuitable oil type. However, the present invention is not limited to this. The image analysis system may extract a fraudulent addict (addicted vehicle) from the past fraudulent acts accumulated as a refueling history and create a caution list. By inquiring the store entrant (entry vehicle) to this caution list, the image analysis system can proactively detect the addict (addict vehicle) before committing fraudulent activity. As a result, the image analysis system can alert the observer in advance to the addict. In addition, the image analysis system can prioritize the detection process of fraudulent activity against the addict.

The present invention is not limited to the above-described examples, and includes various modifications. For example, the above-described embodiment has been described in detail in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to those having all the described configurations.
In addition, it is possible to replace a part of the configuration of an embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of an existing embodiment.
Further, it is possible to add / delete / replace a part of the configuration of each embodiment with another configuration.

GS ... Gas station, 10 ... Refueling device, 11 ... Video acquisition unit, 12 ... Detection unit, 13 ... Judgment unit, 14 ... Refueling history, 15 ... Monitoring personal computer, 16 ... Alarm, 16a ... Alarm, 17 ... Handheld terminal , 18 ... wireless LAN, 19 ... interface unit, 20 ... refueling control device, 21 ... network switch, 22 ... image processing server, 31 ... preprocessing unit, 32 ... input layer, 33 ... convolution layer / activation function, 34 ... Pooling layer, 35 ... Neural network, 100 ... Image analysis system, 200 ... Learning model, 210 ... Learning model, 220 ... Learning model, 230 ... Learning model, 240 ... Learning model, 250 ... Learning model, 260 ... Learning model, 301 ... Live display area, 302 ... Information column, 303 ... Analysis display area, 304 ... Confirmation required button, 305 ... Operation area, GS2 ... Dedicated refueling station, 500 ... Image analysis system, 510 ... Refueling device, 520 ... Integrated camera unit , 521 ... High-sensitivity camera, 522 ... Infrared lighting equipment, 523 ... Heater glass equipment, 524 ... Installation pillar, 525 ... Alarm, 530 ... Detection unit, 540 ... Judgment unit, 550 ... Registration unit, 560 ... Surveillance personal computer, 570 … Image processing server

Claims (13)

An image acquisition unit that acquires an image of the refueling space,
Using a learning model in which fraudulent activity is machine-learned for the video group of the refueling space, a detection unit that detects the fraudulent activity in the video acquired by the video acquisition unit, and a detection unit.
A determination unit that generates information on whether or not refueling is possible based on the detection result of the fraudulent activity,
An image analysis system characterized by being equipped with. In the image analysis system according to claim 1,
The detector is
Using a learning model in which the vehicle is machine-learned whether or not the vehicle is stopped at the refueling position for the video group of the refueling space, a fraudulent act in which the vehicle does not stop at the refueling position is detected in the video acquired by the video acquisition unit. An image analysis system characterized by learning. In the image analysis system according to any one of claims 1 and 2.
The detector is
Using a learning model that machine-learns whether or not the refueling person has performed static electricity removal behavior for the video group of the refueling space, the refueling person does not perform static electricity removal behavior for the video acquired by the video acquisition unit. An image analysis system characterized by detection. In the image analysis system according to any one of claims 1 to 3.
The detector is
Using a learning model that machine-learned whether or not the refueling nozzle was correctly inserted in the refueling destination for the video group of the refueling space, the video acquired by the video acquisition unit was fraudulently engaged in inserting the refueling nozzle. An image analysis system characterized by detecting. In the image analysis system according to any one of claims 1 to 4.
The detector is
Using a learning model in which the number of people in the refueling space is machine-learned for the video group of the refueling space, the video acquired by the video acquisition unit is characterized by detecting fraudulent activities in which the number of people in the refueling space exceeds a predetermined number. Image analysis system. In the image analysis system according to any one of claims 1 to 5,
The detector is
Using a learning model in which it is machine-learned whether or not the refueling destination is a vehicle for the image group of the refueling space, the image acquired by the image acquisition unit is characterized in that a fraudulent act of refueling other than the vehicle is detected. Image analysis system. In the image analysis system according to any one of claims 1 to 6,
The detector is
About the video group of the refueling space
Use of fire, cigarettes, lighters, matches, smoking equipment, smoking, ignition, sparks, high temperatures, flammables, combustibles, explosives, dangerous goods, carrying cans, kerosene tanks, calls and screen gazes, mobile phones, mobile devices Machine-learned about at least one or more alert situations in the group consisting of children outside the vehicle, looking away, talking, falling, fighting, mayhem, lost or forgetting to close the refueling cap, abnormal behavior, careless behavior, and dangerous behavior. An image analysis system characterized in that the video acquired by the video acquisition unit is detected as a fraudulent act by using a learning model. In the image analysis system according to any one of claims 1 to 7.
The detector is
Detects the vehicle number or characteristics of the vehicle to be refueled,
The determination unit
An image characterized in that the vehicle number or the feature detected by the detection unit is collated with the refueling history, and if refueling is performed for the second time or later and refueling of the same oil type as in the past is not selected, it is determined that refueling is not possible. Analysis system. In the image analysis system according to any one of claims 1 to 8.
The detector is
Detects the vehicle number or characteristics of the vehicle to be refueled,
The determination unit
The oil type or vehicle type is registered in association with the vehicle number or feature, and the vehicle number or feature read by the detection unit is inquired from the registration database to acquire information on the oil type or vehicle type, and the information-acquired oil. An image analysis system characterized in that if an oil type suitable for a type or vehicle type is not selected, it is determined that refueling is not possible. In the image analysis system according to any one of claims 1 to 9,
The detector is
Detects the vehicle number of the vehicle to be refueled and
The determination unit
An image analysis system characterized in that if the vehicle number detected by the detection unit matches the vehicle number of the rental car, it is determined that there is a risk of an incorrect oil type. Registration department for registering the car number or characteristics of vehicles that are qualified to refuel,
A detector that detects the vehicle number or characteristics from the vehicle to be refueled,
If the vehicle number or feature of the vehicle detected by the detection unit is not registered in the registration unit, the determination unit that determines that the vehicle is a fraudulent act of unqualified refueling, and
An image analysis system characterized by being equipped with. An image acquisition step to acquire an image of the refueling space, and
Using a learning model in which fraudulent activity is machine-learned for the video group of the refueling space, a detection step for detecting the fraudulent activity in the video acquired by the video acquisition step and a detection step
A determination step for generating information on whether or not refueling is possible based on the detection result of the fraudulent activity,
An image analysis method characterized by being equipped with. Registration steps for registering vehicle numbers or features of vehicles that are eligible for refueling,
A detection step that detects the vehicle number or feature from the vehicle to be refueled,
If the vehicle number or feature of the vehicle detected by the detection step is not registered in the registration step, the determination step of determining that the vehicle is a fraudulent act of unqualified refueling, and
An image analysis method characterized by being equipped with.

Images