WO2024236748A1 - 異常判定装置、および、異常判定方法 - Google Patents

異常判定装置、および、異常判定方法 Download PDF

Info

Publication number
WO2024236748A1
WO2024236748A1 PCT/JP2023/018362 JP2023018362W WO2024236748A1 WO 2024236748 A1 WO2024236748 A1 WO 2024236748A1 JP 2023018362 W JP2023018362 W JP 2023018362W WO 2024236748 A1 WO2024236748 A1 WO 2024236748A1
Authority
WO
WIPO (PCT)
Prior art keywords
unit
state
value
neural network
output
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2023/018362
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
拓也 上杉
将人 後町
勝己 高橋
廣愛 浅見
洋 酒巻
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Mitsubishi Electric Corp
Original Assignee
Mitsubishi Electric Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Mitsubishi Electric Corp filed Critical Mitsubishi Electric Corp
Priority to JP2025520317A priority Critical patent/JP7696532B2/ja
Priority to PCT/JP2023/018362 priority patent/WO2024236748A1/ja
Publication of WO2024236748A1 publication Critical patent/WO2024236748A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/10—Interfaces, programming languages or software development kits, e.g. for simulating neural networks

Definitions

  • the disclosed technology relates to an abnormality determination technology that determines the condition of a subject.
  • abnormality determination techniques there are those that determine the condition of a person to be determined by using the results of measurements taken by a sensor device (such as a camera, radar, or infrared sensor).
  • a sensor device such as a camera, radar, or infrared sensor.
  • abnormality judgment technologies for example, in abnormality judgment technologies that judge the state of a subject of judgment, such as a driver (occupant) of a vehicle (mobile body), since this concerns the safety of the subject of judgment, it is desirable to make the judgment with high accuracy to avoid erroneous judgments.
  • Deep learning uses a mathematical model called a neural network (NN).
  • a neural network outputs results through layers such as an input layer, a hidden layer (intermediate layer), and an output layer.
  • Patent Document 1 discloses an identification device that uses a convolutional neural network (CNN), which is a type of neural network, to identify when a driver of a mobile object is in an abnormal state. Specifically, in the process of identifying the driver's condition (e.g., whether or not the driver is in an abnormal state) based on an acquired image, the identification device of Patent Document 1 obtains the driver's skeletal information from the image using a convolutional neural network. The identification device of Patent Document 1 improves the accuracy of the driver's skeletal information by using a neural network.
  • CNN convolutional neural network
  • the neural network extracts and uses a large number of judgment indices indicating the state of the person being judged from an image (for example, a large number of types of indices such as the open/closed state of the eyelids, the number of blinks, the position of the face, the state of brain waves, the heat dissipation state, driving performance (reaction time, etc.)).
  • a large number of judgment indices indicating the state of the person being judged from an image (for example, a large number of types of indices such as the open/closed state of the eyelids, the number of blinks, the position of the face, the state of brain waves, the heat dissipation state, driving performance (reaction time, etc.)).
  • Patent Document 1 merely outputs skeletal information from an image via a neural network, and is not capable of solving the above-mentioned problems.
  • the present disclosure aims to solve the above problems and to enable anomaly detection technology to quickly output highly accurate detection results using a neural network.
  • the abnormality determination device of the present disclosure is an image acquisition unit that acquires and outputs an image; a neural network unit having a plurality of layers in a neural network, acquiring an image output by the image acquisition unit, and outputting a judgment result indicating a state of a judgment target included in the image; an output storage unit that stores output data output by the neural network unit; a state comparison unit that prestores a reference value that is a reference for an input value to a first layer that is at least one of the multiple layers of the neural network unit, and judges a change in the state of the object to be judged using a difference value between the reference value and a current value that is a value that is to be newly input to the first layer; a processing control unit that instructs the neural network unit not to execute processing in the first layer and subsequent layers when the state comparison unit determines that the state of the object to be determined has not changed, and instructs the neural network unit to output the output data stored in the output storage unit; Equipped with:
  • the present disclosure has the effect of enabling anomaly detection technology to quickly output highly accurate detection results using a neural network.
  • FIG. 1 is a diagram showing an example of the configuration of an abnormality warning device 100A and an abnormality determination device 1000A according to a first embodiment of the present disclosure.
  • FIG. 2 is a flowchart showing an example of the processing of the abnormality warning device 100A and the abnormality determination device 1000A according to the first embodiment of the present disclosure.
  • FIG. 3 is a flowchart showing an example of more detailed processing of the abnormality warning device 100A and the abnormality determination device 1000A according to the first embodiment of the present disclosure.
  • FIG. 4 is a diagram showing an example of the configuration of an abnormality warning device 100B and an abnormality determination device 1000B according to the second embodiment of the present disclosure.
  • FIG. 1 is a diagram showing an example of the configuration of an abnormality warning device 100A and an abnormality determination device 1000A according to a first embodiment of the present disclosure.
  • FIG. 2 is a flowchart showing an example of the processing of the abnormality warning device 100A and the abnormality determination device 1000A according to the first embodiment of the present disclosure.
  • FIG. 5 is a diagram showing an example of the internal configuration of the abnormality warning device 100B and the feature extraction unit 1300B in the abnormality determination device 1000B.
  • FIG. 6 is a diagram showing an example of the internal configuration of the abnormality warning device 100B and the abnormality state determination unit 1500B in the abnormality determination device 1000B.
  • FIG. 7 is a flowchart showing an example of processing by the abnormality warning device 100B and the abnormality determination device 1000B according to the second embodiment of the present disclosure.
  • FIG. 8 is a flowchart showing an example of more detailed processing of the abnormality warning device 100B and the abnormality determination device 1000B according to the second embodiment of the present disclosure.
  • FIG. 9 is a flowchart showing an example of a process of the feature extraction unit 1300B in a case where the state comparison unit 1030 (first state comparison unit 1030B) in the feature extraction unit 1300B according to the second embodiment of the present disclosure is the image state comparison unit 1341.
  • FIG. 10 is a flowchart illustrating an example of a process of the feature extraction unit 1300B in the second embodiment of the present disclosure when the state comparison unit 1030 (first state comparison unit 1030B) is a convolutional layer state comparison unit.
  • FIG. 11 is a flowchart illustrating an example of a process performed by the feature extraction unit 1300B in the second embodiment of the present disclosure when the state comparison unit 1030 (first state comparison unit 1030B) is a pooling layer state comparison unit.
  • FIG. 10 is a flowchart illustrating an example of a process of the feature extraction unit 1300B in a case where the state comparison unit 1030 (first state comparison unit 1030B) in the feature extraction unit 1300B according to the second embodiment
  • FIG. 12 is a flowchart showing an example of processing of the abnormal state determination unit 1500B in the case where the state comparison unit (second state comparison unit 1540B) in the abnormal state determination unit 1500B according to the second embodiment of the present disclosure is the first fully connected layer state comparison unit 1541.
  • FIG. 13 is a flowchart showing an example of processing by the abnormal state determination unit 1500B in the second embodiment of the present disclosure when the state comparison unit (second state comparison unit 1540B) in the abnormal state determination unit 1500B is a second fully connected layer state comparison unit.
  • FIG. 14 is a diagram showing an example of the configuration of an abnormality warning device 100C and an abnormality determination device 1000C according to the third embodiment of the present disclosure.
  • FIG. 15 is a diagram showing an example of the internal configuration of the feature extraction unit 1300C in the abnormality warning device 100C and the abnormality determination device 1000C.
  • FIG. 16 is a diagram showing an example of the internal configuration of the abnormality warning device 100C and the abnormality state determination unit 1500C in the abnormality determination device 1000C.
  • FIG. 17 is a flowchart showing an example of the processing of the abnormality warning device 100C and the abnormality determination device 1000C according to the third embodiment of the present disclosure.
  • FIG. 18 is a flowchart showing a detailed first example of the processing of the feature extraction unit 1300C in the abnormality warning device 100C and the abnormality determination device 1000C according to the third embodiment of the present disclosure.
  • FIG. 19 is a flowchart showing a detailed first example of the processing of the abnormal state determination unit 1500C in the abnormality warning device 100C and the abnormality determination device 1000C according to the third embodiment of the present disclosure.
  • FIG. 20 is a flowchart showing a second detailed example of the processing of the feature extraction unit 1300C in the abnormality warning device 100C and the abnormality determination device 1000C according to the third embodiment of the present disclosure.
  • FIG. 21 is a flowchart showing a second detailed example of the processing of the abnormal state determination unit 1500C in the abnormality warning device 100C and the abnormality determination device 1000C according to the third embodiment of the present disclosure.
  • FIG. 22 is a diagram illustrating a first example of a hardware configuration for realizing the functions of the present disclosure.
  • FIG. 23 is a diagram illustrating a second example of a hardware configuration for realizing the functions of the present disclosure.
  • Embodiment 1 In the first embodiment, one mode of a basic configuration of the present disclosure will be described.
  • FIG. 1 is a diagram showing an example of the configuration of an abnormality warning device 100A and an abnormality determination device 1000A according to a first embodiment of the present disclosure.
  • the abnormality warning device 100A acquires an image, uses the image to judge the state of an object to be judged that is captured in the image, and outputs a warning if the state of the object to be judged indicates an abnormality.
  • the abnormality warning device 100 shown in FIG. 1 includes an abnormality determination device 1000A and a warning output unit 2000.
  • the abnormality determination device 1000A obtains an image and uses the image to output the state of the object to be determined captured in the image.
  • the state of the object to be determined is output data including a value in the form of, for example, a determination value or a probability value.
  • the abnormality determination device 1000A includes an image acquisition unit 1010, a neural network unit 1020, a state comparison unit 1030, a processing control unit 1040, and an output storage unit 1050.
  • the image acquisition unit 1010 acquires and outputs an image.
  • the image acquisition unit 1010 acquires an image by inputting an image (moving image or still image) captured by a camera, for example.
  • the camera is, for example, an imaging device installed inside the vehicle cabin, which captures an image of a determination target present inside the vehicle cabin (the determination target is, for example, a living body such as a passenger including the driver) and outputs the captured image.
  • the image is a moving image
  • the moving image is divided by a camera into still images (frames) at regular time intervals, and the divided images are input to the image acquisition unit 1010 .
  • the neural network unit 1020 is composed of a neural network.
  • the neural network unit 1020 has multiple layers in a neural network, acquires the image output by the image acquisition unit 1010, and outputs output data that is a judgment result indicating the state of the judgment target contained in the image.
  • the multiple layers in a neural network are an input layer, a hidden layer (an intermediate layer), and an output layer, and more specifically, layers such as a convolutional layer and a pooling layer.
  • a neural network with many layers (deep) is particularly called a DNN (Deep Neural Network), and its derivatives include, for example, a CNN (Convolutional Neural Network) and an RNN (Recurrent Neural Network).
  • CNN is often used in the fields of object recognition and image recognition, while RNN is often used in time series processing, voice recognition, natural language processing, and the like.
  • the neural network in the present disclosure is applicable regardless of the form of the neural network. In this description, the form of CNN will be described as an example.
  • the state comparison unit 1030 judges a change in the state of the object to be judged.
  • the state comparison unit 1030 pre-stores a reference value that is a standard for the input value to the first layer, which is at least one of the multiple layers of the neural network unit 1020, and determines a change in the state of the object to be judged using the difference value between the reference value and the current value, which is the value that is to be newly input to the first layer.
  • the first layer may be a predetermined one of all layers in the neural network unit 1020, a predetermined portion of all layers, or all layers.
  • the reference value is a previous value, which is an input value of at least one layer, the first layer, stored as a result of previous processing by multiple layers of the neural network unit 1020, and is used to obtain a difference value between the previous value and a current value, which is an input value that is about to be newly input to the first layer.
  • a previous value which is an input value of at least one layer, the first layer
  • a current value which is an input value that is about to be newly input to the first layer.
  • the reference value can use typical processing results that have been learned and modeled in advance.
  • the reference value may be a typical output value of a pre-modeled intermediate layer in a neural network.
  • the reference value may use a part of the results determined based on advance information.
  • the partial results determined based on the prior information are assumed to be, for example, pixels that have a high probability of indicating the presence of a head. Also, for example, pixels that are known to have a large change but low importance, such as the background, may not be used.
  • the reference value may be the output value of a predetermined node among the nodes included in the neural network. This allows for reduced memory usage and data handling, resulting in faster processing speeds.
  • the state comparison unit 1030 uses the difference value and a pre-stored threshold value to determine the magnitude of the difference between the reference value and the current value, and determines a change in the state of the object to be determined based on the result. Specifically, for example, in units of images, if the sum of squares of the difference values is smaller than a pre-stored threshold value, the state comparison unit 1030 determines that the state of the determination target is unchanged. More specifically, the state comparison unit 1030 determines that the state of the determination target is unchanged when the sum of the absolute values of the difference values is smaller than a pre-stored threshold value, for example, for each image.
  • the processing control unit 1040 In response to changes in the state of the object to be judged, the processing control unit 1040 issues a command to restrict the current processing operation in the neural network unit 1020, and also issues a command to output the output data output by the previous processing of the neural network unit 1020.
  • the processing control unit 1040 instructs the neural network unit 1020 not to execute processing in at least one layer out of the multiple layers, that is, the first layer or subsequent layers, and also instructs the neural network unit 1020 to output the output data stored in the output storage unit 1050.
  • the output data is, for example, a judgment value that is a value indicating the state of the person to be judged. In the embodiment of the present disclosure, the judgment value is also referred to as a probability value that includes a probabilistic element.
  • the output storage section 1050 stores the output data (decision value (probability value)) output by the neural network section 1020 .
  • the output storage unit 1050 receives a command from the process control unit 1040, it outputs the stored output data.
  • the output storage unit 1050 outputs the output data to the warning output unit 2000, for example.
  • the output storage unit 1050 holds the latest output data output as a result of processing performed by the neural network unit 1020. For example, every time the output storage unit 1050 obtains output data output by the neural network unit 1020, it erases the output data previously stored and stores the latest output data.
  • the alarm output unit 2000 acquires output data, and outputs an alarm signal based on the output data to an alarm device (not shown) or the like.
  • the warning output unit 2000 acquires output data that is a judgment value indicating the drowsy state or dozing state of the person to be judged, and outputs a warning to the person to be judged in accordance with the judgment value.
  • the abnormality determination device 1000A shown in FIG. 1 is shown as being configured not to include the alarm output unit 2000, but may be configured to include the alarm output unit 2000. When configured in this manner, the abnormality determination device 1000A is equivalent to the abnormality warning device 100A shown in FIG. 14. In the following explanation, the abnormality determination device 1000A will be described as being configured to include the alarm output unit 2000, except in cases where it is necessary to distinguish between the abnormality warning device 100A and the abnormality determination device 1000A.
  • abnormality determination device 1000A may be configured to include a control unit (not shown), a storage unit (not shown), and a communication unit (not shown).
  • a control unit (not shown) controls the entire abnormality determination device 1000A and each of its components.
  • the control unit (not shown) starts up the abnormality determination device 1000A in response to, for example, an external command.
  • a storage unit (not shown) stores each piece of data used in the abnormality determination device 1000A.
  • the storage unit stores, for example, outputs (output data) from each component in the abnormality determination device 1000A, and outputs data requested by each component to the component that has made the request.
  • a communication unit (not shown) communicates with an external device. For example, communication is performed between the abnormality determination device 1000A and an imaging device such as an in-vehicle camera. In addition, for example, if the abnormality determination device 1000A does not have a display unit or an audio output unit, communication is performed between the abnormality determination device 1000A and an external device such as a display unit or an audio output device.
  • FIG. 2 is a flowchart showing an example of the processing of the abnormality warning device 100A and the abnormality determination device 1000A according to the first embodiment of the present disclosure.
  • the abnormality determination device 1000A starts the process shown in FIG. 2 when an image is input from a camera, for example.
  • Abnormality determination device 1000A executes image acquisition processing (step ST100). In the image acquisition process, the image acquisition unit 1010 of the abnormality determination device 1000A acquires and outputs an image.
  • Abnormality determination device 1000A executes a state storage and state comparison process (step ST200).
  • the state comparison unit 1030 of the abnormality determination device 1000A stores the previous value, which is the input value of at least one layer, that is, the first layer, among the processing results in all layers of the neural network unit 1020 for at least the first time after the start of processing.
  • the state comparison unit 1030 does not perform state comparison processing for the first processing, but simply stores the previous value, but judges whether there has been a change in the state of the object to be judged from the second processing onwards.
  • the state comparison unit 1030 pre-stores a reference value that is a standard for the input value to the first layer, which is at least one of the multiple layers of the neural network unit 1020, and determines a change in the state of the object to be judged using the difference value between the reference value and the current value, which is the value that is to be newly input to the first layer.
  • Abnormality determination device 1000A executes a process control process (step ST300).
  • the process control unit 1040 of the abnormality determination device 1000A issues a command to restrict the current processing operation in the neural network unit 1020 in response to a change in the state of the object to be determined, and also issues a command to output the output data output by the previous processing of the neural network unit 1020.
  • the processing control unit 1040 instructs the neural network unit 1020 not to execute processing in at least one layer out of the multiple layers, that is, the first layer or subsequent layers, and also instructs the processing control unit 1040 to output the output data stored in the output storage unit 1050.
  • Abnormality determination device 1000A executes a state output process (step ST400).
  • the neural network unit 1020 or the output storage unit 1050 of the abnormality determination device 1000A outputs output data.
  • the neural network unit 1020 outputs output data to the warning output unit 2000.
  • the output storage unit 1050 receives a command from the process control unit 1040 , it outputs the latest stored output data to the alarm output unit 2000 .
  • Abnormality determination device 1000A executes an alarm output process (step ST500).
  • the alarm output unit 2000 of the abnormality determination device 1000A acquires output data and outputs an alarm signal based on the output data to an alarm device (not shown) etc.
  • the alarm output unit 2000 determines whether to output an alarm based on a determination value included in the output data, and when it determines to output an alarm, outputs an alarm signal to an alarm device (not shown) etc.
  • step ST500 When the abnormality determination device 1000A executes the process of step ST500, The series of processes shown in FIG. 2 ends, and the process is repeated from step ST100. For example, when the camera is turned off, the abnormality determination device 1000A is also turned off in conjunction with the camera.
  • FIG. 3 is a flowchart showing an example of more detailed processing of the abnormality warning device 100A and the abnormality determination device 1000A according to the first embodiment of the present disclosure.
  • abnormality determination device 1000A starts the process shown in FIG. 3, first, similarly to step ST100 described above, it executes image acquisition processing (step ST100).
  • Abnormality determination device 1000A executes a storage process (step ST201).
  • the state comparison unit 1030 of the abnormality determination device 1000A stores the previous value, which is the input value of at least one layer, the first layer, among the processing results in all layers of the neural network unit 1020 for at least the first time after the start of processing. Furthermore, the state comparison unit 1030 stores the input values of the first layer each time processing is performed in the first layer.
  • the state comparison unit 1030 of the abnormality determination device 1000A executes a previous storage determination process to determine whether the data has been stored previously (step ST202).
  • the state comparison unit 1030 of the abnormality determination device 1000A executes a comparison process between the current value and the previous value (step ST203).
  • the state comparison unit 1030 calculates the difference (difference value) between the current value, which is the value that is about to be newly input to the first layer, and the previous value (reference value).
  • State comparison section 1030 of abnormality determination device 1000A determines a change in the state of the object to be determined using the difference value (step ST204).
  • the state comparison unit 1030 uses the difference value and a pre-stored threshold value to determine the magnitude of the difference between the reference value and the current value, and determines a change in the state of the object to be determined based on the result. Specifically, for example, in units of images, if the sum of squares of the difference values is smaller than a pre-stored threshold value, the state comparison unit 1030 determines that the state of the determination target is unchanged. More specifically, the state comparison unit 1030 determines that the state of the determination target is unchanged when the sum of the absolute values of the difference values is smaller than a pre-stored threshold value, for example, for each image.
  • step ST204 determines that the difference is equal to or greater than the threshold value and that the state of the object to be determined has changed.
  • the processing control unit 1040 of the abnormality determination device 1000A executes a normal processing command process (step ST301).
  • the processing control unit 1040 outputs a command to execute processing in the first layer of the neural network unit 1020.
  • the neural network unit 1020 of the abnormality determination device 1000A executes output processing (step ST401).
  • the neural network unit 1020 outputs the output data to the alarm output unit 2000.
  • Output storage unit 1050 of abnormality determination device 1000A executes an output storage process (step ST402).
  • the output storage section 1050 stores the output data output by the neural network section 1020 .
  • the processing control unit 1040 of the abnormality determination device 1000A executes an omission processing command processing (step ST302).
  • the processing control unit 1040 instructs the neural network unit 1020 not to execute processing in the first layer or subsequent layers, which is at least one layer among the multiple layers, and also instructs the output storage unit 1050 to output the output data stored therein.
  • the output storage unit 1050 of the abnormality determination device 1000A executes a stored data output process (step ST403).
  • the output storage unit 1050 receives a command from the process control unit 1040, it outputs the stored output data to the alarm output unit 2000.
  • the alarm output unit 2000 executes an alarm output process (step ST500).
  • the alarm output unit 2000 determines whether to output an alarm based on the judgment value contained in the output data, similar to the processing of step ST500 already described, and if it determines to output an alarm, outputs an alarm signal to an alarm device or the like (not shown).
  • abnormality determination device 1000A executes the process of step ST500, it ends the series of processes shown in FIG. 3 and repeats the process from step ST100. For example, when the camera is turned off, the abnormality determination device 1000A is also turned off in conjunction with the camera.
  • the configuration and processing described above allows the anomaly determination device to omit processing of layers in the neural network depending on the conditions.
  • the abnormality determination device of the present disclosure has the following configuration. "An image acquisition unit that acquires and outputs an image; a neural network unit having a plurality of layers in a neural network, acquiring an image output by the image acquisition unit, and outputting a judgment result indicating a state of a judgment target included in the image; an output storage unit that stores output data (judgment value (probability value)) output by the neural network unit; a state comparison unit that prestores a reference value that is a reference for an input value to a first layer that is at least one of the multiple layers of the neural network unit, and judges a change in the state of the object to be judged using a difference value between the reference value and a current value that is a value that is to be newly input to the first layer; a processing control unit that instructs the neural network unit not to execute processing in the first layer and subsequent layers when the state comparison unit determines that the state of the object to be determined has not changed, and instructs the neural network unit to output the output data stored in the output storage unit
  • the abnormality determination method of the present disclosure has the following configuration.
  • the image acquisition unit outputs the acquired image to the neural network unit, the neural network unit having a plurality of layers in a neural network acquires the image output by the image acquisition unit, and outputs a judgment result indicating a state of a judgment target included in the image;
  • an output storage unit stores the output data (judgment value (probability value)) output by the neural network unit;
  • a state comparison unit pre-stores a reference value that is a reference for an input value to a first layer that is at least one of the multiple layers of the neural network unit, and judges a change in the state of the object to be judged using a difference value between the reference value and a current value that is a value that is to be newly input to the first layer;
  • the processing control unit instructs the neural network unit not to execute processing in the first layer and subsequent layers, and instructs the neural network unit to output the output data stored in the output storage unit.
  • the abnormality determination device of the present disclosure is further configured as follows. "The anomaly determination device, wherein the reference value is a previous value stored for each layer as a result of the previous processing in the neural network.” As a result, the present disclosure has the effect of providing an abnormality determination device in an abnormality determination technique that can quickly output highly accurate determination results using a neural network. Furthermore, the present disclosure achieves the same effect as the above by applying the above configuration to the abnormality determination method.
  • the abnormality determination device of the present disclosure is further configured as follows. "The reference value is a typical output value of a pre-modeled intermediate layer in the neural network. An abnormality determination device characterized by the above. As a result, the present disclosure has the effect of providing an abnormality determination device in an abnormality determination technique that can quickly output highly accurate determination results using a neural network. Furthermore, the present disclosure achieves the same effect as the above by applying the above configuration to the abnormality determination method.
  • the abnormality determination device of the present disclosure is further configured as follows. "The reference value is an output value of a predetermined node among the nodes included in the neural network. An abnormality determination device characterized by the above. As a result, the present disclosure has the effect of providing an abnormality determination device in an abnormality determination technique that can quickly output highly accurate determination results using a neural network. Furthermore, the present disclosure achieves the same effect as the above by applying the above configuration to the above abnormality determination method.
  • the abnormality determination device of the present disclosure is further configured as follows. "The state comparison unit is a change in the state of the object to be determined based on a result of determining whether a difference between the reference value and the current value is greater or smaller using the difference value and a pre-stored threshold value; An abnormality determination device characterized by the above. As a result, the present disclosure has the effect of providing an abnormality determination device in an abnormality determination technique that can quickly output highly accurate determination results using a neural network. Furthermore, the present disclosure achieves the same effect as the above by applying the above configuration to the abnormality determination method.
  • the abnormality determination device of the present disclosure is further configured as follows. "The state comparison unit is When the sum of squares of the difference values is smaller than a pre-stored threshold value for each image, it is determined that the state of the object to be determined is unchanged.
  • the present disclosure has the effect of providing an abnormality determination device in an abnormality determination technique that can quickly output highly accurate determination results using a neural network. Furthermore, the present disclosure achieves the same effect as the above by applying the above configuration to the abnormality determination method.
  • the abnormality determination device of the present disclosure is further configured as follows. "The state comparison unit is When the sum of the absolute values of the difference values is smaller than a pre-stored threshold value for each image, it is determined that the state of the object to be determined is unchanged.
  • the present disclosure has the effect of providing an abnormality determination device in an abnormality determination technique that can quickly output highly accurate determination results using a neural network. Furthermore, the present disclosure achieves the same effect as the above by applying the above configuration to the abnormality determination method.
  • the abnormality determination device of the present disclosure is further configured as follows. "The present invention further includes an alarm output unit that acquires the output data, which is a judgment value indicating a drowsy state or a dozing state of a person to be judged, and outputs an alarm to the person to be judged according to the judgment value.
  • the present disclosure has the effect of providing an abnormality determination device in an abnormality determination technique that can quickly output highly accurate determination results using a neural network. Furthermore, the present disclosure achieves the same effect as the above by applying the above configuration to the abnormality determination method.
  • Embodiment 2 a form in which the basic mechanism of the present disclosure is applied to layers in the neural networks of a feature extraction unit and an abnormal state determination unit will be described.
  • FIG. 4 is a diagram showing an example of the configuration of an abnormality warning device 100B and an abnormality determination device 1000B according to the second embodiment of the present disclosure.
  • the abnormality warning device 100B includes an abnormality determination device 1000B and a warning output unit 2000.
  • the abnormality determination device 1000B obtains an image, and uses the image to output the state of the determination target captured in the image.
  • the abnormality determination device 1000B shown in FIG. 4 includes an image acquisition unit 1100B, a feature extraction unit 1300B, and an abnormal state determination unit 1500B.
  • the neural network unit already described is configured to include, as will be described later, a first neural network unit 1320 and a second neural network unit 1520.
  • First neural network unit 1320 is included in feature extraction unit 1300B
  • second neural network unit 1520 is included in abnormal state determination unit 1500B.
  • the output storage unit already described is configured to include a first output storage unit 1390 and a second output storage unit.
  • the first output storage unit 1390 is included in the feature extraction unit 1300B, and the second output storage unit is included in the abnormal state determination unit 1500B.
  • the state comparison unit already described is configured to include a first state comparison unit 1340 and a second state comparison unit 1540.
  • First state comparison unit 1340 is included in feature extraction unit 1300B
  • second state comparison unit 1540 is included in abnormal state determination unit 1500B.
  • the process control unit already described is configured to include a first process control unit 1380 and a second process control unit 1580.
  • the first process control unit 1380 is included in the feature extraction unit 1300B
  • the second process control unit 1580 is included in the abnormal state determination unit 1500B.
  • the image acquisition unit 1100B is similar to the image acquisition unit 1100A already described, a detailed description of the image acquisition unit 1100B will be omitted here.
  • the feature extraction unit 1300B uses the image to output a feature map that represents the characteristic state of the subject contained in the image. For example, the feature extraction unit 1300B extracts parts of the image that show signs of drowsiness, such as the eyelids and eyeballs, and generates a feature map that represents the state of the subject that is characteristic of an abnormal state such as drowsiness.
  • FIG. 5 is a diagram showing an example of the internal configuration of the abnormality warning device 100B and the feature extraction unit 1300B in the abnormality determination device 1000B.
  • the feature extraction unit 1300B shown in FIG. 5 is configured to include a neural network unit (first neural network unit 1320), a state comparison unit (first state comparison unit 1340B), a processing control unit (first processing control unit 1380B), and an output storage unit (first output storage unit 1390).
  • the first neural network unit 1320 has multiple layers that are part of the layers that make up the neural network, acquires the image output by the image acquisition unit 1100, and outputs a feature map that represents the characteristic state of the object to be judged contained in the image.
  • the first neural network unit 1320 shown in FIG. 5 includes an image branching unit 1321, a convolution layer unit 1322, a pooling layer unit 1325, and an image combination unit 1328.
  • the image branching unit 1321 branches and inputs an image to multiple nodes in a layer of a neural network.
  • the image is branched according to the number of subsequent convolutional layers and pooling layers. In FIG. 5, the image is branched into two.
  • the convolution layer unit 1322 performs filtering to extract characteristic parts of the object to be determined in the image.
  • the convolution layer unit 1322 shown in FIG. 5 includes a first convolution layer 1323 and a second convolution layer 1324.
  • the first convolutional layer 1323 and the second convolutional layer 1324 perform filtering to extract face (body) parts for detecting a drowsy state, for example, by convolution processing (cross-correlation processing) using pre-prepared convolution filters of size 3x3 or 5x5.
  • the pooling layer unit 1325 shown in FIG. 5 includes a first pooling layer 1326 and a second pooling layer 1327 .
  • the first pooling layer 1326 and the second pooling layer 1327 generate an image relating to features that are robust against image position, for example, by calculating the maximum or average value for each predetermined region.
  • the first neural network unit 1320 shown in FIG. 5 includes two pairs of convolutional layers and pooling layers, but it is also effective to include one pair or three or more pairs.
  • the first pooling layer and the second pooling layer 1327 may be followed by two or more convolution layers and two or more pooling layers.
  • a normalized linear unit layer (activation function) or the like may be included after the convolution layer.
  • the image combination unit 1328 combines multiple images output through the convolution layer and the pooling layer.
  • the image combining unit 1328 extracts areas that indicate signs of drowsiness, such as the eyelids and eyeballs, and generates a feature map.
  • the first state comparison unit 1340B prestores a reference value, which is the standard for the input value to each layer in the first neural network unit 1320, and uses the difference between the reference value and the current value, which is the value that is about to be newly input to the layer, to determine a change in the state of the object to be determined.
  • the reference value is a previous value, which is an input value for each layer, stored as a result of the previous processing by the multiple layers of the first neural network unit 1320, and is used to obtain a difference value between the previous value and a current value, which is an input value that is about to be newly input to the layer.
  • the first state comparison unit 1340 does not perform state comparison processing for the first processing (processing for the first image) and simply stores the value, but in the second and subsequent processing (processing for the second image), it determines changes in the state of the object to be judged.
  • the following values may be stored and used as the reference values.
  • the reference value can use typical processing results that have been learned and modeled in advance.
  • the reference value may be a typical output value of a pre-modeled intermediate layer in a neural network.
  • the reference value may use a part of the results determined based on advance information.
  • the partial results determined based on the prior information are assumed to be, for example, pixels that have a high probability of indicating the presence of a head. Also, for example, pixels that are known to have a large change but low importance, such as the background, may not be used.
  • the reference value may be the output value of a predetermined node among the nodes included in the neural network. This allows for reduced memory usage and data handling, resulting in faster processing speeds.
  • the first state comparison section 1340 uses the difference value and a pre-stored threshold value to determine the magnitude of the difference between the reference value and the current value, and determines a change in the state of the object to be determined based on the result. Specifically, the first state comparing section 1340 determines that the state of the determination target is unchanged when the sum of squares of the difference values is smaller than a pre-stored threshold value for each image. More specifically, the first state comparing section 1340 determines that the state of the determination target is unchanged when the sum of the absolute values of the difference values recorded in image units is smaller than a pre-stored threshold value.
  • "storing" means holding the values of the two-dimensional images (feature maps) of the respective input sources.
  • comparison means for example, taking the absolute value of the difference for each element (pixel) between a stored two-dimensional image (from the previous time/previous frame) and the latest two-dimensional image, and then comparing the sum or average with a predetermined threshold value.
  • the image state comparison unit 1341 shown in FIG. 5 includes a storage unit 1341a and a comparison processing unit 1341b.
  • the first processing control unit 1380B instructs the first neural network unit 1320 not to execute processing in the second layer and subsequent layers of the object to be determined, and also instructs the first processing control unit 1380B to output the feature map stored in the first output storage unit 1390.
  • the first process control unit 1380B shown in FIG. 15 includes a feature extraction process control unit 1381B.
  • the feature extraction process control unit 1381B executes the function of the first process control unit 1380B in the feature extraction unit 1300.
  • the first output storage section 1390 stores the output data (decision value (probability value)) output by the first neural network section 1320 .
  • the first output storage unit 1390 stores the feature map output by the first neural network unit 1320 .
  • the combined image storage unit 1391 stores a feature map, which is a combined image combined and output by the first neural network unit 1320 .
  • FIG. 6 is a diagram showing an example of the internal configuration of the abnormality warning device 100B and the abnormality state determination unit 1500B in the abnormality determination device 1000B.
  • the abnormal condition determination unit 1500B uses a feature map, which is a two-dimensional image, to output a determination value indicating the condition of the object to be determined.
  • the abnormal state determination unit 1500B shown in FIG. 6 is configured to include a neural network unit (second neural network unit 1520), a state comparison unit (second state comparison unit 1540), and an output storage unit (second output storage unit) 1590.
  • the neural network unit (second neural network unit 1520) has multiple layers in a neural network, acquires the feature map output by the feature extraction unit 1300, and uses the feature map to output a judgment value indicating the state of the object to be judged as output data.
  • the neural network unit (second neural network unit 1520 ) shown in FIG. 6 includes a state classification unit 1525 and a probability output layer 1529 .
  • the state classification unit 1525 has a function of, for example, converting a two-dimensional image (feature map) into a one-dimensional vector, and further consolidating the output into an indication of the drowsy state (for example, four outputs: eyelids: dozing/not dozing, eyeballs: dozing/not dozing).
  • the state classification unit 1525 shown in FIG. 6 includes a first fully connected layer 1527 and a second fully connected layer 1528.
  • the state classification unit 1525 generates a one-dimensional vector whose number of elements is the desired number of outputs through a first fully connected layer 1527 and a second fully connected layer 1528 .
  • the probability output layer 1529 applies a softmax function, for example, and sets the sum of the output values to 1.0, thereby giving the output results a probabilistic meaning.
  • the second neural network unit 1520 shown in FIG. 6 has two fully connected layers, but it may be configured to have three or more fully connected layers if there are no fully connected layers.
  • the second state comparison unit 1540B prestores a reference value that is a standard for the input value to the third layer, which is at least one of the layers in the second neural network unit 1520, and uses the difference between the reference value and the current value, which is the value that is about to be newly input to the third layer, to determine a change in the state of the object to be determined.
  • the reference value is a previous value, which is an input value for each layer, stored as a result of the previous processing by the multiple layers of the second neural network unit 1520, and is used to obtain a difference value between the previous value and a current value, which is an input value that is about to be newly input to the first layer.
  • the second state comparison unit 1540B does not perform state comparison processing for the first processing (processing for the first image) and simply stores the values, but determines changes in the state of the object to be judged for the second and subsequent processing (processing for the second image).
  • the following values may be stored and used as the reference values.
  • the reference value can use typical processing results that have been learned and modeled in advance.
  • the reference value may be a typical output value of a pre-modeled intermediate layer in a neural network.
  • the reference value may use a part of the results determined based on advance information.
  • the partial results determined based on the prior information are assumed to be, for example, pixels that have a high probability of indicating the presence of a head. Also, for example, pixels that are known to have a large change but low importance, such as the background, may not be used.
  • the reference value may be the output value of a predetermined node among the nodes included in the neural network. This allows for reduced memory usage and data handling, resulting in faster processing speeds.
  • the second state comparison section 1540B uses the difference value and a pre-stored threshold value to determine the magnitude of the difference between the reference value and the current value, and determines a change in the state of the object to be determined based on the result. Specifically, when the sum of squares of the difference values is smaller than a pre-stored threshold value for each image, second state comparison section 1540B determines that the state of the determination target is unchanged. More specifically, when the sum of the absolute values of the difference values recorded in image units is smaller than a pre-stored threshold value, the second state comparing section 1540B determines that the state of the determination target is unchanged.
  • the second state comparison unit 1540B shown in FIG. 6 includes a first fully connected layer state comparison unit 1541.
  • "storing" means holding the values of the one-dimensional vectors of the respective input sources.
  • comparison means for example, taking the absolute value of the difference for each element between the stored one-dimensional vector (of the previous time/previous frame) and the latest one-dimensional vector, and then comparing the sum or average with a predetermined threshold value.
  • the first fully connected layer state comparison unit 1541 shown in FIG. 6 includes a storage unit 1541a and a comparison processing unit 1541b.
  • the storage unit 1541 a stores a reference value, which is an output value of the first fully connected layer 1527 and an input value of the second fully connected layer 1528 , for each output by the first fully connected layer 1527 .
  • the comparison processing unit 1541b compares the current value with a reference value.
  • the second processing control unit 1580B instructs the second neural network unit 1520 not to execute processing in the layer of the object to be judged and subsequent layers, and also instructs the second processing control unit 1580B to output the judgment value stored in the second output storage unit as output data.
  • the second process control unit 1580B shown in FIG. 6 includes an abnormal state determination process control unit 1581B.
  • the abnormal state determination process control unit 1581B functions to limit the processing of the second neural network unit 1520 in the abnormal state determination unit 1500B.
  • the second output storage unit 1590 stores the decision value output by the second neural network unit 1520 .
  • the probability storage unit stores the judgment value output by the second neural network unit 1520 .
  • the judgment value is a value indicating the state of the person to be judged, and is, for example, a probability value output by the probability output layer 1529 so that the output result has a probabilistic meaning.
  • the warning output unit 2000 acquires output data that is a judgment value indicating the drowsy state or dozing state of the person to be judged, and outputs a warning to the person to be judged in accordance with the judgment value.
  • the abnormality determination device 1000B shown in FIG. 4 is shown as being configured not to include the alarm output unit 2000, but may be configured to include the alarm output unit 2000. When configured in this manner, the abnormality determination device 1000B is equivalent to the abnormality warning device 100B shown in FIG. 4. In the following explanation, the abnormality determination device 1000B will be described as being configured to include the alarm output unit 2000, except in cases where it is necessary to distinguish between the abnormality warning device 100B and the abnormality determination device 1000B.
  • abnormality determination device 1000B may be configured to include a control unit (not shown), a storage unit (not shown), and a communication unit (not shown).
  • a control unit (not shown) controls the entire abnormality determination device 1000B and each of its components.
  • the control unit (not shown) starts up the abnormality determination device 1000A in accordance with, for example, an external command.
  • a storage unit (not shown) stores each piece of data used in the abnormality determination device 1000B.
  • the storage unit stores, for example, outputs (output data) from each component in the abnormality determination device 1000B, and outputs data requested by each component to the component that has made the request.
  • the communication unit (not shown) communicates with an external device. For example, the communication unit communicates between the abnormality determination device 1000B and an imaging device such as an in-vehicle camera. In addition, for example, if the abnormality determination device 1000B does not have a display unit or an audio output unit, the communication unit communicates between the abnormality determination device 1000B and an external device such as a display unit or an audio output device.
  • FIG. 7 is a flowchart showing an example of processing by the abnormality warning device 100B and the abnormality determination device 1000B according to the second embodiment of the present disclosure.
  • the abnormality determination device 1000B starts the process shown in FIG.
  • Abnormality determination device 1000B executes image acquisition processing (step ST2100).
  • the image acquisition unit 1100 of the abnormality determination device 1000B acquires and outputs an image.
  • Abnormality determination device 1000B executes a storage and state comparison process (step ST2200).
  • the first state comparison unit 1340B of the abnormality determination device 1000B stores the previous value (reference value), which is the input value to the layer, for each layer of the first neural network unit 1320 for at least the first time after the process starts.
  • Abnormality determination device 1000B executes a feature extraction process control process (step ST2300).
  • the feature extraction process control unit 1381 of the abnormality determination device 1000B executes a normal process command or an omission process command to the first neural network unit 1320 depending on the determination result by the first state comparison unit 1340. Furthermore, when issuing an omission processing command, the feature extraction processing control unit 1381 issues an output command to the combined image storage unit 1391 .
  • Abnormality determination device 1000B executes combined image output processing (step ST2400).
  • the first neural network unit 1320 executes normal processing and outputs a combined image, or the combined image storage unit 1391 outputs the previous combined image, whereby the feature extraction unit 1300 outputs a combined image.
  • Abnormality determination device 1000B executes a process of storing the fully connected layer states and comparing the fully connected layer states (step ST2500).
  • the second state comparison unit 1540 in the abnormality determination device 1000B pre-stores, for each of the multiple layers of the second neural network unit 1520, a reference value that is a standard for the input value to that layer, and determines a change in the state of the object to be determined using the difference value between the reference value and the current value, which is the value that is to be newly input to that layer.
  • the second state comparison unit 1540 does not perform state comparison processing for the first processing (processing for the first image) and simply stores the values, but determines changes in the state of the object to be judged for the second and subsequent processing (processing for the second image).
  • Abnormality determination device 1000B executes an abnormal state determination process control process (step ST2600).
  • the abnormal state determination process control unit 1581B issues a normal command or an omission command to the second neural network unit 1520.
  • the abnormal state determination process control unit 1581B issues an output command to the probability storage unit 1591.
  • Abnormality determination device 1000B executes a result output process (step ST2700).
  • the second neural network unit 1520 executes normal processing and outputs a judgment value, or the probability storage unit 1591 outputs a previous judgment value (probability value), so that the abnormal state judgment unit 1500 outputs the judgment value as output data.
  • Abnormality determination device 1000B executes an alarm output process (step ST2800).
  • the alarm output unit 2000 of the abnormality determination device 1000B acquires output data and outputs an alarm signal based on the output data to an alarm device (not shown) etc.
  • the alarm output unit 2000 determines whether to output an alarm based on a determination value included in the output data, and when it determines to output an alarm, outputs an alarm signal to an alarm device (not shown) etc.
  • abnormality determining device 1000B executes the process of step ST3800, it ends the series of processes shown in FIG. 7 and repeats the process from step ST3100.
  • the abnormality determination device 1000B is also turned off in conjunction with, for example, the camera being turned off.
  • FIG. 8 is a flowchart showing an example of more detailed processing of the abnormality warning device 100B and the abnormality determination device 1000B according to the second embodiment of the present disclosure.
  • the flowchart in FIG. 8 shows an example of processing equivalent to the processing from step ST2200 to step ST2700 in the flowchart in FIG.
  • the feature extraction unit 1300B in the abnormality determination device 1000B starts the process of step ST2200
  • the feature extraction unit 1300B first executes a storage process (step ST2201).
  • the state comparison unit 1340 of the feature extraction unit 1300B stores a previous value that is an input value of the first layer, which is at least one layer among all layers of the neural network unit 1320 (first neural network unit 1320). Furthermore, the state comparison unit 1340 stores the input values of the first layer each time processing is performed in the first layer.
  • the feature extraction unit 1300B in the abnormality determination device 1000B executes a previous storage determination process (step ST2202).
  • the feature extraction unit 1300B in the abnormality determination device 1000B executes a comparison process (step ST2203).
  • the state comparison unit 1340 of the feature extraction unit 1300B calculates the difference (differential value) between the current value, which is the value that is about to be newly input to the first layer, and the previous value (reference value).
  • Feature extraction unit 1300B in abnormality determination device 1000B executes a difference determination process (step ST2204).
  • state comparison unit 1340 of feature extraction unit 1300B uses the difference value to determine a change in the state of the determination target. Specifically, for example, in units of images, if the sum of squares of the difference values is smaller than a pre-stored threshold value, the state comparison section 1340 determines that the state of the determination target is unchanged. More specifically, the state comparison unit 1340 determines that the state of the determination target is unchanged when the sum of the absolute values of the difference values is smaller than a pre-stored threshold value, for example, for each image.
  • step ST2202 determines that the data was not previously stored (step ST2202 "NO"), or if the state comparison unit 1340 determines that the difference is equal to or greater than the threshold value and that the state of the object to be determined has changed (step ST2204 "NO")
  • the feature extraction unit 1300B in the abnormality determination device 1000B executes a normal processing command (step ST2301).
  • the processing control unit 1380 of the feature extraction unit 1300B outputs a command to execute processing in the first layer of the neural network unit 1320 (first neural network unit 1320).
  • the feature extraction unit 1300B in the abnormality determination device 1000B executes output processing (step ST2401).
  • the first neural network unit 1320 of the feature extraction unit 1300B executes processing according to the normal processing command, and outputs output data indicating the determination result.
  • the feature extraction unit 1300B in the abnormality determination device 1000B executes an output storage process (step ST2402).
  • the output storage unit 1390 (first output storage unit 1390) of the feature extraction unit 1300B stores the output data output by the neural network unit 1320.
  • the feature extraction unit 1300B in the abnormality determination device 1000B executes an omission processing command (step ST2302).
  • the processing control unit 1380 of the feature extraction unit 1300B instructs the neural network unit 1320 not to execute processing in the first layer or subsequent layers, which is at least one layer out of the multiple layers, and instructs the output storage unit 1390 to output the output data stored therein.
  • the feature extraction unit 1300B in the abnormality determination device 1000B executes a stored data output process (step ST2403).
  • a stored data output process when the output storage unit 1390 (combined image storage unit 1391 of the output storage unit 1390) of the feature extraction unit 1300B receives a command from the process control unit 1380, it outputs the stored output data, that is, the combined image, to the alarm output unit 2000.
  • the abnormal state determination unit 1500B in the abnormality determination device 1000B executes a fully connected layer output storage process (step ST2501).
  • the state comparison unit 1540 of the abnormal state determination unit 1500B stores the value output from the fully connected layer (the first fully connected layer 1527 or the second fully connected layer 1528) in the state classification unit 1525 as the state of the fully connected layer.
  • the abnormal state determination unit 1500B in the abnormality determination device 1000B executes a previous storage determination process (step ST2502).
  • the abnormal state determination unit 1500B in the abnormality determination device 1000B determines that a previous value has been stored ("YES" in step ST2502), it executes a comparison process (step ST2503).
  • the state comparison unit 1540 of the abnormal state determination unit 1500B calculates the difference (differential value) between the current value, which is the value that is about to be newly input to the first layer (the second fully connected layer 1528 or the probability output layer 1529), and the previous value (reference value).
  • the abnormal state determination unit 1500B in the abnormality determination device 1000B executes a difference determination process (step ST2504).
  • the state comparison unit 1540 of the abnormal state determination unit 1500B determines the magnitude of the difference between the reference value and the current value using the difference between the current value and the previous value (reference value) and a pre-stored threshold value.
  • step ST2504 determines that the difference is equal to or greater than the threshold value (step ST2504 "NO")
  • the abnormal state determination unit 1500B in the abnormality determination device 1000B executes a normal processing command (step ST2601).
  • the processing control unit 1580 of the abnormal state determination unit 1500B executes a normal processing command to the neural network unit 1520 (second neural network unit 1520).
  • the abnormal state determination unit 1500B in the abnormality determination device 1000B executes an output process (step ST2701).
  • the neural network unit 1520 (second neural network unit 1520) of the abnormal state determination unit 1500B outputs the determination value as output data.
  • the abnormal state determination unit 1500B in the abnormality determination device 1000B executes an output storage process (step ST2702).
  • the output storage unit 1590 (second output storage unit 1590) in the abnormal state determination unit 1500B stores the determination value output by the neural network unit 1520 (second neural network unit 1520).
  • the abnormal state determination unit 1500B in the abnormality determination device 1000B executes an omission processing command (step ST2602).
  • the processing control unit 1580 of the abnormal state determination unit 1500B instructs the second neural network unit 1520 not to execute processing in layers subsequent to the layer to be determined, and instructs the second output storage unit to output the determination value stored therein.
  • the abnormal state determination unit 1500B in the abnormality determination device 1000B executes a stored data output process (step ST2703).
  • the second output storage unit 1590 of the abnormal state determination unit 1500B stores the determination value output by the neural network unit 1520 (second neural network unit 1520).
  • the configuration may be one or a combination of the following: the image state comparison unit 1341, a convolution layer state comparison unit that determines a state change in the convolution layer (see the image state comparison unit 1342 in the embodiment described later), a pooling layer state comparison unit that determines a state change in the pooling layer (see the pooling layer state comparison unit 1343 in the embodiment described later), a first fully connected layer state comparison unit (see the first fully connected layer state comparison unit 1541 in the embodiment described later), and a second fully connected layer state comparison unit that determines a state change in the second fully connected layer 1528 (see the second fully connected layer state comparison unit 1542 in the embodiment described later).
  • the image state comparison unit 1341 a convolution layer state comparison unit that determines a state change in the convolution layer
  • a pooling layer state comparison unit that determines a state change in the pooling layer
  • a first fully connected layer state comparison unit see the first fully connected layer state comparison unit 1541 in the embodiment described later
  • a second fully connected layer state comparison unit that determine
  • FIG. 9 is a flowchart showing an example of a process of the feature extraction unit 1300B in the case where the state comparison unit (first state comparison unit 1340B) in the feature extraction unit 1300B according to the second embodiment of the present disclosure is the image state comparison unit 1341.
  • the first state comparison section 1340B of the feature extraction section 1300B acquires an image acquisition command and executes an image storage process (step ST2211).
  • the image state comparison section 1341 of the first state comparison section 1340B stores the image that is the input data for the first neural network section 1320. Furthermore, every time the first neural network unit 1320 acquires an image, the image state comparison unit 1341 stores the image (the image before being processed by the first neural network unit 1320) in the storage unit 1341a.
  • the image state comparison section 1341 executes a process of determining whether the image has been previously stored (step ST2212).
  • the image state comparison section 1341 refers to the storage section 1341a to determine whether or not the previously input image is stored.
  • the image state comparison unit 1341 executes a process of comparing the current image with the previous image (step ST2213).
  • the comparison processing section 1341b of the image state comparison section 1341 compares the currently input image with the previously input image.
  • the comparison processing unit 1341b executes a process of determining whether the difference is less than a threshold value (step ST2214).
  • the comparison processing unit 1341b uses the difference value between the image input this time and the image input last time and a pre-stored threshold value to determine the magnitude of the difference between a reference value (the image input last time) and a current value (the image to be input this time).
  • the feature extraction processing control unit 1381 which is the first processing control unit 1380, issues a normal processing command to the first neural network unit 1320 (step ST2311).
  • First neural network unit 1320 executes normal processing in accordance with the normal processing command, and outputs a combined image (step ST2411).
  • the combined image storage section 1391 which is the first output storage section 1390, stores the combined image output by the first neural network section 1320 (step ST2412).
  • the feature extraction processing control unit 1381 which is the first processing control unit 1380, issues an omission processing command to the first neural network unit 1320 (step ST2312) and also issues an output command to the combined image storage unit 1391, which is the first output storage unit 1390 (step ST2413).
  • the first neural network unit 1320 does not execute processing of the subsequent layers in the first neural network unit 1320 .
  • the combined image storage unit 1391 outputs the combined image in accordance with the output command.
  • FIG. 10 is a flowchart illustrating an example of a process of the feature extraction unit 1300B in the second embodiment of the present disclosure when the state comparison unit (first state comparison unit 1340B) in the feature extraction unit 1300B is a convolutional layer state comparison unit.
  • the first state comparison unit 1340B of the feature extraction unit 1300B executes a convolution layer state storage process (step ST2221).
  • the convolution layer state comparison unit of the first state comparison unit 1340B stores an input value (a value before being processed by the pooling layer unit 1325) which is an output value of the convolution layer unit 1322 and is input data to the pooling layer unit 1325.
  • the convolution layer state comparison unit stores the output value (the value before being processed by the pooling layer unit 1325) in the storage unit.
  • the convolution layer state comparison unit executes a process of determining whether the data has been stored previously (step ST2222).
  • the convolution layer state comparison unit refers to the storage unit and determines whether a previously input value (reference value: in this explanation, the previous value) is stored.
  • the convolution layer state comparison unit determines that the previous state has been stored ("YES" in step ST2222), it executes a comparison process between the current state and the previous state (step ST2223).
  • the comparison processing unit executes a process of determining whether the difference is less than a threshold value (step ST2224).
  • the comparison processing unit determines the magnitude of the difference between a reference value (the value input previously) and the current value (the value being input currently) using a difference value between the value being input currently (the current value) and the value input previously (the previous value) and a pre-stored threshold value.
  • the feature extraction processing control unit 1381 which is the first processing control unit 1380, issues a normal processing command to the first neural network unit 1320 (step ST2321).
  • First neural network unit 1320 executes normal processing in accordance with the normal processing command, and outputs a combined image (step ST2421).
  • the combined image storage section 1391 which is the first output storage section 1390, stores the combined image output by the first neural network section 1320 (step ST2422).
  • the feature extraction processing control unit 1381B which is the first processing control unit 1380, issues an omission processing command to the first neural network unit 1320 (step ST2322) and also issues an output command to the combined image storage unit 1391, which is the first output storage unit 1390 (step ST2423).
  • the first neural network unit 1320 does not execute processing of the subsequent layers in the first neural network unit 1320 .
  • the combined image storage unit 1391 outputs the combined image in accordance with the output command.
  • FIG. 11 is a flowchart illustrating an example of a process performed by the feature extraction unit 1300B in the second embodiment of the present disclosure when the state comparison unit (first state comparison unit 1340B) in the feature extraction unit 1300B is a pooling layer state comparison unit.
  • the first state comparison unit 1340B of the feature extraction unit 1300B executes a pooling layer state storage process (step ST2221).
  • the pooling layer state comparison unit of the first state comparison unit 1340B stores the input value (the value before being processed by the image combination unit 1328), which is the output value of the pooling layer unit 1325 and is input data to the image combination unit 1328.
  • the pooling layer state comparison unit stores the output value (the value before being processed by the image combination unit 1328) in the storage unit.
  • the pooling layer state comparison unit executes a process of determining whether or not the data has been stored previously (step ST2222).
  • the pooling layer state comparison unit refers to the storage unit and determines whether a previously input value (reference value: in this explanation, the previous value) is stored.
  • the pooling layer state comparison unit executes a process of comparing the current data with the previous data (step ST2223).
  • the comparison processing unit of the pooling layer state comparison unit compares the value currently being input to the image combination unit 1328 (current value) with the value previously input (previous value).
  • the comparison processing unit executes a process of determining whether the difference is less than a threshold value (step ST2224).
  • the comparison processing unit determines the magnitude of the difference between a reference value (the value input previously) and the current value (the value being input currently) using a difference value between the value being input currently (the current value) and the value input previously (the previous value) and a pre-stored threshold value.
  • the feature extraction processing control unit 1381B which is the first processing control unit 1380, issues a normal processing command to the first neural network unit 1320 (step ST2321).
  • First neural network unit 1320 executes normal processing in accordance with the normal processing command, and outputs a combined image (step ST2421).
  • the combined image storage section 1391 which is the first output storage section 1390, stores the combined image output by the first neural network section 1320 (step ST2422).
  • the feature extraction processing control unit 1381B which is the first processing control unit 1380, issues an omission processing command to the first neural network unit 1320 (step ST2322) and also issues an output command to the combined image storage unit 1391, which is the first output storage unit 1390 (step ST2423).
  • the first neural network unit 1320 does not execute processing of the subsequent layers in the first neural network unit 1320 .
  • the combined image storage unit 1391 outputs the combined image in accordance with the output command.
  • FIG. 12 is a flowchart showing an example of processing of the abnormal state determination unit 1500B in the case where the state comparison unit (second state comparison unit 1540B) in the abnormal state determination unit 1500B according to the second embodiment of the present disclosure is the first layer state comparison unit 1541.
  • the second state comparison unit 1540B When the abnormal state determination unit 1500B executes processing on the data output by the feature extraction unit 1300B (step ST2511), the second state comparison unit 1540B first acquires the first fully connected layer output (step ST2512). It then executes a first fully connected layer state storage process (step ST2513). The first layer state comparison unit 1541 in the second state comparison unit 1540B stores the output value from the first fully connected layer 1527, which is the input value of the second fully connected layer 1528 (the value before being processed by the second fully connected layer 1528).
  • the first layer state comparing unit 1541 executes a process of determining whether or not the data has been previously stored (step ST2514).
  • the first layer state comparison unit 1541 refers to the storage unit 1541a and determines whether a reference value (previous value) is stored.
  • the first layer state comparison unit 1541 determines that the previous value has been stored ("YES" in step ST2514), it executes a comparison process between the current value and the previous value (step ST2515).
  • the first fully connected layer state comparison unit 1541 compares the current value with a reference value (previous value).
  • the first layer state comparing unit 1541 executes a process of determining whether the difference is smaller than a threshold value (step ST2516).
  • the comparison processing unit 1541b judges whether the difference between the current value and the reference value (previous value) is large or small, using a difference between the current value and the reference value and a pre-stored threshold value.
  • the abnormal state determination processing control unit 1581B which is the second processing control unit 1580, issues a normal processing command to the second neural network unit 1520 (step ST2611).
  • Second neural network unit 1520 executes normal processing in accordance with the normal processing command, and outputs a decision value (step ST2711).
  • Probability storage section 1591 which is second output storage section 1590, stores the decision value output by second neural network section 1520 (step ST2712).
  • the abnormal state determination process controller 1581B which is the second process controller 1580, executes an omission process command (step ST2612).
  • the abnormal state judgment processing control unit 1581B instructs the second neural network unit 1520 not to execute processing in layers subsequent to the layer to be judged, and also instructs it to output the judgment value stored in the second output storage unit 1590.
  • the second neural network unit 1520 does not execute processing of the subsequent layers in the second neural network unit 1520 .
  • the probability storage section 1591 which is the second output storage section 1590, outputs the decision value in accordance with the output command (step ST2713).
  • FIG. 13 is a flowchart showing an example of processing by abnormal state determination unit 1500B in the case where the state comparison unit (second state comparison unit 1540B) in abnormal state determination unit 1500B according to embodiment 2 of the present disclosure is the second layer state comparison unit.
  • the abnormal state determination unit 1500B executes processing on the data output by the feature extraction unit 1300B (step ST2521).
  • the second fully connected layer 1528 then executes processing on the output by the first fully connected layer (step ST2522).
  • the second state comparison unit 1540B acquires the second fully connected layer output (step ST2523).
  • the second state comparison unit 1540B executes a second fully connected layer state storage process (step ST2524).
  • the second layer state comparison unit in the second state comparison unit 1540B stores the output value from the second fully connected layer 1528 and the input value of the probability output layer 1529 (the value before being processed by the probability output layer 1529).
  • the second layer state comparing unit executes a process of determining whether the data has been stored previously (step ST2525).
  • the second layer state comparison unit refers to the storage unit and determines whether a reference value (previous value) is stored.
  • the second layer state comparison unit determines that the value was stored last time ("YES" in step ST2525), it executes a comparison process between the current value and the previous value (step ST2526).
  • the second layer state comparison unit compares the current value with the reference value (previous value).
  • the second layer state comparing unit executes a process of determining whether the difference is smaller than a threshold value (step ST2527).
  • the comparison processing section determines whether the difference between the current value and the reference value (previous value) is large or small, using a difference between the current value and the reference value (previous value) and a pre-stored threshold value.
  • the abnormal state determination processing control unit 1581B which is the second processing control unit 1580, issues a normal processing command to the second neural network unit 1520 (step ST2621).
  • Second neural network unit 1520 executes normal processing in accordance with the normal processing command, and outputs a determination value (step ST2721).
  • Probability storage section 1591 which is second output storage section 1590, stores the decision value output by second neural network section 1520 (step ST2722).
  • the abnormal state judgment process control unit 1581B which is the second process control unit 1580, executes an omission process command (step ST2622).
  • the abnormal state judgment processing control unit 1581B instructs the second neural network unit 1520 not to execute processing in layers subsequent to the layer to be judged, and also instructs it to output the judgment value stored in the second output storage unit 1590.
  • the second neural network unit 1520 does not execute processing of the subsequent layers in the second neural network unit 1520 .
  • the probability storage section 1591 which is the second output storage section 1590, outputs a decision value in accordance with the output command (step ST2723).
  • the abnormality determination device of the present disclosure is further configured as follows. "The neural network unit includes a first neural network unit and a second neural network unit,
  • the output storage unit includes a first output storage unit and a second output storage unit,
  • the state comparison unit includes a first state comparison unit and a second state comparison unit
  • the processing control unit includes a first processing control unit and a second processing control unit, the first neural network unit having a plurality of layers that are a part of the layers constituting the neural network, acquiring an image output by the image acquisition unit, and outputting a feature map that represents a characteristic state of the object to be determined that is included in the image;
  • the first output storage unit that stores the feature map output by the first neural network unit;
  • the first state comparison unit pre-stores a reference value that is a reference for an input value to a second layer that is at least one of the layers in the first neural network unit, and judges a change in the state of the object to be judged by using a difference value between the reference value and a current value
  • the present disclosure has the effect of providing an abnormality determination device that can use a neural network in an abnormality determination technique to more quickly output highly accurate determination results. Furthermore, the present disclosure achieves the same effect as the above by applying the above configuration to the abnormality determination method.
  • the abnormality determination device of the present disclosure is further configured as follows. "The first state comparison unit and the second state comparison unit are a change in the state of the object to be determined based on a result of determining whether a difference between the reference value and the current value is greater or smaller using the difference value and a pre-stored threshold value; An abnormality determination device characterized by the above. As a result, the present disclosure has the effect of providing an abnormality determination device that can use a neural network in an abnormality determination technique to more quickly output highly accurate determination results. Furthermore, the present disclosure achieves the same effect as the above by applying the above configuration to the abnormality determination method.
  • the abnormality determination device of the present disclosure is further configured as follows. "The first state comparison unit and the second state comparison unit are When the sum of squares of the difference values is smaller than a pre-stored threshold value for each image, it is determined that the state of the object to be determined is unchanged.
  • the present disclosure has the effect of providing an abnormality determination device that can use a neural network in an abnormality determination technique to more quickly output highly accurate determination results. Furthermore, the present disclosure achieves the same effect as the above by applying the above configuration to the abnormality determination method.
  • the abnormality determination device of the present disclosure is further configured as follows. "The first state comparison unit and the second state comparison unit are When the sum of the absolute values of the difference values is smaller than a pre-stored threshold value for each image, it is determined that the state of the object to be determined is unchanged.
  • the present disclosure has the effect of providing an abnormality determination device that can use a neural network in an abnormality determination technique to more quickly output highly accurate determination results. Furthermore, the present disclosure achieves the same effect as the above by applying the above configuration to the abnormality determination method.
  • Embodiment 3 a form in which the basic mechanism of the present disclosure is applied to all layers in a neural network will be described. In the third embodiment, the configuration and processing already described will be omitted as appropriate.
  • FIG. 14 is a diagram showing an example of the configuration of an abnormality warning device 100C and an abnormality determination device 1000C according to the third embodiment of the present disclosure.
  • the abnormality warning device 100C includes an abnormality determination device 1000C and a warning output unit 2000.
  • the abnormality determination device 1000C obtains an image, and uses the image to output the state of the object to be determined that is captured in the image.
  • An abnormality determination device 1000C shown in FIG. 14 includes an image acquisition unit 1100C, a feature extraction unit 1300C, and an abnormal state determination unit 1500C.
  • the neural network unit already described is configured to include, as will be described later, a first neural network unit 1320 and a second neural network unit 1520.
  • First neural network unit 1320 is included in feature extraction unit 1300C
  • second neural network unit 1520 is included in abnormal state determination unit 1500C.
  • the output storage unit already described is configured to include a first output storage unit 1390 and a second output storage unit.
  • the first output storage unit 1390 is included in the feature extraction unit 1300C, and the second output storage unit is included in the abnormal state determination unit 1500C.
  • the state comparison unit already described is configured to include a first state comparison unit 1340 and a second state comparison unit 1540.
  • First state comparison unit 1340 is included in feature extraction unit 1300C
  • second state comparison unit 1540 is included in abnormal state determination unit 1500C.
  • the process control unit already described is configured to include a first process control unit 1380 and a second process control unit 1580.
  • the first process control unit 1380 is included in the feature extraction unit 1300C
  • the second process control unit 1580 is included in the abnormal state determination unit 1500C.
  • the image acquisition unit 1100C is similar to the image acquisition units 1100A and 1100B already described, a detailed description of the image acquisition unit 1100C will be omitted here.
  • the feature extraction unit 1300C uses the image to output a feature map that represents the characteristic state of the subject contained in the image. For example, the feature extraction unit 1300C extracts parts of the image that show signs of drowsiness, such as the eyelids and eyeballs, and generates a feature map that represents the state of the subject that is characteristic of an abnormal state such as drowsiness.
  • FIG. 15 is a diagram showing an example of the internal configuration of the feature extraction unit 1300C in the abnormality warning device 100C and the abnormality determination device 1000C.
  • the feature extraction unit 1300C shown in FIG. 15 is configured to include a neural network unit (first neural network unit 1320), a state comparison unit (first state comparison unit 1340C), a processing control unit (first processing control unit 1380C), and an output storage unit (first output storage unit 1390).
  • the first neural network unit 1320 has multiple layers that are part of the layers that make up the neural network, acquires the image output by the image acquisition unit 1100, and outputs a feature map that represents the characteristic state of the object to be judged contained in the image.
  • the first neural network unit 1320 shown in FIG. 15 includes an image branching unit 1321, a convolution layer unit 1322, a pooling layer unit 1325, and an image combination unit 1328.
  • the image branching unit 1321 branches and inputs an image to multiple nodes in a layer of a neural network.
  • the image is branched according to the number of subsequent convolutional layers and pooling layers. In FIG. 15, the image is branched into two.
  • the convolution layer unit 1322 performs filtering to extract characteristic parts of the object to be determined in the image.
  • the convolution layer unit 1322 shown in FIG. 15 includes a first convolution layer 1323 and a second convolution layer 1324.
  • the first convolutional layer 1323 and the second convolutional layer 1324 perform filtering to extract face (body) parts for detecting a drowsy state, for example, by convolution processing (cross-correlation processing) using pre-prepared convolution filters of size 3x3 or 5x5.
  • the pooling layer unit 1325 shown in FIG. 15 includes a first pooling layer 1326 and a second pooling layer 1327 .
  • the first pooling layer 1326 and the second pooling layer 1327 generate an image relating to features that are robust against image position, for example, by calculating the maximum or average value for each predetermined region.
  • the first neural network unit 1320 shown in FIG. 15 includes two pairs of convolutional layers and pooling layers, but it is also effective to include one pair or three or more pairs.
  • the first pooling layer and the second pooling layer 1327 may be followed by two or more convolution layers and two or more pooling layers.
  • a normalized linear unit layer (activation function) or the like may be included after the convolution layer.
  • the image combination unit 1328 combines multiple images output through the convolution layer and the pooling layer.
  • the image combining unit 1328 extracts areas that indicate signs of drowsiness, such as the eyelids and eyeballs, and generates a feature map.
  • the first state comparison unit 1340C prestores a reference value, which is the standard for the input value to each layer in the first neural network unit 1320, and uses the difference between the reference value and the current value, which is the value that is about to be newly input to the layer, to determine a change in the state of the object to be determined.
  • the reference value is a previous value, which is an input value for each layer, stored as a result of the previous processing by the multiple layers of the first neural network unit 1320, and is used to obtain a difference value between the previous value and a current value, which is an input value that is about to be newly input to the layer.
  • the first state comparison unit 1340 does not perform state comparison processing for the first processing (processing for the first image) and simply stores the value, but in the second and subsequent processing (processing for the second image), it determines changes in the state of the object to be judged.
  • the following values may be stored and used as the reference values.
  • the reference value can use typical processing results that have been learned and modeled in advance.
  • the reference value may be a typical output value of a pre-modeled intermediate layer in a neural network.
  • the reference value may use a part of the results determined based on advance information.
  • the partial results determined based on the prior information are assumed to be, for example, pixels that have a high probability of indicating the presence of a head. Also, for example, pixels that are known to have a large change but low importance, such as the background, may not be used.
  • the reference value may be the output value of a predetermined node among the nodes included in the neural network. This allows for reduced memory usage and data handling, resulting in faster processing speeds.
  • the first state comparison section 1340 uses the difference value and a pre-stored threshold value to determine the magnitude of the difference between the reference value and the current value, and determines a change in the state of the object to be determined based on the result. Specifically, the first state comparing section 1340 determines that the state of the determination target is unchanged when the sum of squares of the difference values is smaller than a pre-stored threshold value for each image. More specifically, the first state comparing section 1340 determines that the state of the determination target is unchanged when the sum of the absolute values of the difference values recorded in image units is smaller than a pre-stored threshold value.
  • the first state comparison unit 1340C shown in FIG. 15 includes an image state comparison unit 1341, a convolution layer state comparison unit 1342, and a pooling layer state comparison unit 1343.
  • "storing" means holding the values of the two-dimensional images (feature maps) that are the respective input sources.
  • comparison means for example, taking the absolute value of the difference for each element (pixel) between a stored two-dimensional image (of the previous time/frame) and the latest two-dimensional image, and further comparing the sum or average with a predetermined threshold value.
  • the image state comparison unit 1341 shown in FIG. 15 includes a storage unit 1341a and a comparison processing unit 1341b.
  • the convolutional layer state comparison unit 1342 shown in FIG. 15 includes a storage unit 1342a and a comparison processing unit 1342b.
  • the pooling layer state comparison unit 1343 shown in FIG. 15 includes a storage unit 1343a and a comparison processing unit 1343b.
  • the first processing control unit 1380C instructs the first neural network unit 1320 not to execute processing in layers subsequent to the layer of the object to be judged, and also instructs the first output storage unit 1390 to output the feature map stored in the first output storage unit 1390.
  • the first process control unit 1380C shown in FIG. 15 includes a feature extraction process control unit 1381C.
  • the feature extraction process control unit 1381C executes the function of the first process control unit 1380C in the feature extraction unit 1300.
  • the first output storage section 1390 stores the output data (decision value (probability value)) output by the first neural network section 1320 .
  • the first output storage unit 1390 stores the feature map output by the first neural network unit 1320 .
  • the combined image storage unit 1391 stores a feature map, which is a combined image combined and output by the first neural network unit 1320 .
  • the abnormal condition determination unit 1500C uses a feature map, which is a two-dimensional image, to output a determination value indicating the condition of the object to be determined.
  • FIG. 16 is a diagram showing an example of the internal configuration of the abnormality warning device 100C and the abnormality state determination unit 1500C in the abnormality determination device 1000C.
  • the abnormal state determination unit 1500C shown in FIG. 16 is configured to include a neural network unit (second neural network unit 1520), a state comparison unit (second state comparison unit 1540), and an output storage unit (second output storage unit) 1590.
  • the neural network unit (second neural network unit 1520) has multiple layers in a neural network, acquires the feature map output by the feature extraction unit 1300, and uses the feature map to output a judgment value indicating the state of the object to be judged as output data.
  • the neural network unit (second neural network unit 1520 ) shown in FIG. 16 includes a state classification unit 1525 and a probability output layer 1529 .
  • the state classification unit 1525 has a function of, for example, converting a two-dimensional image (feature map) into a one-dimensional vector, and further consolidating the output into an indication of the drowsy state (for example, four outputs: eyelids: dozing/not dozing, eyeballs: dozing/not dozing).
  • the state classification unit 1525 shown in FIG. 16 includes a first fully connected layer 1527 and a second fully connected layer 1528.
  • the state classification unit 1525 generates a one-dimensional vector whose number of elements is the desired number of outputs through a first fully connected layer 1527 and a second fully connected layer 1528 .
  • the probability output layer 1529 applies a softmax function, for example, and sets the sum of the output values to 1.0, thereby giving the output results a probabilistic meaning.
  • the second neural network unit 1520 shown in FIG. 16 has two fully connected layers, but it may be configured to have three or more fully connected layers if there are no fully connected layers.
  • the second state comparison unit 1540 prestores a reference value, which is the standard for the input value to each layer of the second neural network unit 1520, and uses the difference between the reference value and the current value, which is the value that is about to be newly input to the layer, to determine a change in the state of the object to be determined.
  • the reference value is a previous value, which is an input value for each layer, stored as a result of the previous processing by the multiple layers of the second neural network unit 1520, and is used to obtain a difference value between the previous value and a current value, which is an input value that is about to be newly input to the first layer.
  • the previous value is used as the reference value.
  • the second state comparison unit 1540 does not perform state comparison processing for the first processing (processing for the first image) and simply stores the value, but determines changes in the state of the object to be judged for the second and subsequent processing (processing for the second image).
  • the following values may be stored and used as the reference values.
  • the reference value can use typical processing results that have been learned and modeled in advance.
  • the reference value may be a typical output value of a pre-modeled intermediate layer in a neural network.
  • the reference value may use a part of the results determined based on advance information.
  • the partial results determined based on the prior information are assumed to be, for example, pixels that have a high probability of indicating the presence of a head. Also, for example, pixels that are known to have a large change but low importance, such as the background, may not be used.
  • the reference value may be the output value of a predetermined node among the nodes included in the neural network. This allows for reduced memory usage and data handling, resulting in faster processing speeds.
  • the second state comparison section 1540 uses the difference value and a pre-stored threshold value to determine the magnitude of the difference between the reference value and the current value, and determines a change in the state of the object to be determined based on the result. Specifically, when the sum of squares of the difference values is smaller than a pre-stored threshold value for each image, second state comparing section 1540 determines that the state of the determination target is unchanged. More specifically, when the sum of the absolute values of the difference values recorded in image units is smaller than a pre-stored threshold value, the second state comparing section 1540 determines that the state of the determination target is unchanged.
  • the 16 includes a first layer state comparison unit 1541 and a second layer state comparison unit 1542.
  • the first layer state comparison unit 1541 is also referred to as a first fully connected layer state comparison unit 1541.
  • the second layer state comparison unit 1542 is also referred to as a second fully connected layer state comparison unit 1542.
  • "storing" means holding the values of the one-dimensional vectors of the respective input sources.
  • comparison means for example, taking the absolute value of the difference between a stored one-dimensional vector (of the previous time/frame) and the latest one-dimensional vector for each element, and then comparing the sum or average with a predetermined threshold value.
  • the first fully connected layer state comparison unit 1541 shown in FIG. 16 includes a storage unit 1541a and a comparison processing unit 1541b.
  • the storage unit 1541a stores a reference value, which is an output value of the first fully connected layer 1527 and an input value of the second fully connected layer, for each output by the first fully connected layer 1527.
  • the comparison processing unit 1541b compares the current value with a reference value.
  • the second fully connected layer state comparison unit 1542 shown in FIG. 16 includes a storage unit 1542a and a comparison processing unit 1542b.
  • the storage unit 1542a stores a reference value, which is an output value of the second fully connected layer and an input value of the probability output layer 1529, for each output by the second fully connected layer.
  • the comparison processing unit 1542b compares the current value with a reference value.
  • the second processing control unit 1580C instructs the second neural network unit 1520 not to execute processing in the layer of the object to be judged and subsequent layers, and also instructs the second processing control unit 1580C to output the judgment value stored in the second output storage unit as output data.
  • the second process control unit 1580C shown in FIG. 16 includes an abnormal state determination process control unit 1581C.
  • the abnormal state determination processing control unit 1581C functions to limit the processing of the second neural network unit 1520 in the abnormal state determination unit 1500C.
  • the second output storage unit 1590 stores the decision value output by the second neural network unit 1520 .
  • the probability storage unit 1591 stores the judgment value output by the second neural network unit 1520 .
  • the judgment value is a value indicating the state of the person to be judged, and is, for example, a probability value output by the probability output layer 1529 so that the output result has a probabilistic meaning.
  • the warning output unit 2000 acquires output data that is a judgment value indicating the drowsy state or dozing state of the person to be judged, and outputs a warning to the person to be judged in accordance with the judgment value.
  • the abnormality determination device 1000C shown in FIG. 14 is shown as being configured not to include the alarm output unit 2000, but may be configured to include the alarm output unit 2000. When configured in this manner, the abnormality determination device 1000C is equivalent to the abnormality warning device 100C shown in FIG. 14. In the following explanation, the abnormality determination device 1000C will be explained as being configured to include the alarm output unit 2000, except in cases where it is necessary to distinguish between the abnormality warning device 100C and the abnormality determination device 1000C.
  • abnormality determination device 1000C may be configured to include a control unit (not shown), a storage unit (not shown), and a communication unit (not shown).
  • a control unit (not shown) controls the entire abnormality determination device 1000C and each of its components.
  • the control unit (not shown) starts up the abnormality determination device 1000C in response to, for example, an external command.
  • a storage unit (not shown) stores each piece of data used in the abnormality determination device 1000C.
  • the storage unit stores, for example, outputs (output data) from each component in the abnormality determination device 1000C, and outputs data requested by each component to the component that has made the request.
  • the communication unit (not shown) communicates with an external device. For example, the communication unit communicates between the abnormality determination device 1000C and an imaging device such as an in-vehicle camera. In addition, for example, if the abnormality determination device 1000C does not have a display unit or an audio output unit, the communication unit communicates between the abnormality determination device 1000C and an external device such as a display unit or an audio output device.
  • FIG. 17 is a flowchart showing an example of the processing of the abnormality warning device 100C and the abnormality determination device 1000C according to the third embodiment of the present disclosure.
  • the abnormality determination device 1000C starts the process shown in FIG. 17 when an image is input from a camera, for example.
  • Abnormality determination device 1000C executes image acquisition processing (step ST3100).
  • the image acquisition unit 1100 of the abnormality determination device 1000C acquires and outputs an image.
  • abnormality determination device 1000C executes a storage and state comparison process (step ST3200).
  • the first state comparison unit 1340C of the abnormality determination device 1000C stores the previous values, which are the input values for each layer, which are the processing results for all layers of the first neural network unit 1320 for at least the first time after the start of processing.
  • the first state comparison unit 1340 does not perform state comparison processing for the first processing (processing for the first image) and simply stores the values, but in the second and subsequent processing (processing for the second image), it determines whether there has been a change in the state of the object to be judged.
  • abnormality determination device 1000C executes feature extraction process control (step ST3300).
  • the feature extraction process control unit 1381 of the abnormality determination device 1000C executes a normal process command or an omission process command to the first neural network unit 1320 depending on the determination result by the first state comparison unit 1340. Furthermore, when issuing an omission processing command, the feature extraction processing control unit 1381 issues an output command to the combined image storage unit 1391 .
  • abnormality determination device 1000C executes combined image output processing (step ST3400).
  • the first neural network unit 1320 executes normal processing and outputs a combined image, or the combined image storage unit 1391 outputs the previous combined image, whereby the feature extraction unit 1300 outputs a combined image.
  • Abnormality determination device 1000C executes a process of storing the fully connected layer states and comparing the fully connected layer states (step ST3500).
  • the second state comparison unit 1540 in the abnormality determination device 1000C pre-stores, for each of the multiple layers of the second neural network unit 1520, a reference value that is a standard for the input value to that layer, and determines a change in the state of the object to be determined using the difference value between the reference value and the current value, which is the value that is to be newly input to that layer.
  • the second state comparison unit 1540 does not perform state comparison processing for the first processing (processing for the first image) and simply stores the values, but determines changes in the state of the object to be judged for the second and subsequent processing (processing for the second image).
  • Abnormality determination device 1000C executes an abnormal state determination process control process (step ST3600).
  • the abnormal state determination process control unit 1581C issues a normal command or an omission command to the second neural network unit 1520.
  • the abnormal state determination process control unit 1581C issues an output command to the probability storage unit 1591.
  • Abnormality determination device 1000C executes a result output process (step ST3700).
  • the second neural network unit 1520 executes normal processing and outputs a judgment value, or the probability storage unit outputs a previous judgment value (probability value), which causes the abnormality judgment unit to output the judgment value as output data.
  • Abnormality determination device 1000C executes an alarm output process (step ST3800).
  • the alarm output unit 2000 of the abnormality determination device 1000C acquires output data and outputs an alarm signal based on the output data to an alarm device (not shown) etc.
  • the alarm output unit 2000 determines whether to output an alarm based on a determination value included in the output data, and when it determines to output an alarm, outputs an alarm signal to an alarm device (not shown) etc.
  • abnormality determining device 1000C executes the process of step ST3800, it ends the series of processes shown in FIG. 17 and repeats the process from step ST3100.
  • the abnormality determination device 1000C is also turned off in conjunction with, for example, the camera being turned off.
  • FIG. 18 is a flowchart showing a detailed first example of the processing of the feature extraction unit 1300C in the abnormality warning device 100C and the abnormality determination device 1000C according to the third embodiment of the present disclosure.
  • the first state comparison section 1340C of the feature extraction section 1300C executes an image storage process (step ST3210).
  • the image state comparison section 1341 of the first state comparison section 1340C stores the image that is the input data for the first neural network section 1320. Furthermore, every time the first neural network unit 1320 acquires an image, the image state comparison unit 1341 stores the image in the storage unit 1341a.
  • the image state comparison section 1341 executes a process of determining whether the image has been previously stored (step ST3211).
  • the image state comparison section 1341 refers to the storage section 1341a to determine whether the previously input image is stored therein.
  • the image state comparison unit 1341 executes a process of comparing the current image with the previous image (step ST3212).
  • the comparison processing section 1341b of the image state comparison section 1341 compares the currently input image with the previously input image.
  • the comparison processing unit 1341b executes a process of determining whether the difference is less than a threshold value (step ST3213).
  • the comparison processing unit 1341b judges the magnitude of the difference between the reference value and the current value using a difference value between the currently input image and the previously input image and a pre-stored threshold value.
  • the convolution layer state comparison unit 1342 executes a convolution layer state storage process (step ST3214).
  • the convolution layer state comparison unit 1342 stores input data for the convolution layer in the storage unit 1342a.
  • the convolution layer state comparison unit 1342 executes a process of determining whether or not the data has been previously stored (step ST3215).
  • the convolution layer state comparison unit 1342 refers to the storage unit 1342a and determines whether a previous value, which is a value input previously, is stored.
  • step ST3215 determines that the previous state has been stored (step ST3215 "YES")
  • step ST3216 executes a comparison process between the current state and the previous state
  • the feature extraction section 1300C executes a process of determining whether the difference is less than a threshold value (step ST3217).
  • the comparison processing unit 1342b judges whether the difference between the previous value and the current value is large or small, using a difference value between the currently input image and the previously input image and a pre-stored threshold value.
  • the pooling layer state comparing unit 1343 executes a pooling layer state storage process (step ST3218). In the pooling layer state storage process, the pooling layer state comparison unit 1343 stores input data for the pooling layer in the storage unit 1343a.
  • the comparison processing unit 1343b in the pooling layer state comparison unit 1343 executes a process to determine whether the data was previously stored (step ST3219).
  • step ST3219 If the comparison processing unit 1343b of the pooling layer state comparison unit 1343 determines that the value was stored previously ("YES" in step ST3219), it executes a comparison process between the current value and the reference value (previous value) (step ST3220).
  • the comparison processing unit 1343b of the pooling layer state comparison unit 1343 executes a process to determine whether the difference between the reference value (previous value) and the current value is less than a threshold value (step ST3221).
  • the comparison processing unit 1341b of the image state comparison unit 1341, the comparison processing unit 1342b of the convolution layer state comparison unit 1342, and the comparison processing unit 1343b of the pooling layer state comparison unit 1343 determine that the difference between the reference value (previous value) and the current value is greater than the threshold value (step ST3213 "NO", step ST3217 “NO”, step ST3221 “NO"), the feature extraction unit 1300C executes normal processing command processing (step ST3311). In the normal processing command process, the feature extraction processing control unit 1381 issues a normal processing command to the first neural network unit 1320 .
  • the feature extraction section 1300C executes an output process (step ST3411).
  • the first neural network portion 1320 of the feature extraction portion 1300C outputs the combined image.
  • the feature extraction section 1300C executes an output storage process (step ST3412).
  • the combined image storage section 1391 of the output storage section in the feature extraction section 1300C stores the combined image output from the first neural network section 1320.
  • the feature extraction unit 1300C executes an omission processing command processing (step ST3312).
  • the feature extraction processing control unit 1381 commands the first neural network unit 1320 not to execute processing in layers subsequent to the layer to be judged, and also commands the first output storage unit 1390 to output the feature map stored therein.
  • the feature extraction section 1300C executes a stored data output process (step ST3413).
  • the feature extraction unit 1300C ends the process shown in FIG.
  • FIG. 19 is a flowchart showing a detailed first example of the processing of the abnormal state determination unit 1500C in the abnormality warning device 100C and the abnormality determination device 1000C according to the third embodiment of the present disclosure.
  • the abnormal condition determination unit 1500C starts processing when it acquires the feature map output from the feature extraction unit 1300C.
  • the abnormal condition determination unit 1500C first executes processing on the data output by the feature extraction unit 1300C (step ST3510).
  • the abnormal state determination unit 1500C acquires the fully connected layer output (step ST3511).
  • the second state comparison section 1540 in the abnormal state determination section 1500C obtains the output value output from the first fully connected layer 1527.
  • the first fully connected layer state comparison unit 1541 executes a first fully connected layer state storage process (step ST3512).
  • the first fully connected layer state comparison unit 1541 stores the output value from the first fully connected layer 1527, which is the input value to the second fully connected layer 1528, as a reference value.
  • the first fully connected layer state comparing unit 1541 executes a process of judging whether the state has been stored previously (step ST3513).
  • the first fully connected layer state comparison unit 1541 refers to the storage unit 1541a to determine whether a reference value (previous value) is stored.
  • the first fully connected layer state comparison unit 1541 determines that the previous value has been stored ("YES" in step ST3513), it executes a comparison process between the current value and the previous value (step ST3514).
  • the first fully connected layer state comparison unit 1541 compares the current value with a reference value (previous value).
  • the first fully connected layer state comparing unit 1541 executes a process of determining whether the difference is smaller than a threshold value (step ST3515).
  • the comparison processing unit 1541b judges whether the difference between the current value and the reference value (previous value) is large or small, using a difference between the current value and the reference value and a pre-stored threshold value.
  • the second fully connected layer state comparison unit 1542 stores the output value from the second fully connected layer 1528, which is the input value to the probability output layer 1529, as a reference value.
  • the second fully connected layer state comparing unit 1542 executes a process of judging whether the state has been stored previously (step ST3517).
  • the second fully connected layer state comparison unit 1542 refers to the storage unit 1542a to determine whether a reference value (previous value) is stored.
  • the second fully connected layer state comparison unit 1542 determines that the previous value has been stored ("YES" in step ST3517), it executes a comparison process between the current value and the previous value (step ST3518).
  • the second fully connected layer state comparison unit 1542 compares the current value with a reference value (previous value).
  • the second fully connected layer state comparing unit 1542 executes a process of determining whether the difference is smaller than the threshold value (step ST3519).
  • the comparison processing unit 1542b judges whether the difference between the current value and the reference value (previous value) is large or small, using a difference between the current value and the reference value (previous value) and a pre-stored threshold value.
  • step ST3515 determines that the difference is greater than the threshold value
  • step ST3519 determines that the difference is greater than the threshold value
  • Abnormal state determination section 1500C executes output processing (step ST3711).
  • Neural network section 1520 (second neural network section 1520) in abnormal state determination section 1500C outputs a determination value.
  • Abnormal state determination section 1500C executes an output storage process (step ST3712).
  • An output storage unit 1590 (second output storage unit 1590) in the abnormal state determination unit 1500C stores the determination value output by the neural network unit 1520 (second neural network unit 1520).
  • the abnormal state determination unit 1500C executes an omission processing command (step ST3519).
  • the abnormal state judgment processing control unit 1581 instructs the second neural network unit 1520 not to execute processing in layers subsequent to the layer to be judged, and also instructs the second output storage unit 1590 to output the judgment value stored therein.
  • Abnormal state determination section 1500C executes a stored data output process (step ST3713).
  • the second output storage section 1590 outputs the decision value.
  • FIG. 20 is a flowchart showing a second detailed example of the processing of the feature extraction unit 1300C in the abnormality warning device 100C and the abnormality determination device 1000C according to the third embodiment of the present disclosure.
  • the first state comparison section 1340C of the feature extraction section 1300C executes an image storage process (step ST3210).
  • the image state comparison section 1341 of the first state comparison section 1340C stores the image that is the input data for the first neural network section 1320. Furthermore, every time the first neural network unit 1320 acquires an image, the image state comparison unit 1341 stores the image in the storage unit 1341a.
  • the image state comparison section 1341 executes a process of determining whether the image has been previously stored (step ST3211).
  • the image state comparison section 1341 refers to the storage section 1341a to determine whether the previously input image is stored therein.
  • the image state comparison unit 1341 executes a process of comparing the current image with the previous image (step ST3212).
  • the comparison processing section 1341b of the image state comparison section 1341 compares the currently input image with the previously input image.
  • the comparison processing unit 1341b executes a process to determine whether the difference is less than a threshold value (step ST3213).
  • the comparison processing unit 1341b determines the magnitude of the difference between the reference value and the current value using the difference value between the currently input image and the previously input image and a pre-stored threshold value.
  • step ST3211 "NO" determines that the value has not been stored previously
  • step ST3213 "NO” determines that the difference between the previous value and the current value is equal to or greater than a threshold value
  • step ST3213 "NO" the feature extraction processing control unit 1381C issues a normal processing command
  • step ST3231 processing is performed by the convolution layer unit 1322.
  • step ST3214 executes a convolution layer state storage process (step ST3214).
  • the convolution layer state comparison unit 1342 stores input data for the convolution layer in the storage unit 1342a.
  • the convolutional layer state comparison unit 1342 executes a process to determine whether a previous value has been stored (step ST3215).
  • the convolutional layer state comparison unit 1342 refers to the storage unit 1342a and determines whether a previous value (reference value), which is a value input previously, has been stored.
  • step ST3215 determines that the previous state has been stored (step ST3215 "YES")
  • step ST3216 executes a comparison process between the current state and the previous state
  • the feature extraction unit 1300C executes a process to determine whether the difference is less than a threshold value (step ST3217). Specifically, the comparison processing unit 1342b of the convolution layer state comparison unit 1342 determines the magnitude of the difference between the previous value and the current value using the difference between the value to be input this time (current value) and the value input last time (previous value) to the pooling layer unit 1325 and a pre-stored threshold value.
  • the convolutional layer state comparison unit 1342 determines that the value was not stored previously ("NO" in step ST3211), or if the comparison processing unit 1342b of the convolutional layer state comparison unit 1342 determines that the difference between the previous value and the current value is equal to or greater than a threshold value ("NO" in step ST3217), the feature extraction processing control unit 1381C issues a normal processing command (step ST3232), and processing is performed by the pooling layer unit 1325.
  • the pooling layer state comparison unit 1343 stores the input data for the pooling layer (the input value to be processed) in the storage unit 1343a.
  • the comparison processing unit 1343b in the pooling layer state comparison unit 1343 executes a process to determine whether the data was previously stored (step ST3219).
  • step ST3219 If the comparison processing unit 1343b of the pooling layer state comparison unit 1343 determines that the value was stored previously ("YES" in step ST3219), it executes a comparison process between the current value and the reference value (previous value) (step ST3220).
  • the comparison processing unit 1343b of the pooling layer state comparison unit 1343 executes a process to determine whether the difference between the reference value (previous value) and the current value is less than a threshold value (step ST3221).
  • the feature extraction unit 1300C executes normal processing command processing (step ST3311).
  • the feature extraction processing control unit 1381 issues a normal processing command to the first neural network unit 1320.
  • the image combination unit 1328 in the first neural network unit 1320 executes image combination processing.
  • the feature extraction section 1300C executes an output storage process (step ST3412).
  • the combined image storage section 1391 of the output storage section in the feature extraction section 1300C stores the combined image output from the first neural network section 1320.
  • the feature extraction unit 1300C executes an omission processing command processing (step ST3312).
  • the feature extraction processing control unit 1381 commands the first neural network unit 1320 not to execute processing in layers subsequent to the layer to be judged, and also commands the first output storage unit 1390 to output the feature map stored therein.
  • the feature extraction section 1300C executes a stored data output process (step ST3413).
  • the feature extraction unit 1300C ends the process shown in FIG.
  • FIG. 21 is a flowchart showing a second detailed example of the processing of the abnormal state determination unit 1500C in the abnormality warning device 100C and the abnormality determination device 1000C according to the third embodiment of the present disclosure.
  • the abnormal condition determination unit 1500C starts processing when it acquires the feature map output from the feature extraction unit 1300C.
  • the abnormal condition determination unit 1500C first executes processing on the data output by the feature extraction unit 1300C (step ST3510).
  • the abnormal state determination unit 1500C acquires the fully connected layer output (step ST3511).
  • the second state comparison section 1540 in the abnormal state determination section 1500C obtains the output value output from the first fully connected layer 1527.
  • the first layer state comparison unit 1541 executes a first layer state storage process (step ST3512).
  • the first fully connected layer state comparison unit 1541 stores the output value from the first fully connected layer 1527 and the input value to the second fully connected layer 1528 (the value before being processed by the second fully connected layer 1528) as a reference value.
  • the first layer state comparing unit 1541 executes a process of determining whether or not the data has been previously stored (step ST3513).
  • the first layer state comparison unit 1541 refers to the storage unit 1541a and determines whether a reference value (previous value) is stored.
  • the first layer state comparison unit 1541 determines that the previous value has been stored ("YES" in step ST3513), it executes a comparison process between the current value and the previous value (step ST3514).
  • the first fully connected layer state comparison unit 1541 compares the current value with a reference value (previous value).
  • the first layer state comparison unit 1541 executes a process of determining whether the difference is smaller than a threshold value (step ST3515).
  • the comparison processing unit 1541b in the first layer state comparison unit 1541 determines the magnitude of the difference between the reference value and the current value using the difference between the current value and the reference value (previous value) and a pre-stored threshold value.
  • the abnormal state determination processing control unit 1581C issues a normal processing command (step ST3520), and processing is performed by the second fully connected layer 1528.
  • the second fully connected layer state comparison unit 1542 executes a second fully connected layer state storage process (step ST3516).
  • the second fully connected layer state comparison unit 1542 stores the output value from the second fully connected layer 1528, which is the input value to the probability output layer 1529 (the value before being processed by the probability output layer 1529), as a reference value.
  • the second fully connected layer state comparison unit 1542 executes a process to determine whether a previous value has been stored (step ST3517).
  • the second fully connected layer state comparison unit 1542 refers to the storage unit 1542a to determine whether a reference value (previous value) has been stored.
  • the second fully connected layer state comparison unit 1542 determines that the previous value has been stored ("YES" in step ST3517), it executes a comparison process between the current value and the previous value (step ST3518).
  • the second fully connected layer state comparison unit 1542 compares the current value with a reference value (previous value).
  • the second fully connected layer state comparison unit 1542 executes a determination process of whether the difference is less than the threshold value (step ST3519).
  • the comparison processing unit 1542b in the second fully connected layer state comparison unit 1542 determines the magnitude of the difference between the reference value and the current value using the difference between the current value and the reference value (previous value) and a pre-stored threshold value.
  • step ST3517 “NO” determines that the data was not previously stored (step ST3517 "NO"), or if the comparison processing unit 1542b in the second fully connected layer state comparison unit 1542 determines that the difference is equal to or greater than the threshold value (step ST3519 "NO")
  • the abnormal state determination unit 1500C executes a normal processing command process (step ST3611).
  • Abnormal state determination section 1500C executes output processing (step ST3711).
  • the second neural network unit 1520 in the abnormal state determination unit 1500C outputs a determination value.
  • Abnormal state determination section 1500C executes an output storage process (step ST3712).
  • the second output storage unit 1590 in the abnormal state determination unit 1500C stores the determination value output by the second neural network unit 1520 in a probability storage unit 1591 .
  • the abnormal state determination unit 1500C executes an omission processing command (step ST3519).
  • the abnormal state judgment processing control unit 1581C instructs the second neural network unit 1520 not to execute processing in layers subsequent to the layer to be judged, and also instructs it to output the judgment value stored in the second output storage unit 1590.
  • the abnormal state determination unit 1500C executes the stored data output process (step ST3713).
  • the second output storage unit 1590 receives a command from the abnormal state determination process control unit 1581C and outputs the determination value (probability value) stored in the probability storage unit 1591.
  • each state storage and comparison processing unit determines that the change is small, it is possible to avoid subsequent processing, thereby reducing the amount of calculations and enabling faster processing. At this time, the result that is output is the previous highly accurate determination result, so that the output of highly accurate results can be continued.
  • the abnormality determination device of the present disclosure is further configured as follows. "The neural network unit includes a first neural network unit and a second neural network unit,
  • the output storage unit includes a first output storage unit and a second output storage unit,
  • the state comparison unit includes a first state comparison unit and a second state comparison unit
  • the processing control unit includes a first processing control unit and a second processing control unit, the first neural network unit having a plurality of layers that are a part of the layers constituting the neural network, acquiring an image output by the image acquisition unit, and outputting a feature map that represents a characteristic state of the object to be determined that is included in the image;
  • the first output storage unit that stores the feature map output by the first neural network unit;
  • the first state comparison unit which stores in advance a reference value that is a reference for an input value to each of a plurality of layers in the first neural network unit, and judges a change in the state of the object to be judged using a difference value between the reference value and a current value that is a value
  • the present disclosure has the effect of providing an abnormality determination device that can use a neural network in an abnormality determination technique to more quickly output highly accurate determination results. Furthermore, the present disclosure achieves the same effect as the above by applying the above configuration to the abnormality determination method.
  • FIG. 22 is a diagram illustrating a first example of a hardware configuration for realizing the functions of the present disclosure.
  • FIG. 23 is a diagram illustrating a second example of a hardware configuration for realizing the functions of the present disclosure.
  • the abnormality warning device 100 (100A, 100B, 100C) and the abnormality determination device 1000 (1000A, 1000B, 1000C) of the present disclosure are each realized by hardware such as that shown in FIG. 22 or FIG. 23.
  • each of the abnormality warning devices 100 (100A, 100B, 100C) and the abnormality determination devices 1000 (1000A, 1000B, 1000C) is configured with, for example, a processor 10001, a memory 10002, and a communication circuit 10004.
  • the processor 10001 and the memory 10002 are installed in, for example, a computer.
  • the memory 10002 stores the computer, an image acquisition unit 1010, a neural network unit 1020, a state comparison unit 1030, a processing control unit 1040, an output storage unit 1050, image acquisition units 1100, 1100B, and 1100C, feature extraction units 1300, 1300B, and 1300C, a first neural network unit 1320, an image branching unit 1321, and a convolution layer unit 1322.
  • the processor 10001 reads out and executes the programs stored in the memory 10002, thereby controlling the image acquisition unit 1010, the neural network unit 1020, the state comparison unit 1030, the processing control unit 1040, the output storage unit 1050, the image acquisition units 1100, 1100B, and 1100C, the feature extraction units 1300, 1300B, and 1300C, the first neural network unit 13 20, image branching unit 1321, convolution layer unit 1322, first convolution layer 1323, second convolution layer 1324, pooling layer unit 1325, first pooling layer 1326, second pooling layer 1327, image combination unit 1328, first state comparison unit 1340, 1340B, 1340C, image state comparison unit 1341, comparison processing unit 1341b, convolution layer state comparison unit 1342, comparison Processing unit 1342b, pooling layer state comparison unit 1343, comparison processing unit 1343b, first processing control units 1380, 1380B, 1380C, feature extraction processing control units 1381, 1381B, 1381C, abnormal state determination units 1500, 1500B, 1500C,
  • a storage unit (not shown) is realized by the memory 10002 or another memory (not shown). Furthermore, the storage units 1341a, 1342a, 1343a, the first output storage unit 1390, the combined image storage unit 1391, the storage units 1541a, 1542a, the second output storage unit 1590, and the probability storage unit 1591 in the abnormality warning device 100 (100A, 100B, 100C) and the abnormality determination device 1000 (1000A, 1000B, 1000C) are realized by the memory 10002 or another memory (not shown). Furthermore, the communication circuit 10004 realizes a communication unit (not shown).
  • the processor 10001 is, for example, a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, a microcontroller, or a digital signal processor (DSP).
  • the memory 10002 may be a non-volatile or volatile semiconductor memory such as a RAM (Random Access Memory), a ROM (Read Only Memory), an EPROM (Erasable Programmable ROM), an EEPROM (Electrically Erasable Programmable Read Only Memory), or a flash memory, or a magnetic disk such as a hard disk or a flexible disk, or an optical disk such as a CD (Compact Disc) or a DVD (Digital Versatile Disc), or a magneto-optical disk.
  • RAM Random Access Memory
  • ROM Read Only Memory
  • EPROM Erasable Programmable ROM
  • EEPROM Electrically Erasable Programmable Read Only Memory
  • flash memory or a magnetic disk such as a hard disk or a flexible disk, or an optical disk such as a CD (Com
  • the processor 10001 and the memory 10002 or the communication circuit 10004 are connected in a state in which data can be transmitted between them.
  • the processor 10001, the memory 10002, and the communication circuit 10004 are also connected via the input/output interface 10003 in a state in which data can be transmitted between them and other hardware.
  • the communication circuit 10004 realizes a communication unit (not shown).
  • the processing circuit 20001 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), an FPGA (Field-Programmable Gate Array), a SoC (System-on-a-Chip), or a system LSI (Large-Scale Integration).
  • the memory 20002 or another memory (not shown) implements a storage unit (not shown).
  • the memory 20002 may be a non-volatile or volatile semiconductor memory such as a RAM (Random Access Memory), a ROM (Read Only Memory), an EPROM (Erasable Programmable ROM), an EEPROM (Electrically Erasable Programmable Read Only Memory), or a flash memory, or a magnetic disk such as a hard disk or a flexible disk, or an optical disk such as a CD (Compact Disc) or a DVD (Digital Versatile Disc), or a magneto-optical disk.
  • the communication circuit 20004 realizes a communication unit (not shown).
  • the processing circuit 20001 and the memory 20002 or the communication circuit 20004 are connected in a state in which they can transmit data to each other.
  • the processing circuit 20001, the memory 20002, and the communication circuit 20004 are connected in a state in which they can transmit data to other hardware via the input/output interface 20003.
  • the present disclosure provides an abnormality determination technology that can quickly output highly accurate determination results using a neural network without performing calculations in all layers of the neural network. For example, this enables rapid and highly accurate abnormality determination, such as drowsiness detection, and is therefore suitable for application to in-vehicle driver monitoring systems.
  • 100, 100A, 100B, 100C abnormality warning device, 1000, 1000A, 1000B, 1000C: abnormality determination device, 1010: image acquisition unit, 1020: neural network unit, 1030: state comparison unit, 1040: processing control unit, 1050: output storage unit, 1100, 1100B, 1100C: image acquisition unit, 1300, 1300B, 1300C: feature extraction unit, 1320: neural network unit (first neural network unit), 1321: image branching unit, 1322: convolution layer unit, 1323: first convolution layer, 1324: second convolution layer, 1325 pooling layer unit 1326 first pooling layer, 1327 second pooling layer, 1328 image combination unit, 1340, 1340B, 1340C state comparison unit (first state comparison unit), 1341 image state comparison unit, 1341a storage unit, 1341b comparison processing unit, 1342 convolution layer state comparison unit, 1342a storage unit, 1342b comparison processing unit, 1343 pooling layer state comparison unit, 1343a storage unit, 1343b comparison processing unit, 1380, 1380

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Image Analysis (AREA)
PCT/JP2023/018362 2023-05-17 2023-05-17 異常判定装置、および、異常判定方法 Ceased WO2024236748A1 (ja)

Priority Applications (2)

Application Number Priority Date Filing Date Title
JP2025520317A JP7696532B2 (ja) 2023-05-17 2023-05-17 異常判定装置、および、異常判定方法
PCT/JP2023/018362 WO2024236748A1 (ja) 2023-05-17 2023-05-17 異常判定装置、および、異常判定方法

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2023/018362 WO2024236748A1 (ja) 2023-05-17 2023-05-17 異常判定装置、および、異常判定方法

Publications (1)

Publication Number Publication Date
WO2024236748A1 true WO2024236748A1 (ja) 2024-11-21

Family

ID=93518926

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2023/018362 Ceased WO2024236748A1 (ja) 2023-05-17 2023-05-17 異常判定装置、および、異常判定方法

Country Status (2)

Country Link
JP (1) JP7696532B2 (https=)
WO (1) WO2024236748A1 (https=)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH06139224A (ja) * 1992-09-08 1994-05-20 Hitachi Ltd 情報処理装置および監視装置
US20200174748A1 (en) * 2018-11-30 2020-06-04 Advanced Micro Devices, Inc. Sorting Instances of Input Data for Processing through a Neural Network
US20210072984A1 (en) * 2019-09-10 2021-03-11 Micron Technology, Inc. Re-USING PROCESSING ELEMENTS OF AN ARTIFICIAL INTELLIGENCE PROCESSOR
WO2022070947A1 (ja) * 2020-09-30 2022-04-07 ソニーセミコンダクタソリューションズ株式会社 信号処理装置、撮像装置、信号処理方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH06139224A (ja) * 1992-09-08 1994-05-20 Hitachi Ltd 情報処理装置および監視装置
US20200174748A1 (en) * 2018-11-30 2020-06-04 Advanced Micro Devices, Inc. Sorting Instances of Input Data for Processing through a Neural Network
US20210072984A1 (en) * 2019-09-10 2021-03-11 Micron Technology, Inc. Re-USING PROCESSING ELEMENTS OF AN ARTIFICIAL INTELLIGENCE PROCESSOR
WO2022070947A1 (ja) * 2020-09-30 2022-04-07 ソニーセミコンダクタソリューションズ株式会社 信号処理装置、撮像装置、信号処理方法

Also Published As

Publication number Publication date
JPWO2024236748A1 (https=) 2024-11-21
JP7696532B2 (ja) 2025-06-20

Similar Documents

Publication Publication Date Title
US11921212B2 (en) Long range lidar-based speed estimation
EP3509011B1 (en) Apparatuses and methods for recognizing a facial expression robust against change in facial expression
US20190164057A1 (en) Mapping and quantification of influence of neural network features for explainable artificial intelligence
EP3674974B1 (en) Apparatus and method with user verification
CN107303907B (zh) 用于确定驾驶者的睡意的装置和方法
US20190095706A1 (en) Image processing device and program
CN113807298B (zh) 行人过街意图预测方法、装置、电子设备及可读存储介质
US12249128B2 (en) Method, device, and computer program for an uncertainty assessment of an image classification
KR102476022B1 (ko) 얼굴검출 방법 및 그 장치
KR102883351B1 (ko) 라이브니스 검사 방법 및 라이브니스 검사 장치
JP7696532B2 (ja) 異常判定装置、および、異常判定方法
US12354292B2 (en) Object ranging apparatus, method, and computer readable medium
Alzami et al. Time Distributed MobileNetV2 with Auto-CLAHE for Eye Region Drowsiness Detection in Low Light Conditions.
US11636698B2 (en) Image processing method and apparatus with neural network adjustment
CN115082875B (zh) 使用深度卷积网络的形状偏差的图像分类
CN112714916A (zh) 机器学习系统及创建机器学习系统的方法、计算机程序和设备
EP4125063B1 (en) Methods and systems for occupancy class prediction and methods and systems for occlusion value determination
CN113728355B (zh) 一种图像处理方法和装置
CN114186223A (zh) 通过将一维测量结果变换成多维图像来改进入侵检测系统的测量结果的系统和方法
EP4002270A1 (en) Image recognition evaluation program, image recognition evaluation method, evaluation device, and evaluation system
EP3792814B1 (en) Method and system for selecting an operation mode for an at least partly self-driving vehicle
JP7175381B2 (ja) 覚醒度推定装置、自動運転支援装置および覚醒度推定方法
US20230186586A1 (en) Method and apparatus with object detection
KR20210079823A (ko) 전자 장치 및 이의 제어 방법
CN113168571A (zh) 用于训练神经网络的方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23937490

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2025520317

Country of ref document: JP

NENP Non-entry into the national phase

Ref country code: DE