US11416610B2 - Threat information evaluation apparatus, threat information evaluation method and program - Google Patents

Threat information evaluation apparatus, threat information evaluation method and program Download PDF

Info

Publication number
US11416610B2
US11416610B2 US17/041,589 US201917041589A US11416610B2 US 11416610 B2 US11416610 B2 US 11416610B2 US 201917041589 A US201917041589 A US 201917041589A US 11416610 B2 US11416610 B2 US 11416610B2
Authority
US
United States
Prior art keywords
threat information
evaluation
estimation accuracy
security operator
evaluation module
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active, expires
Application number
US17/041,589
Other languages
English (en)
Other versions
US20210019405A1 (en
Inventor
Hidetoshi Kawaguchi
Yuichi Ishihara
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nippon Telegraph and Telephone Corp
Original Assignee
Nippon Telegraph and Telephone Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nippon Telegraph and Telephone Corp filed Critical Nippon Telegraph and Telephone Corp
Assigned to NIPPON TELEGRAPH AND TELEPHONE CORPORATION reassignment NIPPON TELEGRAPH AND TELEPHONE CORPORATION ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: ISHIHARA, YUICHI, KAWAGUCHI, Hidetoshi
Publication of US20210019405A1 publication Critical patent/US20210019405A1/en
Application granted granted Critical
Publication of US11416610B2 publication Critical patent/US11416610B2/en
Active legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/50—Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
    • G06F21/55—Detecting local intrusion or implementing counter-measures
    • G06F21/56—Computer malware detection or handling, e.g. anti-virus arrangements
    • G06F21/561—Virus type analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/50—Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
    • G06F21/55—Detecting local intrusion or implementing counter-measures
    • G06F21/552—Detecting local intrusion or implementing counter-measures involving long-term monitoring or reporting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/50—Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
    • G06F21/55—Detecting local intrusion or implementing counter-measures
    • G06F21/554—Detecting local intrusion or implementing counter-measures involving event detection and direct action
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/50—Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
    • G06F21/57—Certifying or maintaining trusted computer platforms, e.g. secure boots or power-downs, version controls, system software checks, secure updates or assessing vulnerabilities
    • G06F21/577—Assessing vulnerabilities and evaluating computer system security
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2221/00—Indexing scheme relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F2221/03—Indexing scheme relating to G06F21/50, monitoring users, programs or devices to maintain the integrity of platforms
    • G06F2221/034—Test or assess a computer or a system

Definitions

  • the present invention relates to a threat information evaluation apparatus, a threat information evaluation method, and a program, and to a technique which is applicable when a security operator handling a security system against a cyber-attack such as an Intrusion Prevention System (IPS) or anti-virus software evaluates threat information.
  • IPS Intrusion Prevention System
  • anti-virus software evaluates threat information.
  • Threat information related cyber-attack activities including attackers, attackers' behavior, criminal techniques, vulnerabilities, and the like is generated every day.
  • a security operator needs to continually and sequentially evaluate threat information.
  • Threat information and a security operator's evaluation for the threat information can be accumulated in a database and retrieved as necessary, but the evaluation of an enormous amount of threat information requires significant efforts by the security operator.
  • Non-Patent Literature 1 Scikit-learn user guide, Release 0.19.1, Internet ⁇ URL: http://scikit-learn.org/stable/_downloads/scikit-learn-docs.pdf>, retrieved Mar. 2, 2018
  • Non-Patent Literature 2 Various hyper parameter automatic adjustment, Internet ⁇ URL: https://qiita.com/cvusk/items/1f3b178f34c39beb29ff>, retrieved Mar.
  • Non-Patent Literature 3 Know-how of teaming in deep learning and guidelines for hyper parameter, Interne ⁇ URL: https://qiita.com/akimach/items/88356ef46e98f045b035>, retrieved Mar. 2, 2018
  • a security operator needs to evaluate threat information, but there is a possibility that evaluation cannot be performed in a case where the amount of threat information itself becomes enormous.
  • the inability to evaluate threat information may lead to a possibility that a cyber-attack cannot be prevented, which is undesirable for an organization that is an operation target.
  • machine learning requires tuning for generating an appropriate pattern using an evaluation module, and a security operator needs to input results of evaluation of several pieces of threat information into the evaluation module as learning data. That is, the security operator needs to divide threat information into threat information to be evaluated by the security operator directly and threat information to be transmitted to the evaluation module. Transmission of all threat information to the evaluation module may lead to a risk of deterioration in the accuracy of evaluation of the evaluation module due to the obsolescence of patterns of the module, and thus allocation is essential.
  • the allocation of threat information requires effort of a security operator.
  • the threat information as an evaluation target is generated every day, and thus it is necessary to allocate threat information every time it is generated, which results in a problem that a significant amount of effort is required.
  • An object of the present invention is to reduce the required amount of effort of a security operator in performing allocation by automating the allocation of threat information.
  • a threat information evaluation apparatus is a threat information evaluation apparatus that evaluates threat information and includes an allocation unit that allocates threat information in an input threat information list to a security operator or an evaluation unit, and an evaluation unit that evaluates the threat information allocated to the evaluation unit, wherein the allocation unit calculates an estimation accuracy on the basis of evaluation performed by the evaluation unit and evaluation determined by the security operator, and allocates the threat information on the basis of the estimation accuracy.
  • a threat information evaluation method is a threat information evaluation method executed by a threat information evaluation apparatus including an evaluation unit that evaluates threat information, and the method includes calculating an estimation accuracy on the basis of evaluation performed by the evaluation unit and evaluation determined by a security operator, inputting a threat information list, and allocating threat information in the input threat information list to the security operator or the evaluation unit on the basis of the estimation accuracy.
  • a program according to still another aspect of the present invention causes a computer to function as each unit of the above-described threat information evaluation apparatus.
  • FIG. 1 is a diagram illustrating a functional configuration of a threat information evaluation apparatus according to an example of the present invention.
  • FIG. 2 is a diagram illustrating an example of threat information.
  • FIG. 3 is a diagram illustrating an example of evaluation of threat information.
  • FIG. 4 is a flowchart illustrating processing in an evaluation module.
  • FIG. 5 is a diagram illustrating an example of a confusion matrix.
  • FIG. 6 is a flowchart illustrating processing in an allocation module.
  • FIG. 7 is a diagram illustrating a hardware configuration example of a threat information evaluation apparatus.
  • FIG. 1 is a diagram illustrating a functional configuration of a threat information evaluation apparatus according to an example of the present invention.
  • a threat information evaluation apparatus 100 is an apparatus that evaluates threat information such as evaluating the importance of threat information.
  • the threat information evaluation apparatus 100 includes an allocation module 101 , a database 102 , and an evaluation module 103 .
  • the allocation module 101 is a processing unit that receives a threat information list as an input from a threat information generation source and allocates threat information in the threat information list to a security operator or the evaluation module 103 .
  • the threat information generation source is where threat information for the security operator is acquired, and is, for example, a vendor that develops IPS signatures or a platform that shares threat information.
  • FIG. 2 is a diagram illustrating an example of threat information.
  • Threat information includes information regarding a cyber-attack and includes content such as an ID which is an identifier of threat information, an IP address of a transmission destination/destination, a port number, and content of a cyber-attack.
  • the security operator is a person who receives threat information, performs the evaluation of the threat information, and provides the evaluation results to a threat information evaluation destination.
  • the threat information evaluation destination is a destination to which a pair including threat information and the evaluation of the threat information is to be applied, and corresponds to, for example, an IPS itself in a case where threat information is a signature of the IPS.
  • FIG. 3 is a diagram illustrating an example of evaluation performed by a security operator or the evaluation module 103 .
  • the importance or the like is determined by the security operator or the evaluation module 103 for each piece of threat information illustrated in FIG. 2 .
  • the allocation module 101 allocates threat information to the security operator.
  • the allocation module 101 can allocate threat information to the evaluation module 103 , but allocates a portion of the threat information to the security operator to prevent the obsolescence of the pattern.
  • the database 102 is a storage medium that stores threat information and evaluation performed by the security operator in association with each other. Information stored in the database 102 is used to derive a pattern for performing evaluation and the accuracy of the evaluation.
  • the evaluation module 103 is a processing unit that evaluates threat information allocated to the evaluation module 103 .
  • the evaluation module 103 can generate a pattern for performing evaluation by machine learning or the like. For example, the evaluation module 103 generates a pattern on the basis of pairs each including threat information stored in the database 102 and evaluation performed by the security operator, and evaluates received threat information using the pattern.
  • data used to calculate the performance of a pattern for performing evaluation for the allocation module 101 for example, a confusion matrix, is also transmitted.
  • the allocation module 101 calculates an estimation accuracy on the basis of evaluation performed by the evaluation module 103 and evaluation determined by the security operator, and performs allocation on the basis of the estimation accuracy. For example, the allocation module 101 receives a confusion matrix of evaluation from the evaluation module 103 , calculates an estimation accuracy on the basis of the confusion matrix, and divides the threat information into the threat information to be transmitted to the security operator and the threat information to be transmitted to the evaluation module 103 based on the estimation accuracy. threat information transmitted to the evaluation module 103 can be automatically evaluated, thereby increasing the effect. On the other hand, when the transmission of threat information to the evaluation module 103 is excessively performed, the accuracy of evaluation of the evaluation module 103 may deteriorate. For this reason, it is also necessary to appropriately transmit threat information to the security operator in order to adjust parameters of the evaluation module 103 .
  • the evaluation module 103 receives threat information allocated by the allocation module 101 (S 101 ) and receives the security operator's evaluation from the database 102 (S 102 ).
  • the evaluation module 103 can generate a pattern for performing evaluation using a method of tuning hyper parameters of machine learning with data (S 103 ). In addition, threat information is evaluated using the generated pattern.
  • the evaluation module 103 generates a confusion matrix on the basis of evaluation performed by the evaluation module and evaluation determined by the security operator (S 104 ).
  • FIG. 5 is a diagram illustrating an example of a confusion matrix.
  • a confusion matrix a matrix used for evaluating the performance of a classifier or a model of machine learning. There are several quantitative indexes for measuring performance, but a generally used index can be calculated from elements of the confusion matrix.
  • a specific example will be described in which there are two types of evaluations of “important” and “unimportant,” a total number of pieces of threat information is 2000, the number of pieces of threat information assumed to be actually “important” by a security operator is 1000, and the number of pieces of threat information assumed to be “unimportant” is 1000.
  • a total number of pieces of threat information is 2000
  • the number of pieces of threat information assumed to be actually “important” by a security operator is 1000
  • the number of pieces of threat information assumed to be “unimportant” is 1000.
  • the evaluation module 103 evaluated 900 pieces of threat information, which actually should be 1000 pieces, as “important” and evaluated 800 pieces of threat information as “unimportant.” That is, 900+800 corresponds to the number of cases correctly evaluated, and 100+200 corresponds to the number of cases incorrectly evaluated.
  • n-dimensional confusion matrix A can be generalized n-dimensionally.
  • An n-dimensional confusion matrix A will be described below.
  • A [ a 1 , 1 ... a 1 , j ... a 1 , n ⁇ ⁇ ⁇ ⁇ a i , 1 ... a i , j ... a i , n ⁇ ⁇ ⁇ ⁇ ⁇ a n , 1 ... a n , j ... a n , n ] [ Formula ⁇ ⁇ 1 ]
  • i and j are IDs indicating evaluation.
  • a i,j represents the number of pieces of threat information evaluated to be i by the security operator and to be j by the evaluation module 103 .
  • the confusion matrix of FIG. 5 can be expressed as follows.
  • the confusion matrix A [ 900 100 200 800 ] [ Formula ⁇ ⁇ 2 ]
  • the confusion matrix A is transmitted to the allocation module 101 for processing in the allocation module 101 .
  • the allocation module 101 receives a threat information list from a threat information generation source (S 201 ) and receives a confusion matrix A from the evaluation module 103 (S 202 ). Note that it is assumed that the confusion matrix A is generated for a threat information list prior to the threat information list received in step S 201 .
  • the allocation module 101 calculates an estimation accuracy ⁇ on the basis of the confusion matrix a (S 203 ).
  • a method of calculating the estimation accuracy ⁇ essentially needs to be set in consideration of a situation of a security operator, or the like, but a simple specific example will be described below.
  • the estimation accuracy ⁇ indicates a percentage of correct answers in the evaluation of all threat information, and can be obtained by the sum of diagonal components with respect to the sum of all components of an n-dimensional confusion matrix.
  • the allocation module 101 compares the estimation accuracy ⁇ with a threshold value ⁇ (S 204 ).
  • ⁇ ⁇ pieces of threat information in a threat information list are allocated to the security operator, and the rest of the threat information is allocated to the evaluation module 103 (S 205 ).
  • ⁇ and ⁇ are parameters which are set in advance. Note that ⁇ pieces of threat information may be allocated to both the security operator and the evaluation module 103 .
  • estimation accuracy ⁇ and the threshold value ⁇ are compared with each other and ⁇ pieces of threat information in the threat information list are allocated to the security operator in step S 205 of FIG. 6
  • an allocation ratio may be calculated on the basis of the estimation accuracy ⁇ , and it may be determined whether to allocate threat information to the security operator or allocate threat information to the evaluation module 103 on the basis of the allocation ratio.
  • the allocation module 101 is not provided. Thus, a security operator needs to receive all threat information from a threat information generation source and perform determination regarding allocation directly.
  • the estimation accuracy of the evaluation module 103 at the present time, or the like is required to be applied to allocation, and thus a non-negligible amount of effort is required.
  • the allocation module 101 can allocate threat information when a security operator receives threat information from a vendor providing an IPS or acquires threat information from the Internet. In a case where the estimation accuracy does not exceed the threshold value, all of the threat information is allocated to the security operator. This corresponds to a case where a sufficient amount of data is not available for pattern generation, such as an initial state.
  • a security operator evaluates each piece of threat information, and a pair including threat information and the evaluation of the threat information is transmitted to a threat information evaluation destination. At the same time, the pair including the threat information and the evaluation thereof is copied to the database 102 to be used for pattern generation of the evaluation module 103 .
  • the evaluation module 103 evaluates the threat information in accordance with a pattern.
  • the security operator also evaluates the received threat information, transmits a pair including the threat information and the evaluation of the threat information to a threat information evaluation destination, and copies the pair to the database 102 .
  • threat information evaluated by the security operator is only a portion of the entirety, and thus the effort of the security operator is reduced.
  • automation of a pattern of the evaluation module 103 is automatically determined, and it is possible to dynamically switch whether the security operator evaluates all threat information or the security operator evaluates only a portion.
  • FIG. 7 illustrates a hardware configuration example of the threat information evaluation apparatus 100 according to the example of the present invention.
  • the threat information evaluation apparatus 100 may be a computer including a processor such as a central processing unit (CPU) 151 , a memory device 152 such as a random access memory (RAM) or a read only memory (ROM), a storage device 153 such as a hard disk, an input and output interface device 154 , and the like.
  • a processor such as a central processing unit (CPU) 151
  • a memory device 152 such as a random access memory (RAM) or a read only memory (ROM)
  • storage device 153 such as a hard disk
  • an input and output interface device 154 an input and output interface device 154
  • functions and processing of the threat information evaluation apparatus 100 are achieved by the CPU 151 executing data or a program stored in the storage device 153 or the memory device 152 .
  • the input of data to the threat information evaluation apparatus 100 may be performed using the input and output interface device 154
  • the threat information evaluation apparatus may be realized by hardware, software, or a combination thereof.
  • examples of the present invention may be realized by a program for causing a computer to realize the functions of the threat information evaluation apparatus according to examples of the present invention, a program for causing a computer to execute each procedure of the method according to examples of the present invention, or the like.
  • respective functional units may be used in combination, as necessary.
  • the method according to the example of the present invention may be performed in an order different from the order shown in the example.
US17/041,589 2018-03-27 2019-03-25 Threat information evaluation apparatus, threat information evaluation method and program Active 2039-04-20 US11416610B2 (en)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
JPJP2018-060577 2018-03-27
JP2018060577A JP6710716B2 (ja) 2018-03-27 2018-03-27 脅威情報評価装置、脅威情報評価方法およびプログラム
JP2018-060577 2018-03-27
PCT/JP2019/012498 WO2019188975A1 (ja) 2018-03-27 2019-03-25 脅威情報評価装置、脅威情報評価方法およびプログラム

Publications (2)

Publication Number Publication Date
US20210019405A1 US20210019405A1 (en) 2021-01-21
US11416610B2 true US11416610B2 (en) 2022-08-16

Family

ID=68060018

Family Applications (1)

Application Number Title Priority Date Filing Date
US17/041,589 Active 2039-04-20 US11416610B2 (en) 2018-03-27 2019-03-25 Threat information evaluation apparatus, threat information evaluation method and program

Country Status (3)

Country Link
US (1) US11416610B2 (ja)
JP (1) JP6710716B2 (ja)
WO (1) WO2019188975A1 (ja)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120232679A1 (en) * 2008-05-12 2012-09-13 Abercrombie Robert K Cyberspace security system
US8832832B1 (en) * 2014-01-03 2014-09-09 Palantir Technologies Inc. IP reputation
US9805338B1 (en) * 2017-03-03 2017-10-31 Shogun Enterprises Inc. Database system and user interfaces for matching related entities
US20210367962A1 (en) * 2018-10-02 2021-11-25 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e. V. Automatic information security risk assessment

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008129714A (ja) * 2006-11-17 2008-06-05 Univ Of Tsukuba 異常検知方法、異常検知装置及び異常検知用プログラム並びに学習モデル生成方法
US10599997B2 (en) * 2016-08-11 2020-03-24 International Business Machines Corporation System and method for ground truth evaluation

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120232679A1 (en) * 2008-05-12 2012-09-13 Abercrombie Robert K Cyberspace security system
US8832832B1 (en) * 2014-01-03 2014-09-09 Palantir Technologies Inc. IP reputation
US9805338B1 (en) * 2017-03-03 2017-10-31 Shogun Enterprises Inc. Database system and user interfaces for matching related entities
US20210367962A1 (en) * 2018-10-02 2021-11-25 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e. V. Automatic information security risk assessment

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
[No Author Listed], "Scikit-learn User Guide Release 0.23.1," Scikit-learn Developers, May 19, 2020, 2,751 pages.
Qiita.com, [online], "Hyperparameter Automatic Adjustment." Sep. 11, 2017, retrieved on Feb. 16. 2018, retrieved from URL<https://qiita.com/cvusk/items/lf3bl78f34c39beb29ff>. 33 pages (with English Translation).
Qiita.com, [online], "Learning Tips and Hyperparameter Guidelines for Deep Learning," Jan. 18, 2018, retrieved on Feb. 16, 2018, retrieved from URL<https://qiita.com/akimacb/items/88356ef46e98f045b035>, 20 pages (with English Translation).

Also Published As

Publication number Publication date
WO2019188975A1 (ja) 2019-10-03
JP6710716B2 (ja) 2020-06-17
US20210019405A1 (en) 2021-01-21
JP2019174988A (ja) 2019-10-10

Similar Documents

Publication Publication Date Title
US11663609B2 (en) Method and apparatus to enforce smart contract execution hierarchy on blockchain
US8606905B1 (en) Automated determination of system scalability and scalability constraint factors
US10387685B2 (en) Method and device for invoking java card object
US9558346B1 (en) Information processing systems with security-related feedback
US9785430B2 (en) Methods, systems and computer-readable media for detecting a partial commit
US20150178419A1 (en) Method and apparatus for estimating a completion time for mapreduce jobs
US11436535B2 (en) Artificial intelligence based application testing and scoring
US20170345102A1 (en) Computer readable recording medium, extraction apparatus, and extraction method
CN108510399B (zh) 投保单自动分配的方法、装置、计算机设备及存储介质
KR20180121459A (ko) 보안 위험 평가 기반 보안 투자 방법 및 장치
US11416610B2 (en) Threat information evaluation apparatus, threat information evaluation method and program
EP3843318A1 (en) Apparatus and method for performing homomorphic operation using approximation function
US9342686B2 (en) Systems and methods for updating scanning rules
EP2966568A1 (en) Stream processing method, apparatus, and system
US11372884B2 (en) Information processing system, information processing apparatus, information processing method, and computer program product
US11005869B2 (en) Method for analyzing cyber threat intelligence data and apparatus thereof
CN111819559A (zh) 以量化步长使用机器学习模型用于恶意软件检测
CN111587433A (zh) 安保评价服务器和安保评价方法
Osipenko et al. Hadamard and Schwarz type theorems and optimal recovery in spaces of analytic functions
Khan et al. A code coverage-based test suite reduction and prioritization framework
US20170262780A1 (en) Adjustable robust optimization for process execution
US20110050924A1 (en) Managing device, image formation processing system, computer readable medium storing program, and image forming method
JP2016081168A (ja) 雑音発生装置、雑音発生方法及びプログラム
US11588714B2 (en) Web quality estimation apparatus and program
Galyean Orthogonal PSF taxonomy for human reliability analysis

Legal Events

Date Code Title Description
FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

STPP Information on status: patent application and granting procedure in general

Free format text: APPLICATION DISPATCHED FROM PREEXAM, NOT YET DOCKETED

AS Assignment

Owner name: NIPPON TELEGRAPH AND TELEPHONE CORPORATION, JAPAN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:KAWAGUCHI, HIDETOSHI;ISHIHARA, YUICHI;SIGNING DATES FROM 20200708 TO 20200713;REEL/FRAME:054600/0281

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS

STPP Information on status: patent application and granting procedure in general

Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED

STCF Information on status: patent grant

Free format text: PATENTED CASE