Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V20/00—Scenes; Scene-specific elements
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/40—Extraction of image or video features
  • G06V10/42—Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/98—Detection or correction of errors, e.g. by rescanning the pattern or by human intervention; Evaluation of the quality of the acquired patterns
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V20/00—Scenes; Scene-specific elements
  • G06V20/50—Context or environment of the image
  • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle

Definitions

• the present invention relates to a perception system and, more specifically, to a system that evaluates and corrects perception errors to optimize perception results through contrast and entropy-based perception adaptation using probabilistic signal temporal logic-based optimization.
• the present disclosure provides a system for contrast and entropy-based perception adaption to optimize perception.
• the system includes one or more processors and a memory.
• the memory is a non-transitory computer-readable medium having executable instructions encoded thereon, such that upon execution of the instructions, the one or more processors perform several operations.
• the system is operable for receiving an input image of a scene with a perception module (i.e., camera system) and detecting one or more objects (having perception data) in the input image.
• the perception data of the one or more objects is converted into probes, which are then converted into axioms using probabilistic signal temporal logic.
• the axioms are evaluated based on probe bounds.
• the system estimates optimal contrast bounds and entropy bounds as perception parameters (i.e., camera system parameters). The contrast and entropy in the camera system are then adjusted based on the perception parameters.
• perception parameters i.e., camera system parameters
• image kernels are applied such that if a change in entropy is positive, a sharpening filter is applied to increase entropy, and if a change in entropy is negative, a smoothing filter is applied to decrease entropy.
• adjusting contrast includes acquiring a desirable contrast deviation, such that once a desirable contrast deviation is acquired, histogram ranges are set to achieve contrast changes using a peak-to-peak contrast.
• the camera system is incorporated into an adaptive sensor system of an autonomous vehicle or an unmanned aircraft system.
• the present invention also includes a computer program product and a computer implemented method.
• the computer program product includes computer-readable instructions stored on a non-transitory computer-readable medium that are executable by a computer having one or more processors, such that upon execution of the instructions, the one or more processors perform the operations listed herein.
• the computer implemented method includes an act of causing a computer to execute such instructions and perform the resulting operations.
• FIG. 1 is a block diagram depicting the components of a system according to various embodiments of the present invention
• FIG. 2 is an illustration of a computer program product embodying an aspect of the present invention
• FIG. 3 is a flowchart depicting an overview of a system according to various embodiments of the present invention.
• FIG. 4 is a graph depicting probability distributions of a probe according to various embodiments of the present invention.
• FIG. 5A is a graph depicting adjusting entropy using a sharpening filter
• FIG. 5B is a graph depicting adjusting entropy using a smoothing filter
• FIG. 6 is an illustration depicting samples of error reduction and detection improvement
• FIG. 7A is a precision-recall curve for different detection thresholds
• FIG. 7B is a receiver operating characteristics (ROC) curve for different detection thresholds.
• the present invention relates to a perception system and, more specifically, to a system that evaluates and corrects perception errors to optimize perception results through contrast and entropy-based perception adaptation using probabilistic signal temporal logic-based optimization.
• the following description is presented to enable one of ordinary skill in the art to make and use the invention and to incorporate it in the context of particular applications. Various modifications, as well as a variety of uses in different applications, will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to a wide range of aspects. Thus, the present invention is not intended to be limited to the aspects presented, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
• any element in a claim that does not explicitly state “means for” performing a specified function, or “step for” performing a specific function, is not to be interpreted as a “means” or “step” clause as specified in 35 U.S.C. Section 112(f).
• the use of “step of’ or “act of’ in the claims herein is not intended to invoke the provisions of 35 U.S.C. 112(f).
• Luminance Contrast found at colorusage .arc. nasa gov/luminance_cont.php, taken on December 02, 2020.
• Various embodiments of the invention include three “principal” aspects.
• the first is a system for contrast and entropy-based perception adaption to optimize perception.
• the system is typically in the form of a computer system operating software or in the form of a “hard-coded” instruction set. This system may be incorporated into a wide variety of devices that provide different functionalities.
• the second principal aspect is a method, typically in the form of software, operated using a data processing system (computer).
• the third principal aspect is a computer program product.
• the computer program product generally represents computer-readable instructions stored on a non-transitory computer-readable medium such as an optical storage device, e.g., a compact disc (CD) or digital versatile disc (DVD), or a magnetic storage device such as a floppy disk or magnetic tape.
• a non-transitory computer-readable medium such as an optical storage device, e.g., a compact disc (CD) or digital versatile disc (DVD), or a magnetic storage device such as a floppy disk or magnetic tape.
• a non-transitory computer-readable medium such as an optical storage device, e.g., a compact disc (CD) or digital versatile disc (DVD), or a magnetic storage device such as a floppy disk or magnetic tape.
• CD compact disc
• DVD digital versatile disc
• magnetic storage device such as a floppy disk or magnetic tape.
• Other, non-limiting examples of computer-readable media include hard disks, read-only memory (ROM), and flash-type memories.
• FIG. 1 A block diagram depicting an example of a system (i.e., computer system
• the computer system 100 is configured to perform calculations, processes, operations, and/or functions associated with a program or algorithm.
• certain processes and steps discussed herein are realized as a series of instructions (e.g., software program) that reside within computer readable memory units and are executed by one or more processors of the computer system 100. When executed, the instructions cause the computer system 100 to perform specific actions and exhibit specific behavior, such as described herein.
• the computer system 100 can be embodied in any device(s) that operates to perform the functions as described herein as applicable to the particular application, such as a desktop computer, a mobile or smart phone, a tablet computer, a computer embodied in a mobile platform, or any other device or devices that can individually and/or collectively execute the instructions to perform the related operations/processes.
• the computer system 100 may include an address/data bus 102 that is configured to communicate information. Additionally, one or more data processing units, such as a processor 104 (or processors), are coupled with the address/data bus 102.
• the processor 104 is configured to process information and instructions.
• the processor 104 is a microprocessor.
• the processor 104 may be a different type of processor such as a parallel processor, application-specific integrated circuit (ASIC), programmable logic array (PLA), complex programmable logic device (CPLD), or a field programmable gate array (FPGA) or any other processing component operable for performing the relevant operations.
• ASIC application-specific integrated circuit
• PLA programmable logic array
• CPLD complex programmable logic device
• FPGA field programmable gate array
• the computer system 100 is configured to utilize one or more data storage units.
• the computer system 100 may include a volatile memory unit 106 (e.g., random access memory (“RAM”), static RAM, dynamic RAM, etc.) coupled with the address/data bus 102, wherein a volatile memory unit 106 is configured to store information and instructions for the processor 104.
• RAM random access memory
• static RAM static RAM
• dynamic RAM dynamic RAM
• the computer system 100 further may include a non-volatile memory unit 108 (e.g., read-only memory (“ROM”), programmable ROM (“PROM”), erasable programmable ROM (“EPROM”), electrically erasable programmable ROM “EEPROM”), flash memory, etc.) coupled with the address/data bus 102, wherein the non-volatile memory unit 108 is configured to store static information and instructions for the processor 104.
• the computer system 100 may execute instructions retrieved from an online data storage unit such as in “Cloud” computing.
• the computer system 100 also may include one or more interfaces, such as an interface 110, coupled with the address/data bus 102.
• the one or more interfaces are configured to enable the computer system 100 to interface with other electronic devices and computer systems.
• the communication interfaces implemented by the one or more interfaces may include wireline (e.g., serial cables, modems, network adaptors, etc.) and/or wireless (e.g., wireless modems, wireless network adaptors, etc.) communication technology.
• the computer system 100 may include an input device 112 coupled with the address/data bus 102, wherein the input device 112 is configured to communicate information and command selections to the processor 104.
• the input device 112 is an alphanumeric input device, such as a keyboard, that may include alphanumeric and/or function keys.
• the input device 112 may be an input device other than an alphanumeric input device.
• the computer system 100 may include a cursor control device 114 coupled with the address/data bus 102, wherein the cursor control device 114 is configured to communicate user input information and/or command selections to the processor 104.
• the cursor control device 114 is implemented using a device such as a mouse, a track-ball, a track pad, an optical tracking device, or a touch screen.
• the cursor control device 114 is directed and/or activated via input from the input device 112, such as in response to the use of special keys and key sequence commands associated with the input device 112.
• the cursor control device 114 is configured to be directed or guided by voice commands.
• the computer system 100 further may include one or more optional computer usable data storage devices, such as a storage device 116, coupled with the address/data bus 102.
• the storage device 116 is configured to store information and/or computer executable instructions.
• the storage device 116 is a storage device such as a magnetic or optical disk drive (e.g., hard disk drive (“HDD”), floppy diskette, compact disk read only memory (“CD-ROM”), digital versatile disk (“DVD”)).
• a display device 118 is coupled with the address/data bus 102, wherein the display device 118 is configured to display video and/or graphics.
• the display device 118 may include a cathode ray tube (“CRT”), liquid crystal display (“LCD”), field emission display (“FED”), plasma display, or any other display device suitable for displaying video and/or graphic images and alphanumeric characters recognizable to a user.
• CTR cathode ray tube
• LCD liquid crystal display
• FED field emission display
• plasma display or any other display device suitable for displaying video and/or graphic images and alphanumeric characters recognizable to a user.
• the computer system 100 presented herein is an example computing environment in accordance with an aspect.
• the non-limiting example of the computer system 100 is not strictly limited to being a computer system.
• the computer system 100 represents a type of data processing analysis that may be used in accordance with various aspects described herein.
• other computing systems may also be implemented.
• the spirit and scope of the present technology is not limited to any single data processing environment.
• one or more operations of various aspects of the present technology are controlled or implemented using computer-executable instructions, such as program modules, being executed by a computer.
• program modules include routines, programs, objects, components and/or data structures that are configured to perform particular tasks or implement particular abstract data types.
• an aspect provides that one or more aspects of the present technology are implemented by utilizing one or more distributed computing environments, such as where tasks are performed by remote processing devices that are linked through a communications network, or such as where various program modules are located in both local and remote computer-storage media including memory- storage devices.
• FIG. 2 An illustrative diagram of a computer program product (i.e., storage device) embodying the present invention is depicted in FIG. 2.
• the computer program product is depicted as floppy disk 200 or an optical disk 202 such as a CD or DVD.
• the computer program product generally represents computer-readable instructions stored on any compatible non-transitory computer-readable medium.
• the term “instructions” as used with respect to this invention generally indicates a set of operations to be performed on a computer, and may represent pieces of a whole program or individual, separable, software modules.
• Non-limiting examples of “instruction” include computer program code (source or object code) and “hard-coded” electronics (i.e. computer operations coded into a computer chip).
• the “instruction” is stored on any non-transitory computer-readable medium, such as in the memory of a computer or on a floppy disk, a CD-ROM, and a flash drive. In either event, the instructions are encoded on a non-transitory computer-readable medium.
• the present disclosure provides a system and method for contrast and entropy-based perception adaption to optimize perception.
• the system operates by evaluating perception errors using various probes of detected objects, then correcting perception errors by solving a contrast/entropy-based optimization problem.
• the method uses the characteristics of geometry, dynamics, and detected blob image quality of the objects, the method develops the probabilistic signal temporal logic and builds axioms with the developed logic components. By evaluating these axioms, the system can verify if the detections or recognitions are valid or erroneous.
• the system is able to develop the probabilistic signal temporal logic based constraints and solve the contrast/entropy-based optimization problem to reduce false positives and acquire more correct detections; which ultimately allows the system to achieve more accurate object recognition.
• the system of the present disclosure provides a marked improvement over the prior art for several reasons, including: (1) perception error evaluation and detection using axioms generated from the perception-probe-induced probabilistic signal temporal logic; (2) perception error correction through image contrast/entropy adjustment by solving a contrast/entropy-based optimization problem under the axiom-generated constraints, and (3) detected object focused image contrast and entropy improvement for more robust object detection and recognition, rather than using the entire image’s contrast ranges and entropy ranges.
• the system also allows for estimating and correcting perception errors with formal verification. Formally verified perception error estimation and correction by solving the corresponding optimization problems themselves is not known in the prior art. With these unique features, it is difficult for other methods to achieve similar results without following the method of the present disclosure.
• the present disclosure is directed to a perception system and, more particularly, to a system for contrast and entropy-based perception adaption to optimize perception results.
• a flowchart depicting structural flow of the system is shown in FIG. 3.
• the system First, from the perception data, the system generates probes which describe characteristics of detections and recognitions, such as the size, type, tracking deviations, contrast, entropy and so on.
• probes which describe characteristics of detections and recognitions, such as the size, type, tracking deviations, contrast, entropy and so on.
• PSTL probabilistic signal temporal logic
• the PSTL provides axioms, each of which is constructed with a single or multiple probes with the corresponding statistical analyses. As an intermediate process, those axioms provide the error analysis on the detections/recognitions. Then, with those axiom-based constraints, an optimization problem is solved to synthesize image contrast controls for the perception modules in order to reduce perception errors and improve valid detection rates.
• the perception module 300 receives input images of a scene.
• the perception module 300 can be a hardware controller (such as a camera system or sensing device with or without associated software) or, in other aspects, the perception module 300 can be a preprocessing component that adjusts parameters prior to the sensing device.
• Object detection 302 is the performed, in which objects in the image are detected and recognized.
• the perception probe generation module 304 converts the perception data into probes that are used for signal temporal logic.
• PSTL-based axiom construction 306 is performed in which the probes are converted into the axioms under the probabilistic signal temporal logic structure.
• the axioms are then evaluated 308 to verify if the corresponding observations are valid or erroneous based on the constraints using the statistically analyzed probe bounds. If the axioms are invalid within certain probabilities (outside of probe bounds), an estimator module 310 estimates the optimal contrast bound and entropy bound as perception parameters to apply by solving the image contrast/entropy-based optimization problem 310. If the axioms are not invalid, then the results are delivered without any modification. Finally, the estimated parameters are delivered back to the perception module 300 to adjust its contrast and entropy. Such settings can be adjusted using any suitable technique or device.
• contrast can be easily adjusted using tools or adjustment settings on a perception device (e.g., camera) that are commonly known to those skilled in the art.
• Entropy can be adjusted or modified using, for example, a filter (see FIGs. 5A and 5B and associated text).
• the first step in the process is to obtain the perception data along with characteristics of detections and recognitions.
• the probes are the quantified version of the characteristics.
• any suitable state-of-the-art detection/recognition technique is used, a non-limiting example of which includes YOLO v3 (see Literature Reference No. 9).
• the following are several non-limiting examples of probes that can be used in accordance with the system of the present disclosure:
• FIG. 4 provides a graph depicting true positives and false positives for a probe.
• x assume that a probe, fix
• probabilistic distributions of true positives 400 and false positives 402 are obtained, as shown in FIG. 4.
• the upper 404 and the lower 406 bounds are set for the true positives 400, with the shaded area 408 representing the confidence probability,
• the probabilistic function (e.g., probability distributions 400 and 402 in FIG. 4) can also be multi-dimensional. Examples of multi-dimensional probes include contrast, entropy, aspect, ration, etc.
• a “multi-dimensional range” is obtained of the corresponding detection or recognition.
• a certain threshold e.g. the detected object size is out of the bound too frequently (as predefined or otherwise input), which means the detection bounding box is not correct
• the threshold can be obtained experimentally and be provided as an input parameter.
• U t where x t is the probe state (i.e., probe signal) at time t and u t is the control input to the perception module, and /( ) is the cost function of estimating perception errors.
• u t can be a single number or an array of numbers, depending on the particular probe.
• a goal is to achieve the optimal u t to reduce perception errors. Therefore, minimizing /( ) can achieve the optimal perception module control input.
• the output of min J() is the input control signal to adapt the perception system to improve the next results.
• the object detection constraints are first set up using five different types of constraints: (1) Detection ID consistency (tracking of the same object); (2) Localization consistency within the expected trajectory; (3) Bounding box size consistency in the image plane; (4) Contrast matching in the desired range; and (5) Entropy matching in a desired range. Details for each constraint are presented below, where t k is the current time and t k-M is the time that the temporal logic window starts.
• Localization deviation from the desired tracking trajectory is determined as follows: where loc t is the detected object’s location at time t and Path Desired is its expected path from the history. As can be appreciated, there are many techniques to computer an expected path, such as curve fitting, etc. Further, Pi oc is the probabilistic threshold for consistent localization.
• Bounding box size deviation over time is determined as follows: where BB t is the bounding box size (e.g., number of pixels in an image) at time t and BB D is the desired bounding box size from its history. P BB is the probabilistic threshold for consistent bounding box size.
• Contrast is determined as follows: where C t is the contrast (defined using the Michelson contrast, described above) of the bounding box at time t and C D is the desired contrast from the training phase. The training phase is the statistical analysis to determine all the constant values (thresholds). P c is the probabilistic threshold for contrast.
• Entropy is determined as follows: where E t is the image entropy (see Literature Reference No. 12) of the bounding box at time t and E D is the desired entropy from the training phase. P E is the probabilistic threshold for entropy.
• the desirable contrast deviation i.e., the desired contrast value c D
• the expansion of histogram ranges are set up (e.g., using commonly known histogram equalization techniques) to achieve the contrast changes using the peak-to-peak contrast (Michelson contrast) (see Literature Reference No. 10).
• the peak-to-peak contrast (as applied to the corresponding bounding box) is defined in the following way: where l max is the maximum image intensity value and I min is the minimum image intensity value. From this definition, one can expect a new contrast will be:
• C(k ) is the kth bounding box contrast. Therefore, the deviation is the desired one minus the current one.
• image kernels are applied depending on Ae. After optimization, the value is estimated. If Ae is positive, a sharpening filter is applied to increase entropy (as shown in FIG. 5A). On the other hand, if Ae is negative, a smoothing filter is applied to decrease entropy (as shown in FIG. 5B). Each filter’s derivative level is proportional to the amount of Ae , and the corresponding proportional relationship is obtained from multiple and uniformly distributed sample detections.
• the outputs using the process described herein are adjusted perception (sensor) module (e.g. cameras) parameters.
• sensor e.g. cameras
• different camera contrast values can be changed since the contrast in the camera system is changed through the system.
• the detection results from the image processing will also be changed.
• the system of the present disclosure shows how such parameters can be changed more appropriately.
• the optimized sensor parameters actually improve the object detection results.
• an adaptive sensor system can be implemented in a variety of application, such as for autonomous vehicles or unmanned aircraft systems.
• FIG. 6 shows a pair of the sample input image and the corresponding perception adapted image.
• the box in the upper image 600 is a wrong person detection from the original image using a perception system of the prior art, while the boxes in the lower image 602 are correct person detections (e.g., each box containing a single person) that were provided using the present method.
• FIG. 7A provides a precision-recall graph, while FIG.
• the adjusted approach 700 of the present invention outperforms that of the prior art 702, presenting better recall rates with the same precision, and also less false positive rates with achieving the same true positive rates. It is noted that in the ROC curve of FIG. 7B, with 10% detection confidence thresholding, the false positive rate of the present invention is reduced by 41.48% relative to the prior art. Thus, it is clear that the method and system of the present disclosure provides a marked improvement over perception systems of the prior art.

Landscapes

• Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Multimedia (AREA)
• Theoretical Computer Science (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Quality & Reliability (AREA)
• Image Analysis (AREA)

Applications Claiming Priority (3)

Publications (1)

Family

ID=77614383

Family Applications (1)

Country Status (3)

Family Cites Families (13)

• 2020
  • 2020-12-23 EP EP20842899.5A patent/EP4115319A1/de active Pending
  • 2020-12-23 WO PCT/US2020/066972 patent/WO2021178027A1/en active Search and Examination
  • 2020-12-23 CN CN202080094356.XA patent/CN115004256A/zh active Pending

Also Published As

Similar Documents

Legal Events