CN115004256A - Perceptual adjustment based on contrast and entropy using optimization based on probability signal temporal logic - Google Patents

Abstract

A system for performing perceptual adjustment based on contrast and entropy to optimize perception is described. The system is operable to receive an input image of a scene with a camera system and detect one or more targets (with perceptual data) in the input image. The perceptual parameters of the one or more targets are converted into probes, which are then converted into axioms using probabilistic signal-time logic. Axioms are evaluated based on probe boundaries. If the axiom is within the probe boundary, providing a result; however, if the axiom is outside the probe boundaries, the system estimates the optimal contrast boundary and entropy boundary as perceptual parameters. Then, contrast and entropy in the camera system are adjusted based on the perceptual parameters.

Description

Perceptual adjustment based on contrast and entropy using optimization based on probability signal temporal logic

Cross Reference to Related Applications

This application is a continuation-in-part application of U.S. serial No.17/030,354 filed on 23/9/2020, No.17/030,354 is a non-provisional patent application of U.S. provisional application serial No.62/905,059 filed on 24/9/2019 and U.S. provisional application serial No.62/984,713 filed on 3/2020, which are incorporated herein by reference in their entireties.

This application also claims the benefit of U.S. provisional application No.62/984,713 filed 3/2020 and is incorporated herein by reference in its entirety as a non-provisional patent application.

Background

(1) Field of the invention

The present invention relates to a perceptual system, and more particularly, to a system for evaluating and correcting perceptual errors by performing a contrast and entropy-based perceptual adaptation (perceptual adaptation) using a probabilistic signal temporal logic (probabilistic signal temporal-based) optimization to optimize a perceptual result.

(2) Description of the related Art

Perceptual systems are commonly used for target recognition and tracking, but often suffer from perceptual errors. Many researchers have attempted to solve this problem; however, despite the improved performance of the sensing system in the last decade, sensing errors remain a challenging problem. In autonomous driving or navigation systems, a large number of false detections and identifications threaten the safety and robust performance of a fully autonomous system. To describe and recover from perceptual errors, there are many directions to study, especially formally validating the system using temporal logic (see references No.1 to No.4 in the incorporated list of references).

Most existing systems plan to control the autonomous system itself rather than the repair awareness system. One prior art technique uses feedback in the system and utilizes image contrast enhancement to provide better saliency to targets in the scene (see reference No. 5). Therefore, it helps to detect the target in a more robust manner (see reference nos. 6 and 7). However, corresponding conventional methods use image contrast information for the entire image. Therefore, if there are some non-target areas that cause high contrast, the contrast adjustment cannot improve the target detection.

Accordingly, there is a continuing need for improved perception systems that use feedback control of the contrast parameters of detection targets and the ability to adjust entropy within a formally validated system to achieve more appropriate significance.

Disclosure of Invention

The present disclosure provides a system for making perceptual adjustments based on contrast and entropy to optimize perception. In one aspect, the system includes a memory and one or more processors. The memory is a non-transitory computer-readable medium having executable instructions encoded thereon such that, when executed, the one or more processors perform operations. The system is operable to receive an input image of a scene with a perception module (i.e., a camera system) and detect one or more targets (with perception data) in the input image. The perceptual parameters of the one or more targets are converted into probes (probes) which are then converted into axioms using probabilistic signal-time logic. Axioms are evaluated based on probe boundaries. If the axiom is within the probe boundaries, the results are provided without modification; however, if the axiom is outside the probe boundaries, the system estimates the optimal contrast boundary and entropy boundary as perceptual parameters (i.e., camera system parameters). Then, contrast and entropy in the camera system are adjusted based on the perceptual parameters.

In yet another aspect, in adjusting entropy, an image kernel is applied such that if the change in entropy is positive, a sharpening filter is applied to increase the entropy, and if the change in entropy is negative, a smoothing filter is applied to decrease the entropy.

Further, adjusting the contrast includes acquiring a desired contrast bias such that once the desired contrast bias is acquired, a histogram range is set to achieve a contrast variation using peak-to-peak contrast.

In another aspect, the camera system is incorporated into an adjustment sensor system of an autonomous vehicle or an unmanned aerial vehicle system.

Finally, the present invention also includes a computer program product and a computer-implemented method. The computer program product includes computer-readable instructions stored on a non-transitory computer-readable medium that are executable by a computer having one or more processors such that, when the instructions are executed, the one or more processors perform the operations listed herein. Alternatively, a computer-implemented method includes acts of causing a computer to execute the instructions and perform the resulting operations.

Drawings

The objects, features and advantages of the present invention will become apparent from the following detailed description of various aspects of the invention, when taken in conjunction with the following drawings, in which:

FIG. 1 is a block diagram depicting components of a system according to various embodiments of the invention;

FIG. 2 is an illustrative diagram of a computer program product that implements aspects of the invention;

FIG. 3 is a flow diagram depicting an overview of a system according to various embodiments of the invention;

FIG. 4 is a graph depicting a probability distribution of probes according to various embodiments of the present invention;

FIG. 5A is a graph depicting the use of a sharpening filter to adjust entropy;

FIG. 5B is a graph depicting the use of a smoothing filter to adjust entropy;

FIG. 6 is an illustration of a sample depicting error reduction and detection improvement;

FIG. 7A is a precision-recall curve for different detection thresholds;

fig. 7B is a Receiver Operating Characteristic (ROC) curve for different detection thresholds.

Detailed Description

The present invention relates to perceptual systems, and more particularly to a system for evaluating and correcting perceptual errors to optimize perceptual results by performing contrast and entropy based perceptual adjustment using probabilistic signal-based temporal logic optimization. The following description is presented to enable any person skilled in the art to make and use the invention and is incorporated in the context of a particular application. Various modifications and applications of the various aspects will be apparent to those skilled in the art, and the generic principles defined herein may be applied to a wide range of aspects. Thus, the present invention is not intended to be limited to the aspects shown, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

In the following detailed description, numerous specific details are set forth in order to provide a more thorough understanding of the invention. It will be apparent, however, to one skilled in the art that the present invention may be practiced without necessarily being limited to these specific details. In other instances, well-known structures and devices are shown in block diagram form, rather than in detail, in order to avoid obscuring the present invention.

The reader is directed to all papers and documents which are filed concurrently with this specification and which are open to public inspection with this specification, and the contents of all such papers and documents are incorporated herein by reference. All the features disclosed in this specification (including any accompanying claims, abstract and drawings), may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise, and each feature disclosed is one example only of a generic series of equivalent or similar features.

Furthermore, any element in the claims that does not explicitly recite "means" or "step" to perform a specified function should not be construed as an "means" or "step" clause as set forth in section 112(f) of 35u.s.c. In particular, the use of "step … …" or "action of … …" in the claims herein is not intended to trigger the clause of section 112(f) of 35u.s.c.

Before describing the present invention in detail, a list of references is first provided. Next, a description is provided of various main aspects of the present invention. The following description provides the reader with a general understanding of the invention. Finally, specific details of various embodiments of the invention are provided to give an understanding of the specific aspects.

(1) List of incorporated references

The following references are cited throughout this application. For clarity and convenience, these references are listed herein as a centralized resource for the reader. The following references are incorporated herein by reference as if fully set forth herein. These references are incorporated by reference into this application by reference to the corresponding reference numbers as follows:

1.A.Dokhanchi,H.B.Amor,J.V.Deshmukh,and G.Fainekos,“Evaluating perception systems for autonomous vehicles using quality temporal logic,”International Conference on Runtime Verification,2018.

2.R.R.da Silva,V.Kurtz,and M.Hebert,“Active Perception and Control from Temporal Logic Specifications,”arXiv:1905.03662,2019.

3.S.Jha,V.Raman,D.Sadigh,and S.A.Seshia,“Safe Autonomy Under Perception Uncertainty Using Chance-Constrained Temporal Logic,”Journal of Automated Reasoning,2018.

4.D.Sadigh and A.Kapoor,“Safe control under uncertainty with Probabilistic Signal Temporal Logic,”in Proc.Of Robotics:Science and Systems,2016.

5.J.A.Stark,“Adaptive Image Contrast Enhancement Using Generalizations of Histogram Equalization,”IEEE Transactions on Image Processing,Vol.9,No.5,pp.889-896,2000.

6.U.S.Patent No.9,626,591,entitled,“Enhanced Contrast for Object Detection and Characterization by Optical Imaging,”by D.Holz and H.Yang.

7.V.Vonikakis,D.Chrysostomou,R.Kouskouridas and A.Gasteratos,“Improving the Robustness in Feature Detection by Local Contrast Enhancement,”2012 IEEE International Conference on Image Systems and Techniques Proceedings,July 2012.

8.U.S.Application No.17/030,354,filed on September 23,2020,entitled,“System and Method of Perception Error Evaluation and Correction by Solving Optimization Problems under the Probabilistic Signal Temporal Logic Based Constraints”.

9.YOLO Real Time Object Detection,located at pjreddie.com/darknet/yolo/,taken on August 12,2020.

10.Luminance Contrast,found at colorusage.arc.nasa.gov/luminance_cont.php,taken on December 02,2020.

11.Multiple Object Tracking Benchmark,located at motchallenge.net,taken on August 12,2020.

12.Information Entropy Measure for Evaluation of Image Quality,Du-Yih Tsai,Yongbum Lee,Eri Matsuyama,J Digit Imaging.2008Sep；21(3):338–347.Published online 2007Jun 19.doi:10.1007/s10278-007-9044-5.

(2) main aspects of the invention

Various embodiments of the present invention include three "primary" aspects. The first main aspect is a system that performs contrast and entropy based perceptual adjustment to optimize perception. The system typically takes the form of the computer system operating software or the form of a "hard-coded" instruction set. The system may be incorporated into a wide variety of devices that provide different functions. The second main aspect is a method, usually in the form of software, operating with a data processing system (computer). A third main aspect is a computer program product. The computer program product generally represents computer-readable instructions stored on a non-transitory computer-readable medium such as an optical storage device (e.g., a Compact Disc (CD) or a Digital Versatile Disc (DVD)) or a magnetic storage device (e.g., a floppy disk or a magnetic tape). Other non-limiting examples of computer readable media include: hard disks, Read Only Memories (ROMs), and flash memory type memories. These aspects will be described in more detail below.

A block diagram depicting an example of the system of the present invention, namely computer system 100, is provided in fig. 1. The computer system 100 is configured to perform calculations, processes, operations, and/or functions associated with a program or algorithm. In one aspect, certain processes and steps discussed herein are implemented as a series of instructions (e.g., a software program) residing in a computer readable memory unit and executed by one or more processors of the computer system 100. When executed, the instructions cause the computer system 100 to perform particular actions and exhibit particular behaviors, as described herein. In various aspects, the computer system 100 may be implemented in any device that operates to perform the functions described herein as being suitable for a particular application, such as a desktop computer, a mobile phone or smartphone, a tablet computer, a computer implemented in a mobile platform, or any other device or devices that individually and/or collectively may execute instructions to perform the relevant operations/processes.

Computer system 100 may include an address/data bus 102 configured to communicate information. In addition, one or more data processing units, such as a processor 104 (or multiple processors), are coupled to the address/data bus 102. The processor 104 is configured to process information and instructions. In an aspect, the processor 104 is a microprocessor. Alternatively, the processor 104 may be a different type of processor, such as a parallel processor, an Application Specific Integrated Circuit (ASIC), a Programmable Logic Array (PLA), a Complex Programmable Logic Device (CPLD), or a Field Programmable Gate Array (FPGA), or any other processing component operable to perform the relevant operations.

Computer system 100 is configured to utilize one or more data storage units. The computer system 100 may include a volatile memory unit 106 (e.g., random access memory ("RAM"), static RAM, dynamic RAM, etc.) coupled to the address/data bus 102, wherein the volatile memory unit 106 is configured to store information and instructions for the processor 104. The computer system 100 may also include a non-volatile memory unit 108 (e.g., read only memory ("ROM"), programmable ROM ("PROM"), erasable programmable ROM ("EPROM"), electrically erasable programmable ROM ("EEPROM"), flash memory, etc.) coupled to the address/data bus 102, wherein the non-volatile memory unit 108 is configured to store static information and instructions for the processor 104. Alternatively, the computer system 100 may execute instructions retrieved from an online data storage unit, such as in "cloud" computing. In an aspect, computer system 100 may also include one or more interfaces, such as interface 110, coupled with address/data bus 102. The one or more interfaces are configured to enable computer system 100 to connect with other electronic devices and computer systems. The communication interfaces implemented by the one or more interfaces may include wired (e.g., serial cable, modem, network adapter, etc.) and/or wireless (e.g., wireless modem, wireless network adapter, etc.) communication technologies.

In one aspect, the computer system 100 may include an input device 112 coupled to the address/data bus 102, wherein the input device 112 is configured to communicate information and command selections to the processor 104. According to one aspect, the input device 112 is an alphanumeric input device (such as a keyboard) that may include alphanumeric and/or function keys. Alternatively, input device 112 may be other than an alphanumeric input device. In one aspect, the computer system 100 may include a cursor control device 114 coupled to the address/data bus 102, wherein the cursor control device 114 is configured to communicate user input information and/or command selections to the processor 104. In one aspect, cursor control device 114 is implemented with a device such as a mouse, trackball, trackpad, optical tracking device, or touch screen. Notwithstanding the foregoing, in one aspect, cursor control device 114 is directed and/or enabled via input from input device 112, such as in response to using special keys and key sequence commands associated with input device 112. In an alternative aspect, cursor control device 114 is configured to be managed or directed by voice commands.

In an aspect, the computer system 100 may also include one or more optional computer usable data storage devices, such as storage device 116 coupled to the address/data bus 102. Storage device 116 is configured to store information and/or computer-executable instructions. In one aspect, storage device 116 is a storage device such as a magnetic or optical disk drive (e.g., hard disk drive ("HDD"), floppy disk, compact disk read-only memory ("CD-ROM"), digital versatile disk ("DVD")). In accordance with one aspect, a display device 118 is coupled with the address/data bus 102, wherein the display device 118 is configured to display video and/or graphics. In an aspect, the display device 118 may include: a cathode ray tube ("CRT"), a liquid crystal display ("LCD"), a field emission display ("FED"), a plasma display, or any other display device suitable for displaying video and/or graphic images, as well as alphanumeric characters recognizable to a user.

The computer system 100 presented herein is an example computing environment in accordance with an aspect. However, a non-limiting example of computer system 100 is not strictly limited to being a computer system. For example, one aspect provides that computer system 100 represents a type of data processing analysis that may be used in accordance with various aspects described herein. Other computing systems may also be implemented. Indeed, the spirit and scope of the present technology is not limited to any single data processing environment. Thus, in one aspect, one or more operations of various aspects of the present technology are controlled or implemented using computer-executable instructions, such as program modules, executed by a computer. In one implementation, such program modules include routines, programs, objects, components, and/or data structures that are configured to perform particular tasks or implement particular abstract data types. In addition, one aspect provides for implementing one or more aspects of the technology by utilizing one or more distributed computing environments, such as where tasks are performed by remote processing devices that are linked through a communications network, or such as where various program modules are located in both local and remote computer storage media including memory-storage devices.

A diagram of a computer program product (i.e., a storage device) embodying the present invention is depicted in fig. 2. The computer program product is depicted as a floppy disk 200 or an optical disk 202 such as a CD or DVD. However, as previously mentioned, the computer program product generally represents computer readable instructions stored on any compatible non-transitory computer readable medium. The term "instructions," as used with respect to the present invention, generally indicates a set of operations to be performed on a computer, and may represent a fragment of an entire program or a single, separate software module. Non-limiting examples of "instructions" include computer program code (source or object code) and "hard-coded" electronic devices (i.e., computer operations encoded into a computer chip). "instructions" are stored on any non-transitory computer readable medium, such as on a floppy disk, CD-ROM, and flash drive or in the memory of a computer. Regardless, the instructions are encoded on a non-transitory computer readable medium.

(3) Introduction to

The present disclosure provides a system and method for perceptual adjustment based on contrast and entropy to optimize perception. The system operates by using various probes that detect the target to assess perception errors, and then by solving a contrast/entropy based optimization problem to correct the perception errors. The method exploits probabilistic signal-time logic with characteristics of geometry, dynamics, and detected blob (blob) image quality of the target, and constructs axioms with the exploited logic components. By evaluating these axioms, the system can verify whether the detection result or the identification result is valid or erroneous. In addition, by utilizing the developed axiom, the system can develop the constraint based on probability signal time logic and solve the optimization problem based on contrast/entropy so as to reduce false positives and obtain more correct detection results; this ultimately allows the system to achieve more accurate target identification.

The system of the present disclosure provides significant improvements over the prior art for several reasons, including: (1) performing perceptual error evaluation and detection using axioms generated from perceptual-probe-firing probability signal temporal logic; (2) performing perceptual error correction through image contrast/entropy adjustment by solving an optimization problem based on contrast/entropy under axiomatic generated constraints; and (3) detecting image contrast and entropy improvement of target focus (focused) for more robust target detection and recognition, rather than using contrast and entropy ranges for the entire image. The system also allows for the use of formal verification to estimate and correct perceptual errors. Perceptual error estimation and correction, formally verified by solving a corresponding optimization problem, is not known per se in the prior art. With these unique features, it is difficult for other methods to achieve similar results without following the methods of the present disclosure.

(4) Details of various embodiments

As mentioned above, the present disclosure is directed to a perceptual system, and more particularly, to a system for perceptual adjustment based on contrast and entropy to optimize perceptual results. Fig. 3 shows a flow chart depicting the flow of the system architecture. First, the system generates probes from the perception data that describe characteristics of the detection results and recognition results, such as size, type, tracking bias, contrast, entropy, and the like. Using these probes, probabilistic signal-time logic (PSTL) was established (see reference No.4 for a discussion of PSTL). PSTL provides axioms, wherein each axiom is constructed from a single or multiple probes with corresponding statistical analysis. As an intermediate process, these axioms provide for error analysis of the detection/recognition results. Then, using these axiom-based constraints, an optimization problem is solved to synthesize image contrast control for the perception module in order to reduce perception errors and improve effective detection rates.

Referring to fig. 3, the sensing module 300 receives an input image of a scene. In one aspect, the perception module 300 may be a hardware controller (such as a camera system or sensing device with or without associated software), or in other aspects, the perception module 300 may be a preprocessing component that adjusts parameters prior to sensing the device. Object detection 302 is performed in which objects in the image are detected and identified. The sensing probe generation module 304 converts the sensing data into probes that are used for signal temporal logic. A PSTL-based axiom construction 306 is performed in which probes are converted to axioms under a probabilistic signal-time logic structure. The axioms are then evaluated 308 to verify whether the corresponding observations are valid or erroneous based on the constraints of the probe boundaries using statistical analysis. If the axiom is invalid within some probability (outside the probe boundaries), the estimator module 310 estimates the optimal contrast boundaries and entropy boundaries as perceptual parameters to apply by solving an image contrast/entropy based optimization problem 310. If the axiom is not invalid, the result is delivered without any modification. Finally, the estimated parameters are passed back to the perception module 300 to adjust its contrast and entropy. Such settings may be adjusted using any suitable technique or device. For example, the contrast can be easily adjusted using tools or adjustment settings on the perception device (e.g., camera) known to those skilled in the art. The entropy can be adjusted or modified using, for example, a filter (see fig. 5A and 5B and related text). Various ones of these aspects are described in further detail below.

The first step in the process is to obtain perceptual data along with the characteristics of the detection results and the recognition results. For clarity, the probes are quantized versions of the features. To efficiently obtain different types of characteristics, any suitable state-of-the-art detection/identification technique is used, non-limiting examples of which include YOLO v3 (see reference No. 9). The following are several non-limiting examples of probes that may be used in accordance with the system of the present disclosure:

1. target size (in image plane and in world coordinate frame of reference)

2. Aspect ratio of detection target

3. Positioning and tracking performance

4. Recognition confidence

5. Contrast of detection frame

6. Entropy of detection Frames

Therefore, in the present system, there may be a plurality of probes such as the size of the detection target, the aspect ratio, the identification ID consistency, the tracking offset, and the like.

Then, based on the true positive detection or identification, a statistical analysis is performed on each probe (i.e., performed in block 308 of FIG. 3). For example, fig. 4 provides a graph depicting true positives (true positves) and false positives (false positves) for a probe. For detection result/recognition result x, it is assumed that probe f (x) is generated. By analyzing the values from true positives as well as the values from false positives, a probability distribution of true positives 400 and a probability distribution of false positives 402 are obtained, as shown in fig. 4. An upper bound 404 and a lower bound 406 of true positives 400 are set according to the intersection between two different distribution plots, where the shaded region 408 represents the confidence probability P of the probe _TP . If the relationship is described mathematically (axiomatically) using probabilistic signal time logic, it becomes as follows:

wherein Pr () is the probability, Pr ≧ P _TP Is predicate (predicate), y is true detection result or identification result, and t _s :t _e Is denoted by t _s And t _e Time series between, thus f (x, t) _s :t _e ) Is a time frame t _s :t _e And a and b are boundaries. True and false positives are provided along with the date of training.

Depending on the probe dimensions, the probability functions (e.g., probability distributions 400 and 402 in FIG. 4) may also be multidimensional. Examples of multi-dimensional probes include contrast, entropy, aspect, ratio, and the like. By integrating all the available axioms from x, a "multidimensional range" of corresponding detection results or recognition results is obtained. When the probabilistic signal-time logic is violated (e.g., becomes false) and exceeds a certain threshold (e.g., the detection target size exceeds the boundary too frequently (as predefined or otherwise entered), which means that the detection bounding box is incorrect), it is verified that the corresponding perceptual processing is considered erroneous processing. In one aspect, the threshold value may be obtained experimentally and provided as an input parameter.

Detecting perceptual errors is not sufficient to restore perceptual quality in subsequent image sequences (detection does not affect any future processing). Therefore, it is also desirable to adjust the sensing module to provide more accurate and robust detection and identification with this knowledge. To this end, the present system uses a new optimization technique that uses a PSTL constraint-based optimization having the following format:

wherein x is _t Is the probe state (i.e., probe signal) at time t, u _t Is the control input to the perception module and J (-) is the cost function to estimate the perception error. Depending on the particular probe, u _t Either as a single number or an array of numbers. The goal is to achieve an optimum u _t To reduce perceptual errors. Therefore, minimizing J (-) can achieve optimal perceptual module control inputs. The output of min J () is the input control signal to let the sensing system adjust to improve the next result. Finally, there are two or more PSTL-based constraints f (x) for the probe _t )，g(z _t ) Etc. becomes the final optimization formula of,

to achieve contrast/entropy based perceptual adjustment, five different types of constraints are first used to establish target detection constraints: (1) detecting ID consistency (tracking the same target); (2) positioning consistency within the expected trajectory; (3) bounding box size consistency in the image plane; (4) contrast matching within a desired range; and (5) entropy matching within the desired range. Details of the various constraints are presented below, where t _k Is the current time, and t _k-M Is the time at which the temporal logical window begins.

The coincidence detection is determined as follows:

wherein, P _X Is a probability threshold for consistent ID detection. If the probability of detecting ID X is low (which means that the IDs are not consistent), the trace result is ignored due to lack of confidence consistency.

The positioning deviation from the desired tracking trajectory is determined as follows:

wherein, loc _t Is the position of the detection target at time t, Path _Desired Is its expected path from history. As can be appreciated, there are many techniques to calculate the expected path, such as curve fitting, and the like. Furthermore, P _loc Is a probability threshold for consistent positioning.

The deviation of the bounding box size over time is determined as follows:

wherein, BB _t Is the bounding box size (e.g., number of pixels in the image), BB, at time t _D Is the expected bounding box size according to its history. P _BB Is a probability threshold for a consistent bounding box size.

The contrast was determined as follows:

wherein, C _t Is the contrast of the bounding box at time t (defined using the above-mentioned Michelson contrast), c _D Is the desired contrast from the training phase. The training phase is a statistical analysis that determines all constant values (thresholds). P _C Is toA probability threshold for the ratio.

The entropy is determined as follows:

wherein E is _t Is the image entropy of the bounding box at time t (see reference No.12), E _D Is the desired entropy from the training phase. P is _E Is a probability threshold of entropy.

Controlling contrast (c) using cost function J (e, c, Δ e, Δ c) _i (t) + Δ c) and entropy (e) _i The corresponding optimization formula for (t) + deltae) is,

wherein, c _i (t) is the contrast value, e _i (t) is the entropy value of the ith detection target at time t (e.g., where C _t And c _i (t) is the same in the desired aspect). c. C _D Is the expected contrast value from the probability distribution process of finding a probe and e _D Is the desired entropy value (e.g., respectively c) _D And E _D The same). Δ c and Δ e are the system control inputs for contrast and entropy, respectively (which are the same as the estimated bias applied to the perception module (i.e., the camera system)).

For contrast control, once the desired contrast deviation (i.e., the desired contrast value c) is obtained _D ) An extension of the histogram range is established (e.g., using well-known histogram equalization techniques) to achieve contrast variation using peak-to-peak contrast (Michelson contrast) (see reference No. 10). The peak-to-peak contrast (when applied to the corresponding bounding box) is defined as follows:

wherein, I _max Is at maximumImage intensity value, I _min Is the minimum image intensity value. From this definition, it is expected that the new contrast will be:

where B is the extended histogram range (in both directions) that achieves the new contrast. Since Δ C ═ C _desired -c (k), so the histogram change range variance will be:

note that c (k) is the kth bounding box contrast. Thus, the offset is the desired one minus the current one.

For entropy, an "image kernel" is applied according to Δ e. After optimization, the value is estimated. If Δ e is positive, a sharpening filter is applied to increase entropy (as shown in FIG. 5A). On the other hand, if Δ e is negative, a smoothing filter is applied to reduce entropy (as shown in fig. 5B). The derivative level (derivative level) of each filter is proportional to the amount of Δ e, and a corresponding proportional relationship is obtained from a plurality of and uniformly distributed sample detection results.

The output using the processing described herein is the adjusted sensing (sensor) module (e.g., camera) parameters. For example, since the contrast in the camera system is changed by the system, different camera contrast values can be changed. The detection result of the image processing will also change. The system of the present disclosure shows how these parameters can be changed more appropriately. The optimized sensor parameters actually improve the target detection results. Those skilled in the art will appreciate that such adjustment sensor systems may be implemented in a variety of applications, such as for autonomous vehicles or unmanned aerial vehicle systems.

To demonstrate the efficacy of the present system, simple test results for one of the multi-target tracking benchmark datasets (as described in reference No. 11) are provided. As shown in fig. 6, the contrast-based optimization method of the present disclosure improves the detection result. Fig. 6 shows a pair of sample input images and corresponding perceptually adjusted images. The boxes in the upper image 600 are false person detections from the original image using prior art perception systems, while the boxes in the lower image 602 are correct person detections provided using the present method (e.g., each box contains a single person). To further support the quantified performance, fig. 7A provides a precision-recall graph, while fig. 7B provides a Receiver Operating Characteristic (ROC) graph. As the graph shows, the adjusted method 700 of the present invention outperforms the prior art method 702, exhibiting better recall at the same accuracy, and less false positive rate where the same true positive rate is achieved. Note that in the ROC curve of fig. 7B, at a detection confidence threshold of 10%, the false positive rate of the present invention is reduced by 41.48% relative to the prior art. It is therefore apparent that the method and system of the present disclosure provide a significant improvement over prior art perception systems.

Finally, while the invention has been described in terms of several embodiments, those of ordinary skill in the art will readily recognize that the invention can have other applications in other environments. It should be noted that many embodiments and implementations are possible. Furthermore, the following claims are in no way intended to limit the scope of the present invention to the specific embodiments described above. Additionally, any recitation of "means for … …" is intended to induce an element and a means-plus-function interpretation of the claims, and any element not specifically recited using "means for … …" should not be interpreted as a means-plus-function element, even if the claims otherwise include the word "means. Further, although specific method steps have been recited in a particular order, the method steps may be performed in any desired order and are within the scope of the invention.

Claims (12)

1. A system for performing perceptual adjustment based on contrast and entropy to optimize perception, the system comprising:

a memory and one or more processors, the memory being a non-transitory computer-readable medium having executable instructions encoded thereon such that, when executed, the one or more processors perform the following:

receiving an input image of a scene with a camera system;

detecting one or more targets in the input image, the one or more targets having perceptual data;

converting the perception data of the one or more targets into probes;

converting the probes into axioms using probabilistic signal temporal logic;

evaluating the axiom based on a probe boundary such that if the axiom is outside the probe boundary, an optimal contrast boundary and an entropy boundary are estimated as perceptual parameters; and

adjusting contrast and entropy in the camera system based on the perceptual parameters.

2. The system of claim 1, wherein in adjusting entropy, an image kernel is applied such that if the change in entropy is positive, a sharpening filter is applied to increase entropy and if the change in entropy is negative, a smoothing filter is applied to decrease entropy.

3. The system of claim 1, wherein adjusting the contrast comprises acquiring a desired contrast bias such that once the desired contrast bias is acquired, a histogram range is set to achieve a contrast change using peak-to-peak contrast.

4. The system of claim 1, wherein the camera system is incorporated into an adjustment sensor system of an autonomous vehicle or an unmanned aerial vehicle system.

5. A computer program product for perceptual adjustment based on contrast and entropy to optimize perception, the computer program product comprising:

a non-transitory computer-readable medium having executable instructions encoded thereon such that, when executed by one or more processors, the one or more processors perform operations comprising:

receiving an input image of a scene with a camera system;

detecting one or more targets in the input image, the one or more targets having perceptual data;

converting the perception data of the one or more targets into probes;

converting the probes into axioms using probabilistic signal time logic;

evaluating the axiom based on a probe boundary such that if the axiom is outside the probe boundary, an optimal contrast boundary and an entropy boundary are estimated as perceptual parameters; and

adjusting contrast and entropy in the camera system based on the perceptual parameters.

6. The computer program product of claim 5, wherein in adjusting entropy, an image kernel is applied such that if the change in entropy is positive, a sharpening filter is applied to increase entropy and if the change in entropy is negative, a smoothing filter is applied to decrease entropy.

7. The computer program product of claim 5, wherein adjusting the contrast comprises obtaining a desired contrast bias such that once the desired contrast bias is obtained, a histogram range is set to achieve a contrast change using peak-to-peak contrast.

8. The computer program product of claim 5, wherein the camera system is incorporated into an adjustment sensor system of an autonomous vehicle or an unmanned aerial vehicle system.

9. A computer-implemented method of making perceptual adjustments based on contrast and entropy to optimize perception, the method comprising the acts of:

causing one or more processors to execute instructions encoded on a non-transitory computer-readable medium such that, when executed, the one or more processors perform the following:

receiving an input image of a scene with a camera system;

detecting one or more targets in the input image, the one or more targets having perceptual data;

converting the perception data of the one or more targets into probes;

converting the probes into axioms using probabilistic signal time logic;

evaluating the axiom based on a probe boundary such that if the axiom is outside the probe boundary, an optimal contrast boundary and an entropy boundary are estimated as perceptual parameters; and

adjusting contrast and entropy in the camera system based on the perceptual parameters.

10. The method of claim 9, wherein in adjusting entropy, an image kernel is applied such that if the change in entropy is positive, a sharpening filter is applied to increase entropy and if the change in entropy is negative, a smoothing filter is applied to decrease entropy.

11. The method of claim 9, wherein adjusting the contrast comprises acquiring a desired contrast bias such that once the desired contrast bias is acquired, a histogram range is set to achieve a contrast change using peak-to-peak contrast.

12. The method of claim 9, wherein the camera system is incorporated into an adjustment sensor system of an autonomous vehicle or an unmanned aerial vehicle system.

Images