JP2021532357A - Systems and methods for analyzing and displaying acoustic data - Google Patents

Abstract

One system includes an acoustic sensor array configured to receive acoustic signals, an electromagnetic imaging tool configured to receive electromagnetic radiation, a user interface, a display, and a processor. The processor receives electromagnetic data from the electromagnetic imaging tool and acoustic data from the acoustic sensor array. The processor generates acoustic image data of the scene based on the received acoustic data, generates a display image consisting of the combined acoustic image data and electromagnetic image data, and provides the display image to the display. The processor can receive annotation input from the user interface and update the display image based on the received annotation input. The processor may be configured to determine one or more acoustic parameters associated with the received acoustic signal and to determine the criticality associated with the acoustic signal. The user can annotate the display image with defined critical information or other defined information.

Description

Cross-reference of related applications

This application claims the priority of US Patent Application No. 62 / 702,716 filed July 24, 2018, the entire contents of which are incorporated herein by reference.

Currently available acoustic imaging devices include acoustic sensor array configurations with various frequency sensitivity limits due to various factors. For example, some acoustic imaging devices are designed to respond in the acoustic frequency range between approximately 20 Hz and approximately 20 kHz. Other devices (eg, ultrasonic devices) are designed to respond in the acoustic frequency range between approximately 38 kHz and approximately 45 kHz.

However, an acoustic image pickup device generally designed to operate in the frequency range of 20 Hz to 20 kHz cannot effectively detect or image high frequencies up to, for example, approximately 50 kHz or higher. Similarly, an acoustic or ultrasonic device designed to operate in the frequency range of 20 kHz to 50 kHz cannot effectively detect and / or image low frequencies of, for example, 20 kHz or less. This is for various reasons. For example, a sensor array optimized for low frequencies (eg, audible frequencies) generally refers to individual sensors that are far more distant than a sensor array optimized for high frequencies (eg, ultrasonic frequencies) does. Includes.

In addition to or apart from hardware concerns, different computational algorithms and methods of acoustic imaging are often more suitable for acoustic signals with different frequencies and / or different distances to the target, especially inexperienced. It is difficult for the user to determine the best way to image the scene acoustically without them.

Such discrepancies when imaging different acoustic frequency ranges are due in part to the physical phenomenon behind the propagation of sound waves of different frequencies and wavelengths in the air. A plurality of array orientations, array sizes and calculation methods are generally more suitable for acoustic signals with different frequency characteristics (eg, audible frequency, ultrasonic frequency, etc.).

Similarly, different array characteristics and / or computational methods may be more suitable for acoustic scenes at different distances to the target. For example, near-field acoustic holography aimed at a target at a very short distance and various acoustic beam forming methods aimed at a target at a farther distance.

Therefore, acoustic inspection using an acoustic array (eg, for acoustic imaging) is, for example, extensive equipment for the analysis of acoustic signals with different frequency ranges, and when performing acoustic analysis with different hardware and computation. You may need specialized skills to understand if the technology is appropriate. This puts time and cost on acoustic inspections and may require specialists to perform such inspections.

For example, the user may be forced to manually select various hardware and / or software for acoustic analysis. However, an inexperienced analyst may not be able to know the preferred combination of hardware and software for a given analysis and / or acoustic scene. Moreover, separating the target sound from within the scene poses a challenge in itself, especially in cumbersome scenes, and may present a boring annoyance to inexperienced users. For example, a given acoustic scene may contain an acoustic signal containing a number of frequencies, intensities, and other characteristics that obscure the target acoustic signal, especially in a noisy environment.

Traditional systems often require the user to manually identify the various acoustic parameters of the target prior to inspection to analyze the target sound. However, inexperienced users may not know how to best separate and / or identify the various sounds of the target.

Moreover, multiple imaging techniques (eg, visible light imaging technology, infrared imaging technology, ultraviolet imaging technology, audible imaging technology or other imaging technology) are used in concert when inspecting the same object or scene. If so, the physical placement of tools and / or other setting (eg, focus position) used to perform different imaging techniques can have a strong impact on the analysis. For example, different positions and / or focus positions of each image pickup device may result in parallax and the resulting images may be misaligned and juxtaposed. This can result in the inability to properly locate target areas and / or problem areas, documentation errors, and misdiagnosis of problems in the scene. For example, with respect to acoustic image data, if the acoustic image data deviates from other imaging techniques (eg, visible light image data and / or infrared image data) with respect to the image data, the position or sound source of the target acoustic signal is specified. It can be difficult to do.

Existing ultrasonic testing and inspection tools use ultrasonic sensors with or without parabolic dishes to help the receiving sensor converge the sound. When a sound of a particular frequency is detected, it is generally displayed on the display of the device as a rising or falling number, or on a frequency or decibel level graph. This can be very confusing and non-intuitive to many users. Visualization of live scene images or sounds is not available.

Separating, locating, and analyzing a particular sound is a tedious process that can be confusing to many end users. The complex and non-intuitive interface between the device and humans can be an obstacle to the effective use of the device and may require additional training even to operate basic functions on the device. ..

A good acoustic imager has the potential to produce a false color visualization of sound integrated with a still or live visible image of the scene. Selection and adjustment control, even in those devices, is important for proper sound visualization. However, traditional controls have been developed for use by highly trained audio technicians and professionals. These controls are often non-intuitive to the average user and can result in some confusion in proper selection and visualization parameter control. The use of these controls by low-level trained personnel can be cumbersome, leading to misselection of parameters and ultimately poor acoustic visualization.

Moreover, additional contextual information is often required along with this method for proper analytical reporting activities. A technician who wants to collect additional contextual information about a scene that is typically inspected by a conventional ultrasound test device or acoustic imager may take a picture with another camera or device and / or note or describe it. Must be recorded with annotations recorded on another device such as a PC, tablet, smartphone or other mobile device. And these secondary annotations must be manually synchronized or matched with the data from the ultrasonic tool or acoustic imager. This takes quite a lot of time and can be erroneous when matching the correct data with the corresponding secondary context information.

Some aspects of this disclosure are directed to acoustic analysis systems. The system is an acoustic sensor array composed of a plurality of acoustic sensor elements, and each acoustic sensor element receives an acoustic signal from an acoustic scene and outputs acoustic data based on the received acoustic signal. May include an array.

The system may include an electromagnetic imaging tool configured to receive electromagnetic radiation from the target scene and output electromagnetic image data indicating the received electromagnetic radiation. Electromagnetic imaging tools may be configured to detect electromagnetic radiation from a wavelength range, such as a range that includes a visible light spectrum and / or a near infrared light spectrum. In some systems, the electromagnetic imaging system may include a visible light camera module and / or an infrared camera module.

The system may include a user interface, a display, and a processor. The processor can communicate with the acoustic sensor array, the electromagnetic imaging tool, the user interface, and the display.

In some systems, the processor may be configured to receive electromagnetic data from an electromagnetic imaging tool and acoustic data from an acoustic sensor array. The processor may also generate acoustic image data for the scene based on the received acoustic data, generate a display image consisting of the combined acoustic image data and electromagnetic image data, and provide the display image to the display. In certain embodiments, the processor may receive annotation input from the user interface and update the display image on the display based on the received annotation input. Annotation input may consist of freestyle annotations received via the touch screen, icon selection or predefined shapes, and / or alphanumerical input.

In some systems, the processor determines one or more acoustic parameters associated with the received acoustic signal, eg, acoustically based on a comparison of one or more acoustic parameters with one or more predetermined thresholds. It is configured to determine the criticality associated with the signal. In certain embodiments, the processor can update the display based on the determined criticality. The user can annotate the image with the determined critical information. The user can also annotate the image with predetermined information such as the distance value to the target.

Details of one or more examples are provided in the accompanying drawings and the description below. Other features, purposes and effects will be apparent from the description, drawings and claims.

FIG. 1A shows a front view and a rear view of an exemplary acoustic image pickup device. FIG. 1B shows a front view and a rear view of an exemplary acoustic image pickup device.

FIG. 2 is a functional block diagram showing components of an example of an acoustic analysis system.

FIG. 3A shows a schematic diagram of an exemplary acoustic sensor array configuration within an acoustic analysis system. FIG. 3B shows a schematic diagram of an exemplary acoustic sensor array configuration within an acoustic analysis system. FIG. 3C shows a schematic diagram of an exemplary acoustic sensor array configuration within an acoustic analysis system.

FIG. 4A shows a schematic diagram of a parallax error in the generation of frames for visible light image data and acoustic image data. FIG. 4B shows a schematic diagram of a parallax error in the generation of frames for visible light image data and acoustic image data.

FIG. 5A shows the parallax correction between the visible light image and the acoustic image. FIG. 5B shows the parallax correction between the visible light image and the acoustic image.

5C is a colored version of FIGS. 5A and 5B. FIG. 5D is a colored version of FIGS. 5A and 5B.

FIG. 6 is a process flow diagram showing an exemplary method for generating a final image by combining acoustic image data and electromagnetic image data.

FIG. 7 is a process flow diagram showing an exemplary process for generating acoustic image data from a received acoustic signal.

FIG. 8 shows an exemplary look-up table for determining the appropriate algorithm and sensor array for use during the acoustic imaging process.

FIG. 9A is an exemplary plot of frequency content of received image data over time in an acoustic scene.

FIG. 9B shows an exemplary scene that includes multiple positions that emit acoustic signals.

FIG. 9C shows a plurality of combined acoustic and visible light image data in a plurality of pre-waiting defined frequency ranges.

FIG. 10A is an exemplary display image containing combined visible light image data and acoustic image data. FIG. 10B is an exemplary display image containing the combined visible light image data and acoustic image data.

FIG. 11A shows an exemplary plot of frequency vs. time of acoustic data in an acoustic scene. FIG. 11B shows an exemplary plot of frequency vs. time of acoustic data in an acoustic scene.

FIG. 12A shows a plurality of exemplary methods for comparing acoustic image data with past acoustic image data stored in a database. FIG. 12B shows a plurality of exemplary methods for comparing acoustic image data with past acoustic image data stored in a database. FIG. 12C shows a plurality of exemplary methods for comparing acoustic image data with past acoustic image data stored in a database.

FIG. 13 is a process flow diagram showing an exemplary operation of comparing received acoustic image data with a database for object diagnosis.

FIG. 14 shows the visualization of acoustic data using a gradation toning technique.

FIG. 15 shows a visualization of acoustic data using a plurality of shaded concentric circles.

FIG. 16 shows an example of visualization including both non-numeric information and alphanumerical information.

FIG. 17 shows another visualization example that includes both non-numeric information and alphanumerical information.

FIG. 18 shows another visualization example that includes both non-numeric information and alphanumerical information.

FIG. 19 shows an exemplary visualization showing different size and color indicators representing different acoustic parameter values.

FIG. 20 shows an exemplary visualization showing a plurality of indicators with different colors indicating the severity indicated by the acoustic signal from the corresponding position.

FIG. 21 shows a scene including indicators at a plurality of positions in the scene, which distinguish and show acoustic signals satisfying a predetermined condition.

FIG. 22 shows a display image including a plurality of icons arranged in the display image, showing the sound profile recognized in the scene.

FIG. 23 shows another exemplary display image showing acoustic data via a plurality of indicators using concentric circles and alphanumerical information that represent the acoustic intensity in relation to each acoustic signal.

FIG. 24 shows an exemplary display image with indicators and additional alphanumerical information associated with the displayed acoustic signal.

FIG. 25A shows a system comprising a display from which an indicator in a display image is selected and a laser pointer that emits a laser towards the scene.

25B shows the display shown in the system view of FIG.

FIG. 26 shows a display image including an indicator that has a gradation toning technique (palettization scene) that represents acoustic image data and includes acoustic image blending control.

FIG. 27 shows a display image having an acoustic image blending technique, including a palettigation technique for representing acoustic image data, including an indicator that includes acoustic image blending control.

FIG. 28 shows a display image including an indicator with a gradation toning indicating a position in the scene that satisfies one or more filter conditions.

FIG. 29 shows a display image including an indicator with concentric toning indicating a position in the scene that satisfies one or more filter conditions.

FIG. 30 shows a display image comprising two indicators, each indicator having a gradation toning indicating a position in the scene that satisfies different filter conditions.

FIG. 31 shows a display interface including a display image and a virtual keyboard.

FIG. 32 shows a display built into eyewear that can be worn by a user to display a display image.

FIG. 33A shows a dynamic display image including an indicator with dynamic intensity based on the pointing of the acoustic sensor array. FIG. 33B shows a dynamic display image including an indicator with dynamic intensity based on the pointing of the acoustic sensor array.

FIG. 34 shows an exemplary display image that provides a user or technician with a display related to the potential criticality of an air leak identified in the scene and the potential cost of loss due to the air leak.

FIG. 35 shows an example of a user annotating a display image with freeform annotations on the display.

FIG. 36 shows an example of an annotated display image that includes location information associated with the instruction.

FIG. 37 shows an example of a user annotating a display image with icon annotations on the display.

FIG. 38 shows an example of a user annotating a display image with shape annotations on the display.

FIG. 39 shows an example of an annotated display image including annotations of shapes and icons on the display.

FIG. 40 shows a display image and an interface that includes a multi-parameter data visualization that includes a plurality of frequency ranges on the right hand side of the display image.

FIG. 41 shows an interface that includes a display image and a multi-parameter display of frequency information, including a plurality of frequency ranges arranged along the lower edge of the display image.

FIG. 42 shows a display image including frequency information of a plurality of frequency bands and peak values of the plurality of frequency bands.

FIG. 43 shows a display image including a multi-parameter display showing intensity information for a plurality of frequencies.

FIG. 44 shows a multi-parameter display including a toned set of frequency ranges, in which the toning represents the decibel range in which each frequency range is classified.

FIG. 45 shows an exemplary display image including a multi-parameter display showing different frequency ranges and an indicator toned according to severity.

FIG. 46 shows the trend of intensity (dB display) vs. time in each of the plurality of frequency ranges in the multi-parameter display of the display image.

1A and 1B show a front view and a rear view of an exemplary acoustic imaging device. FIG. 1A shows the front surface of an acoustic imaging device 100 having a housing 102 that supports an acoustic sensor array 104 and an electromagnetic imaging tool 106. In one embodiment, the acoustic sensor array 104 comprises a plurality of acoustic sensor elements, each of which receives an acoustic signal from an acoustic scene and outputs acoustic data based on the received acoustic signal. It is configured in. The electromagnetic imaging tool 106 may be configured to receive electromagnetic radiation from the target scene and output electromagnetic image data representing the received electromagnetic radiation. The electromagnetic imaging tool 106 may be configured to detect electromagnetic radiation in one or more of a range of wavelengths such as visible light, infrared light, ultraviolet light and the like.

In the illustrated example, the acoustic image pickup device 100 includes an ambient light sensor 108 and a position sensor 116 such as GPS. The device 100 includes a laser pointer 110, which, in certain embodiments, includes a laser rangefinder. The device 100 includes a torch 112 that may be configured to emit visible light radiation towards the scene and an infrared irradiator 118 that may be configured to emit infrared radiation towards the scene. In one example, the device 100 may include an irradiator for illuminating the scene over any range of wavelengths. The apparatus 100 scenes a series of dots to determine a projector 114, such as an image reprojector, which can be configured to project the generated image onto a scene, such as a colored image, and / or, for example, the depth profile of the scene. Further includes a dot projector configured to project on.

FIG. 1B shows the back side of the acoustic imaging device 100. As illustrated, the device includes a display 120 capable of displaying images or other data. In one example, the display 120 includes a touch screen display. The acoustic image pickup device 100 includes a speaker capable of providing a voice feedback signal to the user, and a wireless interface 124 capable of enabling wireless communication between the acoustic image pickup device 100 and an external device. The device further includes a control 126 that may include one or more buttons, knobs, dials, switches, or other interface components to allow the user to connect to the acoustic imaging device 100. In one example, the control 126 and the touch screen display combine to provide a user interface for the acoustic imaging device 100.

In various embodiments, the acoustic imaging device need not include all the elements shown in the embodiments of FIGS. 1A and 1B. One or more of the illustrated components can be excluded from the acoustic imaging device. In one example, one or more of the components shown in the embodiments of FIGS. 1A and 1B may be included as part of the acoustic imaging system, but separately from the housing 102. Such components can be communicated with other components of the acoustic imaging system, for example, using the wireless interface 124, via wired or wireless communication technology.

FIG. 2 is a functional block diagram showing components of an example of the acoustic analysis system 200. The exemplary acoustic analysis system 200 of FIG. 2 can include a plurality of acoustic sensors such as microphones, MEMS, transducers, etc. arranged within the acoustic sensor array 202. Such an array can be one-dimensional, two-dimensional, or three-dimensional. In various examples, the acoustic sensor array can define any suitable size and shape. In one example, the acoustic sensor array 202 includes a plurality of acoustic sensors arranged in a grid pattern, such as, for example, an array of sensor elements arranged in vertical rows and horizontally parallel. In various examples, the acoustic sensor array 202 can include, for example, an array of horizontal x vertical columns such as 8x8, 16x16, 32x32, 64x64, 128x128, 256x256. .. Other examples are possible and the various sensor arrays do not need to contain the same number of rows as the columns. In certain embodiments, such sensors may be placed on a substrate, such as a printed circuit board (PCB) substrate, for example.

In the configuration shown in FIG. 2, the processor 212 communicating with the acoustic sensor array 202 can receive acoustic data from each of the plurality of acoustic sensors. During the exemplary operation of the acoustic analysis system 200, the processor 212 can communicate with the acoustic sensor array 202 to generate acoustic image data. For example, the processor 212 is configured to analyze data received from each of a plurality of acoustic sensors arranged in an acoustic sensor array and "backpropagate" the acoustic signal to an acoustic source to determine the acoustic scene. Can be done. In certain embodiments, processor 212 can generate digital "frames" of acoustic image data by identifying various source locations and intensities of acoustic signals throughout a two-dimensional scene. By generating a frame of acoustic image data, the processor 212 captures an acoustic image of the target scene at a substantially given point in time. In one example, a frame contains multiple pixels that make up an acoustic image, where each pixel represents part of the source scene in which the acoustic signal is backpropagated.

The components described as a processor in the acoustic analysis system 200 including the processor 212 are one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs). , Programmable logic circuits, etc., may be implemented as one or more processors, alone or optionally in combination. Processor 212 may also include memory when executed by processor 212 to store program instructions and related data that causes the acoustic analysis system 200 and processor 212 to perform functions resulting from them in the present disclosure. The memory may include fixed or removable magnetic, optical, or electrical media such as RAM, ROM, CD-ROM, hard disk or floppy magnetic disk, EEPROM, etc. The memory may also include a removable portion of memory that may be used to provide memory updates or memory capacity increases. Removable memory also allows the acoustic image data to be easily transferred to another computing device, or the acoustic image data to be removed before the acoustic analysis system 200 is used for another application. do. Processor 212 may also be implemented as a system-on-chip that integrates some or all components of a computer or other electronic system into a single chip. The processor 212 (processing circuit) may be configured to communicate the processed data to the display 214 or other output / control device 218.

In one embodiment, the acoustic sensors in the acoustic sensor array 202 generate a series of signals corresponding to the acoustic signals received by each acoustic sensor to represent an acoustic image. A "frame" of acoustic image data is generated when a signal from each acoustic sensor is acquired by scanning all the rows that make up the acoustic sensor array 202. In one example, the processor 212 may acquire an acoustic image frame at a speed sufficient to generate a video representation of the acoustic image data (eg, 30 Hz, or 60 Hz). Independent of the particular circuit, the acoustic analysis system 200 manipulates acoustic data representing the sound profile of the target scene to provide output that can be viewed, stored, transmitted, or otherwise used by the user. Can be configured.

In one embodiment, "backpropagation" of a received acoustic signal to generate acoustic image data comprises analyzing the received signal with a plurality of acoustic sensors in the acoustic sensor array 202, eg, via a processor. In various examples, the execution of backpropagation involves the distance to the target, frequency, sound intensity (eg, dB level), eg, the spacing and placement of individual sensors within one or more arrays of the sensor array. A function of one or more parameters, including dimensions / configurations. In certain embodiments, such parameters may be pre-programmed into the system, eg, in memory. For example, the characteristics of the acoustic sensor array 202 can be stored in a memory such as an internal memory or, in particular, a memory associated with the acoustic sensor array 202. Other parameters, such as the distance to the target, can be received in various ways. For example, in one example, the acoustic analysis system 200 includes a distance measuring tool 204 that communicates with the processor 212. The distance measuring tool may be configured to provide distance information representing the distance from the distance measuring tool 204 to a particular position in the target scene. Various distance measuring tools may include other known distance measuring devices such as laser rangefinders or other optical or audio distance measuring devices. Additional or alternative, the distance measurement tool may be configured to generate 3D depth data such that each part of the target scene has a distance value to the associated target. Thus, in one example, the distance measurement to the target as used herein can correspond to the distance to each position in the target scene. Such 3D depth data can be generated, for example, through multiple imaging tools with different views of the target scene, or through other known distance scanning tools. In general, in various embodiments, distance measuring tools are used, including, but not limited to, laser distance measurement, active sound distance measurement, passive ultrasonic distance measurement, LIDAR distance measurement, RADAR distance measurement, millimeter wave distance measurement, and the like. One or more distance measurement functions can be performed.

The distance information from the distance measuring tool 204 can be used for backpropagation calculation. On top of that, or separately, the system 200 may include a user interface 216 that allows the user to manually enter distance parameters to the target. For example, if the user knows the distance to a component suspected of producing an acoustic signal or is difficult to measure with the distance measuring tool 204, the user can give the system 200 a distance value to the target. You can enter it.

In the illustrated embodiment, the acoustic analysis system 200 includes an electromagnetic imaging tool 203 for generating image data representing a target scene. An exemplary electromagnetic imaging tool may be configured to receive electromagnetic radiation from a target scene and generate electromagnetic image data representing the received electromagnetic radiation. In one example, the electromagnetic imaging tool 203 may be configured to generate electromagnetic image data representing a range of wavelengths within the electromagnetic spectrum, such as infrared radiation, visible light radiation, and ultraviolet radiation. For example, in one embodiment, the electromagnetic timing tool 203 is one or more cameras configured to generate image data representing a range of wavelengths within the electromagnetic spectrum, such as the visible light camera module 206. Can include modules.

Visible light camera modules are generally well known. For example, smartphones and many other devices include various visible light camera modules. In certain embodiments, the visible light camera module 206 receives visible light energy from a target scene and, for example, visible light energy that can be displayed in the form of a visible light image on display 214 and / or stored in memory. Visible light energy can be configured to be concentrated on the visible light sensor to generate data. The visible light camera module 206 can now include any suitable component for performing the functions resulting from the module. In the example of FIG. 2, the visible light camera module 206 is shown as including a visible light lens assembly 208 and a visible light sensor 210. In one such embodiment, the visible light lens assembly 208 comprises at least one lens that receives the visible light energy emitted by the target scene and concentrates the visible light energy on the visible light sensor 210. The visible light sensor 210 can include a plurality of visible light sensor elements such as CMOS detectors, CCD detectors, PIN diodes, avalanche photodiodes and the like. The visible light sensor 210 responds to the concentrated energy by generating an electrical signal that can be converted and displayed as a visible light image on the display 214. In one example, the visible light module 206 is user configurable and can provide output to the display 214 in various formats, for example. Visible light camera module 206 may include irradiation or other changes in operating conditions or compensation for user preference. The visible light camera module can provide a digital output containing image data, which may include data in various formats (eg RGB, CYMK, YCbCr, etc.).

In the operation of an exemplary visible light camera module 206, the light energy received from the target scene may pass through the visible light lens assembly 208 and be converged on the visible light sensor 210. When light energy collides with a visible light sensor element of the visible light sensor 210, photons in the photodetector can be emitted and converted into a detection current. Processor 212 can process this detection current to form a visible light image of the target scene.

During use of the acoustic analysis system 200, the processor 212 can control the visible light camera module 206 to generate visible light data from the captured target scene in order to create a visible light image. Visible light data may include luminosity data indicating the color (s) associated with different parts of the captured target scene and / or the magnitude of light associated with different parts of the captured target scene. The processor 212 can generate a "frame" of visible light image data by measuring the response of each visible light sensor element of the acoustic analysis system 200 once. By generating a frame of visible light data, the processor 212 captures a visible light image of the target scene at a given point in time. Processor 212 may also iteratively measure the response of each visible light sensor element of the acoustic analysis system 200 to generate a dynamic visible light image (eg, moving image representation) of the target scene. In one example, the visible light camera module 206 may include its own dedicated processor or other circuit (eg, an ASIC) capable of operating the visible light camera module 206. In one such embodiment, the dedicated processor communicates with the processor 212 to provide visible light image data (eg, RGB image data) to the processor 212. In an alternative to rubbing, a dedicated processor for the visible light camera module 206 may be integrated with the processor 212.

By allowing each sensor element of the visible light camera module 206 to act as a sensor pixel, the processor 212 receives the electrical response of each sensor element, for example for visualization on display 214 and / or storage in memory. By converting to a time-multiplexed electrical signal that can be processed, a two-dimensional image or picture representation of visible light from the target scene can be generated.

The processor 212 can control the display 214 to display at least a portion of the captured visible light image of the target scene. In one example, the processor 212 controls the display 214 so that the electrical response of each sensor element of the visible light camera module 206 is associated with a single pixel on the display 214. In another example, the processor 212 may increase or decrease the resolution of the visible light image so that more or less pixels than the sensor elements in the visible light camera module 206 are displayed on the display 214. The processor 212 controls the display 214 to be less than the entire visible light image (eg, all parts of the target scene captured by the acoustic analysis system 200) or less than the entire visible light image (eg, captured by the acoustic analysis system 200). Can be displayed (less than the entire target scene).

In one embodiment, the processor 212 controls the display 214 to capture at least a portion of the visible light image captured by the acoustic analysis system 200 and at least a portion of the acoustic image generated via the acoustic sensor array 202. It can be displayed at the same time. Such simultaneous display may be useful in that it may help the operator to see the source of the acoustic signal simultaneously displayed in the acoustic image with reference to the features displayed in the visible light image. In various examples, the processor 212 controls the display 214 to arrange the visible and acoustic images side by side in a picture-in-picture arrangement in which one of the images surrounds the other, or the visible and acoustic images. It may be displayed in other appropriate arrangements that are displayed at the same time.

For example, the processor 212 can control the display 214 to display a combination of a visible light image and an acoustic image. In such an arrangement, for a set of pixels or pixels in a visible light image that represents part of the target scene, there is a corresponding set of pixels or pixels in the acoustic image that represent substantially the same part of the target scene. .. In various embodiments, the size and / or resolution of the acoustic and visible light images need not be the same. Thus, there may be a set of the other pixel of the acoustic or visible light image, or a set of pixels of different sizes, corresponding to one single pixel of the acoustic or visible light image. Similarly, one of the visible or acoustic images may have pixels that correspond to a set of pixels in the other image. Thus, as used herein, the correspondence does not need to be a one-to-one pixel relationship, but may include pixels or groups of pixels of mismatched sizes. Various combinatorial techniques can be performed for areas of non-matching size of the image, such as upsampling or downsampling one of the images, or combining pixels with the average value of the corresponding set of pixels. Other examples are known and are within the scope of this disclosure.

Therefore, the corresponding pixels do not have to have a direct one-to-one relationship. Rather, in certain embodiments, a single acoustic pixel has a plurality of corresponding visible light pixels, or a single visible light pixel has a plurality of corresponding acoustic pixels. Moreover or separately, in certain embodiments, not all visible light pixels have corresponding acoustic pixels, and vice versa. Such embodiments may indicate, for example, a picture-in-picture type display as described above. Therefore, visible light pixels do not necessarily have the same pixel coordinates in a visible light image, as do the corresponding acoustic pixels. Therefore, as used herein, the corresponding pixels are generally from any image (eg, visible light image, acoustic image, combined image, display image, etc.) that contains information from substantially the same part of the target scene. Points to the pixel of. Such pixels do not have to have a one-to-one relationship between images and do not have to have similar coordinate positions within each image.

Similarly, an image with corresponding pixels (ie, pixels representing the same part of the target scene) may be referred to as the corresponding image. Thus, in some such arrangements, the corresponding visible light and acoustic images can be superimposed on each other at the corresponding pixels. The operator may contact the user interface 216 to control the transparency or opacity of one or both of the images displayed on the display 214. For example, the operator may contact the user interface 216 to adjust the acoustic image between fully transparent and completely opaque, and the visible light image between completely transparent and completely opaque. Such an exemplary combinational arrangement, which may be referred to as an alpha blend arrangement, adjusts the display 214 to the operator to make the acoustics of any overlapping combination of the two images between the extremes of the acoustic image only and the visible light image only images. It may be possible to display an image of only an image or an image of only a visible light image. Processor 212 can also combine scene information with other data, such as alarm data. In general, alpha-blended combinations of visible and acoustic images can be included anywhere from 200% acoustic and 0% visible light to 0% acoustic and 200% visible light. In certain embodiments, the amount of blend can be adjusted by the user of the camera. Thus, in certain embodiments, the blended image can be adjusted between 200% visible light and 200% acoustics.

Further, in certain embodiments, the processor 212 is capable of interpreting and executing commands from the user interface 216 and / or the output / control device 218. Further, the input signal can be used to modify the processing of visible light and / or acoustic image data generated by the processor 212.

The operator may interact with the acoustic analysis system 200 via a user interface 216, which may include buttons, keys, or another mechanism for receiving input from the user. The operator may receive the output from the acoustic analysis system 200 via the display 214. Display 214 may be configured to display acoustic and / or visible light images in any acceptable toning or color scheme, which may change, for example, in response to user control. In certain embodiments, the acoustic image data may be presented in toning to represent different sizes of acoustic data from different locations in the scene. For example, in one example, the display 214 is configured to display an acoustic image in a single toning such as grayscale. In another example, the display 214 is configured to display an acoustic image in a color palette such as, for example, amber, ironbow, blue-red, or other high contrast color scheme. A combination of grayscale display and color palette display is also conceivable. In one example, a display configured to display such information may include processing power to generate and present such image data. In another example, being configured to display such information may include the ability to receive image data from other components such as processor 212. For example, processor 212 may generate a value for each pixel displayed (eg, RGB values, grayscale values, or other display options). Display 214 may receive such information and map each pixel to a visual display.

The processor 212 can control the display 214 to simultaneously display at least a portion of the acoustic image and at least a portion of the visible light image in any suitable arrangement, whereas the picture-in-picture arrangement is the same scene. Displaying the corresponding visible image of is in adjacent alignment helps the operator to easily focus and / or interpret the acoustic image.

A power source (not shown) supplies operating power to various components of the acoustic analysis system 200. In various examples, the power source may include rechargeable or non-rechargeable batteries and power generation circuits, AC power sources, inductive power pickups, photovoltaic power sources, or any other suitable power source configuration. A combination of the rechargeable battery and a power supply component such as another component configured to provide power for operating the device and / or charging the rechargeable battery is also possible.

During operation of the acoustic analysis system 200, the processor 212 controls the acoustic sensor array 202 and the visible light camera module 206 with the help of instructions related to program information stored in memory to control the visible light of the target scene. Generate images and acoustic images. The processor 212 also controls the display 214 to display a visible light image and / or an acoustic image generated by the acoustic analysis system 200.

As mentioned above, in some situations it can be difficult to identify and distinguish the actual (visible) features of the target scene in the acoustic image. In addition to supplementing the acoustic image with visible light information, in certain embodiments it may be useful to enhance the visible contours within the target scene. In certain embodiments, known contour detection methods can be performed on a visible light image of the target scene. Due to the correspondence between the acoustic image and the visible light image, the visible light pixels determined to represent the visible contour of the target scene correspond to the acoustic pixels that also represent the visible contour of the acoustic image. It will be appreciated that the "edge" used herein does not have to refer to the physical boundaries of the object, but can refer to a sufficiently sharp gradation of the visible light image. Examples may include physical boundaries of the object, color changes within the object, shading of the entire scene, and so on.

Although generally described with respect to FIG. 2 as including the visible light camera module 206, in one example, the electromagnetic imaging tool 203 of the acoustic analysis system 200 can generate imaging data representing various spectra. Tools can be included on or separately from it. For example, in various examples, the electromagnetic imaging tool 203 can generate infrared image data, visible light image data, ultraviolet image data, or any other useful wavelength, or a combination thereof. Can include tools. In certain embodiments, the acoustic imaging system can include an infrared camera module having an infrared lens assembly and an infrared sensor array. For example, it can include additional components to interface with the infrared camera module, for example, filed August 27, 2015, "Thermal Visualizing Combined Image and Contour Enhancement for Cameras (" EDGE ENHANCEMENT FOR THERMAL-VISIBLE COMBINED IMAGES AND CAMERAS) ”, such as those described in US Patent Application No. 14 / 837,757, which are transferred to the transferee of this application and by reference. The whole is incorporated.

In one example, two or more data streams may be blended for display. For example, an exemplary system including a visible light camera module 206, an acoustic sensor array 202, and an infrared camera module (not shown in FIG. 2) may include visible light (VL) image data, infrared (IR) image data, and more. And can be configured to produce an output image containing a blend of acoustic image data. In an exemplary blending technique, a display image can be represented by the equation α × IR + β × VL + γ × acoustic, where α + β + γ = 1. Generally, any number of data streams are combined for display. In various embodiments, blend ratios such as α, β, and γ can be set by the user. In addition to or separately from it, the set display program is, for example, a US patent entitled "VISIBLE LIGHT AND IR COMBINED IMAGE CAMERA WITH A LASER POINT". Different based on alarm conditions (eg, one or more values in one or more data streams satisfy a predetermined threshold) or other conditions, as described in No. 7,538,326. It can be configured to include an image data stream, which is transferred to the transferee of this application, which is incorporated by reference in its entirety.

One or more components of the acoustic analysis system 200 described with respect to FIG. 2 can be included in a portable (eg, handheld) acoustic analysis tool. For example, in one embodiment, the portable acoustic analysis tool may include a housing 230 configured to house components within the acoustic analysis tool. In one example, one or more components of the system 200 may be located outside the housing 230 of the acoustic analysis tool. For example, in one embodiment, the processor 212, display 214, user interface 216, and / or output control device 218 may be located outside the housing of the acoustic analysis tool, eg, wireless communication (eg, Bluetooth communication, etc.). It can communicate with various other system components via Wi-Fi, etc.). Such components that are external to the acoustic analysis tool can be provided via an external device such as, for example, a computer, smartphone, tablet, wearable device. On top of that or separately, other test and measurement or data acquisition tools configured to act as master or slave devices with respect to the acoustic analysis tool as well as various components of the external acoustic analysis system of the acoustic analysis tool. Can be provided to. The external device can communicate with the portable acoustic analysis tool via a wired and / or wireless connection and can be used to perform various processing, display, and / or interface steps.

In certain embodiments, such an external device can provide redundancy as a component housed in a portable acoustic analysis tool. For example, in certain embodiments, the acoustic analysis tool may include a display for displaying acoustic image data and may be further configured to communicate the image data to an external device for storage and / or display. .. Similarly, in one embodiment, a user is running an application (“app”) on a smartphone, tablet, computer, etc. to perform one or more functions that can also be performed by the acoustic analysis tool itself. Can interface with acoustic analysis tools via.

FIG. 3A is a schematic configuration of an exemplary configuration of an acoustic sensor array in an acoustic analysis system. In the illustrated example, the acoustic sensor array 302 includes a plurality of first acoustic sensors (shown in white) and a plurality of second acoustic sensors (shaded). The first acoustic sensor is arranged in the first array 320, and the second acoustic sensor is arranged in the second array 322. In one example, the first array 320 and the second array 322 can be selectively used to receive acoustic signals for producing acoustic image data. For example, in some configurations, the sensitivity of a particular acoustic sensor array to a particular acoustic frequency is a function of the distance between the acoustic sensor elements.

In some configurations, more closely spaced sensor elements (eg, second array 322) have higher frequency acoustic signals (eg, 20 kHz) than more spaced sensor elements (eg, first array 320). Sounds having frequencies above 20 kHz, such as ultrasonic signals of ~ 100 kHz) can be better decomposed. Similarly, a wider spacing sensor element (eg, first array 320) has a lower frequency acoustic signal (eg, less than 20 kHz) than a narrower spacing sensor element (eg, second array 322). May be more suitable for detecting. Various sensor elements placed apart from each other to detect acoustic signals in various frequency ranges such as ultra-low frequency (less than 20Hz), audible frequency (about 20Hz-20kHz), ultrasonic frequency (20kHz-100kHz). An acoustic sensor array may be provided. In certain embodiments, partial arrays (eg, all acoustic sensor elements other than array 320) may be used to optimize detection for a particular frequency band.

Further, in some examples, some acoustic sensor elements may be better suited to detect acoustic signals with different frequency characteristics, such as low or high frequencies. Thus, in certain embodiments, an array configured to detect a low frequency acoustic signal, such as, for example, a first array 320 with more spaced sensor elements, is more suitable for detecting the low frequency acoustic signal. It may also include a first acoustic sensor element. Similarly, an array configured to detect higher frequency acoustic signals, such as the second array 322, may include a second acoustic sensor element that is more suitable for detecting high frequency acoustic signals. Thus, in one example, the first array 320 and the second array 322 of acoustic sensor elements may include different types of acoustic sensor elements. Alternatively, in certain embodiments, the first array 320 and the second array 322 may include the same type of acoustic sensor element.

Thus, in an exemplary embodiment, the acoustic sensor array 302 can include a plurality of acoustic sensor element arrays such as the first array 320 and the second array 322. In certain embodiments, the arrays can be used individually or in combination. For example, in one example, the user may use the first array 320, the second array 322, or the first array 320 and the second array to perform the acoustic imaging procedure. You can choose to use both of the 322 at the same time. In one example, the user may choose which array (s) to use via the user interface. In addition or otherwise, in certain embodiments, the acoustic analysis system determines the array (s) to be used, based on the analysis of the received acoustic signal, or other input data such as the expected frequency range. Can be selected automatically. The configuration shown in FIG. 3A generally includes two arrays (first array 320 and second array 322) roughly arranged in a rectangular grid, with multiple acoustic sensor elements of any shape. It will be appreciated that it can be grouped into any number of individual arrays. Further, in certain embodiments, one or more acoustic sensor elements can be included in a plurality of separate arrays that can be selected for operation. As described elsewhere herein, in various embodiments, the process of backpropagating an acoustic signal to establish acoustic image data from a scene is performed based on the placement of acoustic sensor elements. NS. Therefore, the placement of the acoustic sensor may be known or otherwise accessible by the processor to perform the acoustic image generation technique.

The acoustic analysis system of FIG. 3A further includes a distance measuring tool 304 and a camera module 306 disposed within the acoustic sensor array 302. The camera module 306 may represent a camera module of an electromagnetic imaging tool (for example, 203), and may include a visible light camera module, an infrared camera module, an ultraviolet camera module, and the like. Further, although not shown in FIG. 3A, the acoustic analysis system may include one or more additional camera modules of the same type or different type as the camera module 306. In the illustrated example, the distance measuring tool 304 and the camera module 306 are arranged in a grid of acoustic sensor elements of the first array 320 and the second array 322. Although shown to be located between the grid sites in the first array 320 and the second array 322, in certain embodiments, one or more components (eg, camera module 306 and / or distance). The measurement tool 304 can be located at one or more corresponding grid sites within the first array 320 and / or the second array 322. In one such embodiment, the components (s). Can be placed at the grid site instead of the acoustic sensor element that would typically be in such a position according to the grid layout.

As described elsewhere herein, an acoustic sensor array can include acoustic sensor elements arranged in any of a variety of configurations. 3B and 3C are schematic views showing an exemplary acoustic sensor array configuration. FIG. 3B shows an acoustic sensor array 390 containing a plurality of acoustic sensor elements evenly spaced within a substantially square grid. The distance measuring tool 314 and the camera array 316 are arranged in the acoustic sensor array 390. In the illustrated example, the acoustic sensor elements in the acoustic sensor array 390 are the same type of sensor, but in certain embodiments, different types of acoustic sensor elements can be used in the array 390.

FIG. 3C shows a plurality of acoustic sensor arrays. The acoustic sensor arrays 392, 394, and 396 each include a plurality of acoustic sensor elements arranged in arrays of different shapes. In the example of FIG. 3C, the acoustic sensor arrays 392, 394, and 396 can be used separately or together in any combination to create sensor arrays of various sizes. In the illustrated embodiment, the sensor elements of the array 396 are located closer to each other than the sensor elements of the array 392. In one example, the array 396 is designed to sense high frequency acoustic data and the array 392 is designed to sense low frequency acoustic data.

In various embodiments, the arrays 392, 394, and 396 can include the same or different types of acoustic sensor elements. For example, the acoustic sensor array 392 can include a sensor element having a frequency operating range lower than the frequency operating range of the sensor element of the acoustic sensor array 396.

As described elsewhere herein, in one example, different acoustic sensor arrays (eg, 392, 394, 396) are in different modes of operation (eg, different desired frequency spectra to be imaged). It can be selectively turned off and turned on. On top of that or separately, various acoustic sensor elements (eg, some or all of the acoustic sensor elements in one or more sensor arrays) can be enabled or disabled according to the desired system operation. Is. For example, in one acoustic imaging process, data from a large number of sensor elements (eg, densely arranged sensor elements, such as in a sensor array 396) will slightly improve the resolution of the acoustic image data, but each sensor. Extracting acoustic image data from the data received by the element requires the necessary processing costs. That is, in one example, the increased processing load (eg, in cost, processing time, power consumption, etc.) required to process a large number of input signals (eg, from a large number of acoustic sensor elements) is additional data. Negatively compared to any additional signal resolution provided by the stream. Therefore, in certain embodiments, it may be worthwhile to invalidate or ignore data from one or more acoustic sensor elements, depending on the desired acoustic imaging operation.

Similar to the systems of FIGS. 3A and 3B, the system of FIG. 3C includes an acoustic sensor array 392, 394, and a distance measuring tool 314 and a camera array 316 disposed within 396. In one example, additional components (eg, used to image different parts of the electromagnetic spectrum from the camera array 316), such as an additional camera array, are similarly within the acoustic sensor arrays 392, 394, and 396. Can be placed in. 3A-2C show distance measurement tools and / or one or more imaging tools (eg, visible light camera modules, although shown to be located within one or more acoustic sensor arrays. Infrared camera modules, UV sensors, etc.) can be located outside the acoustic sensor array (s). In one such example, the distance measurement tool and / or one or more imaging tools located outside the acoustic sensor array (s) are provided by the acoustic imaging tool, eg, a housing that houses the acoustic sensor array. Can be supported by or can be placed outside the housing of the acoustic imaging tool.

In one example, a general inconsistency between an acoustic sensor array and an imaging tool such as a camera module can lead to poor overlay of the corresponding image data generated by the acoustic sensor array and the imaging tool. FIG. 4A shows a schematic diagram of a parallax error in the generation of frames for visible light image data and acoustic image data. In general, parallax errors can be vertical, horizontal, or both. In the illustrated embodiment, it is an imaging tool that includes an acoustic sensor array 420 and a visible light camera module 406. The visible light image frame 440 is shown to be captured according to the field of view 441 of the visible light camera module 406, while the acoustic image frame 450 is shown to be captured according to the field of view 451 of the acoustic sensor array 420.

As shown, the visible light image frame 440 and the acoustic imaging frame 450 are not aligned with each other. In certain embodiments, the processor (eg, processor 212 in FIG. 2) operates one or both of the visible light image frame 440 and the acoustic image frame 450 in order to align the visible light image data and the acoustic image data. It is composed. Such an operation may include shifting one image frame with respect to the other. The amount by which the image frames are shifted relative to each other can be determined based on a variety of factors, including, for example, the distance from the visible light camera module 406 and / or the acoustic sensor array 420 to the target. Such distance data can be determined, for example, by using the distance measuring tool 404 or by receiving the distance value via a user interface (eg, 216).

FIG. 4B is a schematic similar to that of FIG. 4A, but includes a visible light image of the scene. In the example of FIG. 4B, the visible light image 442 shows a scene of a plurality of power lines and support towers. The acoustic image 452 includes a plurality of positions 454, 456, 458, such positions indicating high magnitude acoustic data coming from them. As shown, both the visible light image 442 and the acoustic image 452 are displayed simultaneously. However, observations of both images show at least one acoustic image maxima at position 458, which appears to be inconsistent with the particular structure of the visible light image 442. Therefore, a person observing both images can conclude that there is an overlay defect (eg, parallax error) between the acoustic image 452 and the visible light image 442.

5A and 5B show the parallax correction between the visible light image and the acoustic image. FIG. 5A shows a visible light image 542 and an acoustic image 552, similar to FIG. 4B. The acoustic image 552 includes local maxima at positions 554, 556, and 558. As can be seen, the maximum values at positions 554 and 558 do not appear to match any of the structures of the visible light image. In the example of FIG. 5B, the visible light image 542 and the acoustic image 552 are superimposed on each other. Maxima at positions 554, 556, and 558 in the acoustic image appear to coincide with various positions in the visible light image 542.

During use, the operator may look at the representation in FIG. 5B (eg, via display 214) to determine an approximate location within the visible scene 542 that is likely to be the source of the received acoustic signal. Such signals can be further processed to determine information about the acoustic signatures of the various components in the scene. In various embodiments, acoustic parameters such as frequency content, periodicity, amplitude, etc. can be analyzed for different positions in the acoustic image. When overlaying visible light data so that such parameters can be associated with various system components, acoustic image data is used to capture various characteristics (eg, performance characteristics) of the object in the visible light image. Can be analyzed.

5C and 5D are colored versions of FIGS. 5A and 5B. Positions 554, 556, and 558 show a circular gradation of color, as shown in FIGS. 5A and 5B and more easily seen in the colored representations of FIGS. 5C and 5D. As described elsewhere herein, the acoustic image data can be visually represented according to toning techniques, and each pixel of the acoustic image data is colored based on the acoustic intensity at the corresponding location. .. Therefore, in the exemplary representation of FIGS. 5A-5D, the circular gradation at positions 554, 556, 558 generally represents a gradation of acoustic intensity in the imaging plane based on the backpropagated received acoustic signal.

Illustrative diagrams of FIGS. 4A, 4B, 5A-5D are described with respect to acoustic and visible light image data, but such processes are similarly performed with various electromagnetic image data. It will be understood that it can be done. For example, as described elsewhere herein, in various embodiments, various such processes include acoustic image data and one or more visible light image data, infrared image data, ultraviolet images. It can be executed using a combination with data and so on.

As described elsewhere herein, in certain embodiments, the backpropagation of the acoustic signal for forming an acoustic image can be based on the distance value to the target. That is, in one example, the backpropagation calculation can be based on a distance and can include determining a two-dimensional acoustic scene located at that distance from the acoustic sensor array. Given a two-dimensional imaging plane, spherical sound waves emanating from sources in the plane generally appear circular in cross section and are radially attenuated in intensity, as shown in FIGS. 5A-5B.

In one such example, the portion of the acoustic scene that represents the data that is not located at a distance to the target used in the backpropagation calculation is the acoustic image data, such as the inaccuracy of the position of one or more sounds in the scene. Brings an error to. Such an error occurs when the acoustic image is displayed at the same time as other image data (eg, electromagnetic image data such as visible light, infrared, or ultraviolet image data) (eg, blended, combined, etc.). , May lead to misalignment error between acoustic image data and other image data. Thus, in certain embodiments, certain techniques for correcting parallax errors (eg, as shown in FIGS. 5A and 5B) up to the target used in the backpropagation calculation to generate acoustic image data. Includes adjusting the distance value of.

In some cases, the system performs a backpropagation process using the distance value to the first target, and the display image as shown in FIG. 5A where the acoustic image data and another data stream may not be aligned. May be configured to display. The acoustic analysis system can then adjust the distance value to the target used for backpropagation, perform the backpropagation again, and update the display image with the new acoustic image data. This process can be repeated and the acoustic analysis system iterates through distance values to multiple targets while the user observes the resulting display image on the display. As the distance value to the target changes, the user may observe a gradual transition from the display image shown in FIG. 5A to the display image shown in FIG. 5B. In some such cases, the user may visually observe when the acoustic image data appears to be properly superimposed on another data stream, such as electromagnetic image data. The user informs the acoustic analysis system that the acoustic image data appears to be properly superimposed and informs the system that the distance value to the target used to perform the latest backpropagation is almost correct. It may indicate that the value of that distance may be stored in memory as the correct distance to the target. Similarly, when the user updates the display image with the new distance value in the updated backpropagation process, the user reaches the target until the user confirms that the acoustic image data is properly overlaid. The distance value of can be adjusted manually. The user may choose to save the current distance to the target in the acoustic analysis system as the current distance to the target.

In one example, correcting a parallax error is adjusting the position of the acoustic image data relative to other image data (eg, electromagnetic image data) in a predetermined amount and direction based on the distance data to the target. May include. In certain embodiments, such adjustment is independent of the generation of acoustic image data by backpropagating the acoustic signal to the distance to the identified target.

In one embodiment, in addition to being used to generate acoustic image data and reduce parallax errors between the acoustic image data and other image data, the distance value to the target is for other determinations. May be used. For example, in one example, the processor (eg, 212) focuses on or focuses on an image, such as an infrared image, as described in US Pat. No. 7,538,326 incorporated by reference. Distance values to the target can be used to assist the user in aligning. As described therein, it can also be used to correct parallax errors between visible light image data and infrared image data. Therefore, in one example, distance values can be used to superimpose acoustic image data on electromagnetic image data such as infrared image data or visible light image data.

As described elsewhere herein, in one example, a distance measuring tool (eg, 204) provides distance information that can be used by a processor (eg, 212) to generate and superimpose acoustic image data. It is configured to do. In one embodiment, the distance measuring tool comprises a laser rangefinder configured to emit light to the target scene at the position where the distance is measured. In one such example, the laser rangefinder can emit light within the visible spectrum, so the user sees the laser spot in the physical scene and the rangefinder measures the distance to the desired part of the scene. You can confirm that it is. On or apart from that, the laser rangefinder is configured such that one or more imaging components (eg, camera modules) emit light within a spectrum that is sensitive. Therefore, a user viewing a target scene through an analysis tool (eg, through display 214) will observe the laser spot in the scene and measure the distance the laser is to the correct position in the target scene. You can confirm that you are there. In one example, the processor (eg, 212) has the laser spot in the acoustic scene based on the current distance value (eg, based on the known distance-based parallax relationship between the laser rangefinder and the acoustic sensor array). It may be configured to generate a reference mark in the display image that represents the position placed in. The position of the reference mark can be compared to the actual position of the laser mark (eg, graphically on the display and / or physically within the target scene) and the scene can be adjusted until the reference mark matches the laser. Such a process can be performed in the same manner as the infrared overlay and focus technique described in US Pat. No. 7,538,326 incorporated by reference.

FIG. 6 is a process flow diagram showing an exemplary method for generating a final image by combining acoustic image data and electromagnetic image data. This method includes a step of receiving an acoustic signal via an acoustic sensor array (680) and a step of receiving distance information (682). Distance information may be received via a distance measuring device and / or a user interface, for example via manual input or as a result of a distance adjustment process in which the distance is determined based on the observed superposition. can.

The method further comprises backpropagating the received acoustic signal to determine acoustic image data representing the acoustic scene (684). As described elsewhere herein, backpropagation, in combination with received distance information, analyzes multiple acoustic signals received by multiple sensor elements within an acoustic sensor array to analyze the received acoustic signal. Can include determining the source pattern of.

The method of FIG. 6 further includes a step of capturing electromagnetic image data (686) and a step of superimposing the electromagnetic image data on the acoustic image data (688). In one embodiment, superimposing the electromagnetic image data on the acoustic image data is performed as part of a backpropagation step for generating the acoustic image data (684). In another example, superimposing the electromagnetic image data on the acoustic image data is performed separately from the generation of the acoustic image data.

The method of FIG. 6 includes combining acoustic image data with electromagnetic image data (690) to generate a display image. Combining electromagnetic image data with acoustic image data, as described elsewhere herein, may include alpha blending the electromagnetic image data with the acoustic image data. Combining image data means overlaying one image data set to the other image data set, such as in picture-in-picture mode, or image data at a position where certain conditions (such as alarm conditions) are met. Includes overlaying the set. The display image is separate from the sensor array, for example, through a display supported by a housing that supports the acoustic sensor array and / or, such as the display of an external device (eg, smartphone, tablet, computer, etc.). It may be presented to the user via the display.

On top of that or separately, display images can be stored in local (eg, onboard) memory and / or remote memory for future viewing. In certain embodiments, the stored display image contains metadata that allows future adjustment of display image characteristics such as blend ratio, backpropagation distance, or other parameters used to generate the image. be able to. In some examples, raw acoustic signal data and / or electromagnetic image data can be stored with the display image for subsequent processing or analysis.

Although shown as a method of combining acoustic image data and electromagnetic image data to generate a final image, the acoustic image data can be applied to any part of the electromagnetic spectrum such as visible light image data, infrared image data, and ultraviolet image data. It will be appreciated that the method of FIG. 6 can be used to combine with one or more sets of straddling image data. In one such example, multiple sets of image data, such as visible light image data and infrared image data, are both combined with acoustic image data to generate a display image via a method similar to that described with respect to FIG. can do.

In one example, receiving an acoustic signal via a sensor array (680) can include selecting an acoustic sensor array for receiving the acoustic signal. For example, as described with respect to FIGS. 3A-C, an acoustic analysis system can include a plurality of acoustic sensor arrays that may be suitable for analyzing acoustic signals of varying frequencies. Moreover or separately, in one example, different acoustic sensor arrays may be useful for analyzing acoustic signals propagating from different distances. In certain embodiments, different arrays can be nested together. On top of that or separately, partial arrays can be selectively used to receive acoustic image signals.

For example, FIG. 3A shows a first array 320 and a second array 322 nested within a first array. In an exemplary embodiment, the first array 320 is a sensor array configured (eg, at intervals) to receive acoustic signals and generate acoustic image data for frequencies in the first frequency range. Can include. The second array 322 is configured to be used alone or in combination with all or part of the first array 320, for example, to generate acoustic image data for frequencies in the second frequency range. The second sensor array can be included.

Similarly, FIG. 3C shows the first array 392, the second array 394 at least partially nested within the first array 392, and at least a portion within the first array 392 and the second array 394. A third array 396 nested in a targeted manner is shown. In certain embodiments, the first array 392 may be configured to receive an acoustic signal and generate acoustic image data for frequencies in the first frequency range. The second array 394 can be used in conjunction with all or part of the first array 392 to receive acoustic signals and generate acoustic image data for frequencies in the second frequency range. The third array 396 alone, with all or part of the second array 394, and / or the first, to receive the acoustic signal and generate acoustic image data for frequencies in the third frequency range. Can be used with all or part of the array 392 of.

In one embodiment, in a nested array configuration, the acoustic sensor elements from one array are acoustically, for example, generally like the elements of a third array 396 between the elements of the first array 392. May be placed between sensor elements. In one such example, the acoustic sensor element in the nested array (eg, the third array 396) is the acoustic sensor element in the array in which it is nested (eg, the first array 392). Can be placed on the same plane as, in front of, or behind.

In various implementations, arrays used to sense higher frequency acoustic signals generally require shorter distances between individual sensors. Thus, with respect to FIG. 3C, for example, the third array 396 may be more suitable for performing an acoustic imaging process involving high frequency acoustic signals. Other sensor arrays (eg, first array 392) may be sufficient to perform an acoustic imaging process containing lower frequency signals and are from fewer acoustic sensor elements when compared to array 396. It can be used to reduce the computational demand to process the signal. Thus, in one example, the high frequency sensor array can be nested within the low frequency sensor array. As described elsewhere herein, such arrays can generally operate individually (eg, through switching between active arrays) or together.

In addition to or instead of selecting the appropriate sensor array based on the expected / desired frequency spectrum for analysis, in one example, different sensor arrays acoustically image at different distances to the target scene. It may be better suited to run the process. For example, in one embodiment, when the distance between the acoustic sensor array and the target scene is small, the outer sensor elements in the acoustic sensor array provide significantly less useful acoustic information than the more centrally located sensor elements. May be received from the target scene.

On the other hand, if the distance between the acoustic sensor array and the target scene is large, the acoustic sensor elements placed in close proximity may not provide useful information by themselves. That is, when the first and second acoustic sensor elements are close to each other and the target scene is generally far away, the second acoustic sensor element does not provide significantly different information than the first acoustic sensor element. There is. Therefore, the data streams from such first and second sensor elements are redundant and may unnecessarily consume processing time and resources for analysis.

In addition to affecting which sensor array is most suitable for performing acoustic imaging, as described elsewhere herein, the distance to the target is the acoustic image from the received acoustic signal. It can also be used to perform backpropagation to determine data. However, in addition to being the input value to the backpropagation algorithm, the distance to the target can be used to select the appropriate backpropagation algorithm to use. For example, in one example, at long distances, a sound wave propagating in a sphere can be approximated to be substantially planar relative to the size of the acoustic sensor array. Thus, in certain embodiments, backpropagation of the received acoustic signal can include acoustic beamforming calculations if the distance to the target is large. However, if it is close to the source of the sound wave, the plane approximation of the sound wave may not be appropriate. Therefore, various backpropagation algorithms such as near-field acoustic holography can be used.

As described, the distance metric to the target is the determination of the active sensor array (s), the determination of the backpropagation algorithm, the execution of the backpropagation algorithm, and / or the obtained acoustic image as electromagnetic image data. It can be used in a variety of ways in acoustic imaging processes, such as overlaying on (eg, visible light, infrared light, etc.). FIG. 7 is a process flow diagram showing an exemplary process for generating acoustic image data from a received acoustic signal.

The process of FIG. 7 includes receiving distance information from, for example, a distance measuring device (780), or receiving input distance information, such as via a user interface. The method further comprises the step (782) of selecting one or more acoustic sensor arrays (s) for performing acoustic imaging based on the received distance information. As described, in various examples, the selected array can include a single array, a combination of multiple arrays, or a portion of one or more arrays.

The method of FIG. 7 further comprises the step (784) of selecting a processing technique for performing acoustic imaging based on the received distance information. In one example, the choice of processing technique can include the choice of backpropagation algorithm for generating acoustic image data from the acoustic signal.

After selecting an acoustic sensor array (782) and selecting a processing technique (784) to perform acoustic imaging, the method is the step of receiving an acoustic signal via the selected acoustic sensor array (782). 786) is included. The received acoustic signal is then backpropagated using the distance and selected processing technique to determine the acoustic image data (788).

In various embodiments, each step of FIG. 7 can be performed by a user, an acoustic analysis system (eg, via processor 212), or a combination thereof. For example, in certain embodiments, the processor may be configured to receive distance information (780) via a distance measuring tool and / or user input. In one example, for example, if the distance to an object is known and / or difficult to analyze via a distance measuring tool (eg, a small object size and / or a long distance to a target, etc.). ), The user can enter a value that overrides the measured distance for use as distance information. The processor may be further configured, for example, using a look-up table or other database to automatically select the appropriate acoustic sensor array to perform acoustic imaging based on the distance information received. In certain embodiments, selecting an acoustic sensor array comprises enabling and / or disabling one or more acoustic sensor elements in order to utilize the desired acoustic sensor array.

Similarly, in one example, the processor may be configured to automatically select a processing technique (eg, backpropagation algorithm) for performing acoustic imaging based on received distance information. In one such example, this can include selecting one from a plurality of known processing techniques stored in memory. On top of that, or separately, choosing a processing technique can result in the adjustment of part of a single algorithm to reach the desired processing technique. For example, in one embodiment, a single backpropagation algorithm may contain multiple terms and variables (eg, based on distance information). In one such example, selecting the processing technique (784) is a single algorithm, such as adjusting the coefficients of one or more terms (eg, setting various coefficients to zero or one). May include defining one or more values in.

Therefore, in one embodiment, the acoustic imaging system proposes and / or automatically implements a selected acoustic sensor array and / or processing technique (eg, backpropagation algorithm) based on the received distance data. Allows you to automate several steps in the acoustic imaging process. This allows the acoustic imaging process to be accelerated, improved, and simplified, eliminating the need for an acoustic imaging expert to perform the acoustic imaging process. Therefore, in various examples, the acoustic imaging system automatically implements such parameters, informs the user that such parameters are about to be implemented, and permits the user to implement such parameters. And can suggest such parameters for manual input by the user.

Automatic selection and / or proposals for such parameters (eg, processing techniques, sensor arrays) optimize the positioning of acoustic image data with respect to other forms of image data, processing speed, and analysis of acoustic image data. Can be useful for. For example, as described elsewhere herein, accurate backpropagation decisions (eg, using appropriate algorithms and / or accurate distance metrics) can be used for acoustic image data and other (eg, for example). , Visible light, infrared rays, etc.) It is possible to reduce the misalignment error with the image data. In addition, utilizing appropriate algorithms and / or sensor arrays that can be automatically selected or suggested by the acoustic analysis system may optimize the accuracy of the thermal image data and allow analysis of the received acoustic data. ..

As described, in one example, the acoustic analysis system may be configured to automatically select an algorithm and / or sensor array for performing an acoustic imaging process based on received distance information. In one such embodiment, the system looks up table to determine which of a plurality of backpropagation algorithms and acoustic sensor arrays to use, eg, stored in memory, to determine acoustic image data. including. FIG. 8 shows an exemplary look-up table for determining the appropriate algorithm and sensor array for use during the acoustic imaging process.

In the illustrated example, the look-up table of FIG. 8 contains N columns, each representing an array 1, an array 2, ..., An array N of different arrays. In various examples, each array contains a unique set of placed acoustic sensor elements. Different arrays may include sensor elements arranged in a grid (eg, Array 392 and Array 396 in FIG. 3C). The array in the look-up table can also contain a combination of sensor elements from one or more such grids. In general, in one embodiment, each of Array 1, Array 2, ..., Array N corresponds to a unique combination of acoustic sensor elements. Some of such combinations can include the entire set of sensor elements arranged in a particular grid, or can include a subset of the sensor elements placed in a particular grid. Any of the various combinations of acoustic sensor elements are possible options for use as a sensor array in a look-up table.

The lookup table of FIG. 8 further includes M rows, each representing algorithm 1, algorithm 2, ..., Algorithm M of different algorithms. In one example, different algorithms may include different processes for performing backpropagation analysis of the received acoustic signal. As described elsewhere herein, in some examples, different algorithms may be similar to each other, with different coefficients and / or terms for modifying the backpropagation results.

The exemplary lookup table of FIG. 8 contains M × N entries. In one embodiment, an acoustic analysis system utilizing such a look-up table is configured to analyze the received distance information and classify the distance information into one of the M × N bins, where each bin is illustrated. Corresponds to 8 lookup table entries. In such an example, when the acoustic analysis system receives the distance information, the system finds an entry (i, j) in the look-up table corresponding to the bin in which the distance information resides and is suitable for use during the acoustic imaging process. Algorithms and sensor arrays can be determined. For example, if the received distance information corresponds to the bin associated with the entry (i, j), the acoustic analysis system can automatically utilize or suggest the use of algorithm i and array j for the acoustic imaging process.

In various such examples, the distance information bins can accommodate a range of uniform sizes, for example, the first bin corresponds to a distance within one foot and the second bin corresponds to 1-2. For example, it corresponds to the distance of feet. In another example, the bin does not have to accommodate uniform sized distance spans. Further, in certain embodiments, less than M × N bottles can be used. For example, in certain embodiments, there may be an algorithm (eg, algorithm x) that has never been used with a particular array (eg, array y). Therefore, in such an example, there is no corresponding distance information bin corresponding to the entry (x, y) of the M × N look-up table.

In certain embodiments, statistical analysis of populated distance bins can be used to identify the most common distances or distance ranges within a target scene. In certain such embodiments, distance bins with a maximum number of corresponding positions (eg, the maximum number of positions with respect to an acoustic signal) may be used as distance information in the process of FIG. That is, in certain embodiments, the acoustic sensor array and / or processing technique utilized may be implemented and / or recommended based on statistical analysis of the distance distribution of various objects in the target scene. This makes it possible to increase the probability that the sensor array and / or processing technique used for acoustic imaging of the scene is appropriate for the maximum number of positions in the acoustic scene.

Moreover or separately, parameters other than distance information can be used to select the appropriate sensor array and / or processing technique to use for the generation of acoustic image data. As described elsewhere herein, various sensor arrays may be configured to be responsive to specific frequencies and / or frequency bands. In one example, similar different backpropagation calculations may be used according to different acoustic signal frequency content. Therefore, in some examples, one or more parameters may be used to determine the processing technique and / or the acoustic sensor array.

In certain embodiments, an acoustic analysis system may be used to initially analyze various parameters of received acoustic signal processing / analysis. Referring back to FIG. 7, the method for generating acoustic image data can include the step (790) of analyzing the frequency content of the received signal after receiving the acoustic signal (786). In one such example, if an acoustic sensor array (s) and / or processing techniques are selected (eg, via steps 782 and / or 784, respectively), the method is based on the analyzed frequency content. The step (792) of updating the selected array (s) and / or updating the selected processing technique can be included.

After updating the sensor array (s) and / or processing technology, the method can perform various actions using the updated parameters. For example, if the selected sensor array is updated based on the analyzed frequency content (790) (792), the new acoustic signal is received from the (newly) selected acoustic sensor array (786). Can then be backpropagated to determine the acoustic image data (788). Alternatively, if the processing technique is updated at 792, the already captured acoustic signal can be backpropagated according to the updated processing technique to determine the updated acoustic image data. If both the processing technology and the sensor array are updated, the new acoustic signal can be received using the updated sensor array and can be backpropagated according to the updated processing technology.

In certain embodiments, the acoustic analysis system is capable of receiving frequency information (778) without analyzing the frequency content of the received acoustic signal (790). For example, in one example, the acoustic analysis system may receive information about the desired or expected frequency range for future acoustic analysis. In some such examples, the desired or expected frequency information may be used to select one or more sensor arrays and / or processing techniques that best fit the frequency information. In one such example, the step of selecting the acoustic sensor array (s) (782) and / or the step of selecting the processing technique (784) is in addition to or instead of the received distance information the received frequency. May be informed.

In one example, the received acoustic signal (eg, received via an acoustic sensor element) may be analyzed, for example, via a processor in an acoustic analysis system (eg, 210). Such analysis includes acoustic signals such as frequency, intensity, periodicity, apparent proximity (eg, distance estimated based on the received acoustic signal), measured proximity, or any combination thereof. May be used to determine one or more properties. In some examples, the acoustic image data may be filtered to display, for example, only the acoustic image data representing the acoustic signal having a particular frequency content, periodicity, and the like. In one example, any number of such filters can be applied at the same time.

As described elsewhere herein, in certain embodiments, a series of frames of acoustic image data may be captured over time, similar to acoustic video data. Moreover, or separately, the acoustic signal is repeatedly sampled and analyzed in some cases, even if the acoustic image data is not repeatedly generated. Therefore, parameters of acoustic data such as frequency can be monitored over time with or without repeated acoustic image data generation (eg, video).

FIG. 9A is an exemplary plot of frequency content of received image data over time in an acoustic scene. As shown, the acoustic scene represented by the plot of FIG. 9A generally includes four sustained frequencies over time, labeled as frequency 1, frequency 2, frequency 3, and frequency 4. Frequency data, such as the frequency content of the target scene, can be determined, for example, by processing the received acoustic signal using the Fast Fourier Transform (FFT) or other known frequency analysis method.

FIG. 9B shows an exemplary scene that includes multiple positions that emit acoustic signals. In the illustrated image, the acoustic image data is combined with the visible light image data to show the acoustic signals present at positions 910, 920, 930, and 940. In certain embodiments, the acoustic analysis system is configured to display acoustic image data in any detected frequency range. For example, in an exemplary embodiment, position 910 includes acoustic image data including frequency 1, position 920 contains acoustic image data including frequency 2, and position 930 contains acoustic image data including frequency 3. , Position 940 contains acoustic image data including frequency 4.

In one such example, displaying a representative frequency range of acoustic image data is a selectable mode of operation. Similarly, in certain embodiments, the acoustic analysis system is configured to display acoustic image data representing only frequencies within a predetermined frequency band. In one such example, displaying acoustic image data representing a given frequency range selects one or more acoustic sensor arrays for receiving acoustic signals from which acoustic image data is generated. Including that. Such an array may be configured to receive a selective frequency range. Similarly, in some examples, one or more filters may be used to limit the frequency content used to generate acoustic image data. On top of that, or separately, in certain embodiments, acoustic image data containing information representing a wide range of frequencies is analyzed and only if the acoustic image data meets certain conditions (eg, within a given frequency range). , Can be shown on the display.

FIG. 9C shows a plurality of combined acoustic and visible light image data over a plurality of predefined frequency ranges. The first image contains acoustic image data at a first position 910 that includes frequency content of frequency 1. The second image contains acoustic image data at a second position 920 that includes frequency content of frequency 2. The third image contains acoustic image data at a third position 930 that includes frequency content at frequency 3. The fourth image contains acoustic image data at a fourth position 940 that includes frequency content at frequency 4.

In an exemplary embodiment, the user has a variety of ranges including frequency 1, frequency 2, frequency 3, or frequency 4 to filter acoustic image data representing frequency content outside the selected frequency range. You can select the frequency range. Thus, in such an example, any of the first, second, third, or fourth images may be displayed as a result of the desired frequency range selected by the user.

On or apart from that, in one example, the acoustic analysis system may cyclically display between multiple display images, each with different frequency content. For example, with respect to FIG. 9C, in an exemplary embodiment, the acoustic analysis system can display the first, second, third, and fourth images as indicated by the arrows in FIG. 9C.

In one example, the display image may include text or other display representing the frequency content displayed in the image, and the user may include acoustic image data representing which position in the image represents the particular frequency content. Can be observed. For example, with respect to FIG. 9C, each image may indicate a textual representation of the frequency represented by the acoustic image data. With respect to FIG. 9B, an image showing a plurality of frequency ranges may include display of frequency content at each position including acoustic image data. In one such example, the user may, for example, select a position in the image via the user interface and view the frequency content present at that position in the acoustic scene. For example, the user may select the first position 910 and the acoustic analysis system may present the frequency content of the first position (eg, frequency 1). Thus, in various examples, the user can see where in the scene corresponds to a particular frequency content, and / or by seeing which frequency content exists at different locations, and so on. An acoustic analysis system can be used to analyze frequency content.

Filtering acoustic image data by frequency during an exemplary acoustic imaging operation can help reduce image clutter, for example from the background or other non-essential sounds. In an exemplary acoustic imaging procedure, the user may wish to remove background sounds such as floor noise in an industrial environment. In one such example, background noise may primarily include low frequency noise. Therefore, the user may choose to display acoustic image data that represents an acoustic signal that is greater than a predetermined frequency (eg, 10 kHz). In another example, the user wants to analyze a particular object that generally emits an acoustic signal within a particular range, such as a corona discharge from a transmission line (eg, as shown in FIGS. 5A-5D). There is. In such an example, the user may select a specific frequency range (eg, 11 kHz to 14 kHz for corona discharge) for acoustic imaging.

In some examples, an acoustic analysis system may be used to analyze and / or present information related to the strength of the received acoustic signal. For example, in one embodiment, backpropagating a received acoustic signal can include determining acoustic intensity values at multiple locations within the acoustic scene. In one example, similar to the frequencies mentioned above, the acoustic image data is included in the display image only if the intensity of the acoustic signal meets one or more predetermined requirements.

In various such embodiments, the display image is an acoustic signal above a predetermined threshold (eg, 15 dB), below a predetermined threshold (eg, 100 dB), or within a predetermined intensity range (eg, 100 dB). , 15 dB to 40 dB) can include acoustic image data. In certain embodiments, the threshold can be based on a statistical analysis of the acoustic scene, such as above or below the standard deviation from the average acoustic intensity.

Similar to the above with respect to frequency information, in certain embodiments, limiting the acoustic image data to represent an acoustic signal that meets one or more intensity requirements means that only the received signal that meets the predetermined conditions will have the acoustic image data. It can include filtering the received acoustic signal as used to generate it. In another example, the acoustic image data is filtered to adjust which acoustic image data is displayed.

Moreover or separately, in certain embodiments (eg, in combination with a video-acoustic image representation or through background analysis without necessarily updating the display image), the acoustic intensity at a position within the acoustic scene. Can be monitored over time. In one such example, certain requirements for displaying acoustic image data may include the amount or rate of change in acoustic intensity at a location in the image.

10A and 10B are exemplary display images containing combined visible light image data and acoustic image data. FIG. 10A shows a display image containing acoustic image data shown at multiple positions 1010, 1020, 1030, 1040, 1050, 1060, 1070, 1080, and 1090. In one example, the intensity value can be toned, for example, the acoustic intensity value is assigned a color based on a predetermined toning technique. In an exemplary embodiment, the intensity values can be classified according to the intensity range (eg, 10 dB to 20 dB, 20 dB to 30 dB, etc.). Each intensity range can be associated with a particular color according to toning techniques. The acoustic image data can include multiple pixels, and each pixel is colored with a color associated with the intensity range that contains the intensity represented by the pixels of the acoustic image data. In addition to or instead of being distinguished by color, different intensities can be distinguished according to other properties such as transparency (eg, in an image overlay where acoustic image data is overlaid with other image data). ..

Other parameters, such as the rate of change in acoustic intensity, may also be toned. Similar to the intensity, the change in the rate of change in acoustic intensity can be toned so that parts of the scene showing different rates and / or amounts of change in acoustic intensity are displayed in different colors.

In the illustrated example, the acoustic image data is toned according to the intensity palette, so that the acoustic image data representing different acoustic signal intensities is shown in different colors and / or shading. For example, the acoustic image data at positions 1010 and 1030 show a toned representation of the first intensity, positions 1040, 1060, and 1080 show a toned representation of the second intensity, and positions 1020. 1050, 1070, and 1090 indicate a toned representation of the third intensity. As shown in the exemplary representation of FIG. 10A, each position showing the toned representation of the acoustic image data shows a circular pattern with a color gradation extending outward from the center. This may be due to the attenuation of the acoustic intensity as the signal propagates from the source of the acoustic signal.

In the example of FIG. 10A, the acoustic image data is combined with the visible light image data to generate a display image, which can be presented to the user, for example, via a display. The user can see the display image of FIG. 10A to see which position in the visible scene is producing the acoustic signal and the intensity of such signal. Therefore, the user can quickly and easily observe which position is producing the sound and compare the intensities of the sounds coming from different positions in the scene.

Similar to those described with respect to frequency elsewhere herein, in certain embodiments, acoustic image data may only be presented if the corresponding acoustic signal meets certain intensity conditions. FIG. 10B shows an exemplary display image similar to the display image of FIG. 10A, and includes visible light image data and an acoustic image representing an acoustic signal that exceeds a predetermined threshold. As shown, of the positions 1010, 1020, 1030, 1040, 1050, 1060, 1070, 1080, and 1090 of FIG. 10A containing the acoustic image data, only the positions 1020, 1050, 1070, and 1090 are predetermined. It contains acoustic image data representing an acoustic signal that satisfies the conditions.

In an exemplary scenario, FIG. 10A can contain all acoustic image data above the noise floor threshold at each of positions 110-990, while FIG. 10B shows the same scene as FIG. 10A, but with 40 dB. Only acoustic image data with greater intensity is shown. This helps the user identify which sound source in the environment (eg, in the target scene of FIGS. 10A and 10B) contributes to a particular sound (eg, the loudest sound in the scene). Enables.

In addition to, or instead, in direct comparison with an intensity threshold (eg, 40 dB), as described elsewhere herein, to display acoustic image data in some such examples. Certain requirements may include the amount or rate of change in acoustic intensity at a location in the image. In one such example, the acoustic image data is within a predetermined range, where the rate of change or amount of change in acoustic intensity at a given position meets certain conditions (eg, greater than or equal to the threshold, less than or equal to the threshold). Etc.) can only be presented. In certain embodiments, the amount or rate of change in acoustic intensity can be toned and displayed as intensity acoustic image data or in combination with intensity acoustic image data. For example, in an exemplary embodiment, where the rate of change is used as a threshold for determining the position containing the acoustic image data, the acoustic image data includes a toned intensity rate of change metric for display. be able to.

In one example, the user may manually set the intensity requirements for the displayed acoustic image data (eg, minimum, maximum, range, rate of change, amount of change, etc.). As discussed elsewhere here, including only acoustic image data that meets this intensity requirement can be achieved during acoustic image data generation (eg, by filtering the received acoustic signal). , And / or by not displaying the generated acoustic image data representing the acoustic signal that does not meet the set requirements (s). In one such example, filtering the display image according to the intensity value may be performed after the acoustic and visible light image data have been captured and stored in memory. That is, the data stored in memory can be used to generate a display image that includes any number of filtering parameters, for example displaying only acoustic image data that satisfies a predefined intensity condition. ..

In one example, setting a lower bound on the intensity of an acoustic image (eg, displaying only acoustic image data that represents an acoustic signal that exceeds a given intensity) is an undesired background or ambient sound from the acoustic image data. / Or the inclusion of sound reflections can be excluded. In another example, setting an upper bound on the intensity of an acoustic image (eg, displaying only acoustic image data representing an acoustic signal below a given intensity) observes an acoustic signal that is normally hidden by loud noise. Therefore, it is possible to exclude the inclusion of such loud sounds expected in the acoustic image data.

Several display functions are possible. For example, similar to the frequency analysis / display described with respect to FIG. 9C, in one example, the acoustic analysis system may cyclically display a plurality of display images, each showing acoustic image data satisfying different intensity requirements. Similarly, in one example, the user may scroll through a set of acoustic intensity ranges to see positions within acoustic image data that have a given range of acoustic intensity.

Another parameter that can be used to analyze acoustic data is the periodicity value of the acoustic signal. 11A and 11B show exemplary plots of frequency vs. time of acoustic data in an acoustic scene. As shown in the plot of FIG. 11A, the acoustic data is a frequency X signal having a first periodicity, a frequency Y signal having a second periodicity, and a frequency Z signal having a third periodicity. including. In the illustrated example, acoustic signals with different frequencies may also contain different periodicities in the acoustic signal.

In one such example, the acoustic signal can be filtered based on periodicity in addition to or instead of frequency content. For example, in one example, multiple sources of an acoustic signal in an acoustic scene may generate an acoustic signal at a particular frequency. If the user wants to isolate one of such sources for acoustic imaging, the user may choose to include or exclude the acoustic image data in the final display image based on the periodicity associated with the acoustic data. ..

FIG. 11B shows a plot of the frequency vs. time of the acoustic signal. As shown, the frequency increases almost linearly over time. However, as shown, the signal contains a nearly constant periodicity over time. Therefore, such signals may or may not appear in the acoustic image, depending on the display parameters selected. For example, the signal may meet the frequency reference displayed at one point in time, but may be outside the displayed frequency range at another point in time. However, the user can choose to include or exclude such signals from the acoustic image data, based on the periodicity of the signal, regardless of the frequency content.

In one example, extracting an acoustic signal of a particular periodicity can help analyze a particular part of the target scene (eg, a particular device or type of device that normally operates with a particular periodicity). For example, if an object with a target operates at a specific periodicity (eg, once per second), excluding signals with a different periodicity can improve the acoustic analysis of the target object. For example, referring to FIG. 11B, if a target object operates with periodicity 4, separating the signal with periodicity 4 for analysis results in an improved analysis of the target object. There is. For example, an object with a target may have a periodicity of 4, but may emit a sound with increasing frequency, as shown in FIG. 11B. This may mean that the properties of the object may have changed (eg, increased torque or load) and should be inspected.

In an exemplary acoustic imaging process, background noise (eg, floor noise in an industrial environment, wind in an outdoor environment, etc.) is generally not periodic, but certain objects with targets in the scene are periodic. Emit an acoustic signal (eg, a machine that operates at regular intervals). Therefore, the user may choose to exclude the aperiodic acoustic signal from the acoustic image in order to remove the background signal and present the target acoustic data more clearly. In another example, the user may try to find the source of a constant tone and therefore choose to exclude the periodic signal from the acoustic image data which may obscure the display of the constant tone. Can be. In general, the user may choose to include the acoustic signal in the acoustic image data above the specific periodicity, below the specific periodicity, or within the desired range of periodicity. In various examples, periodicity can be specified either by the length of time between the periodic signals or by the frequency of occurrence of the periodic signals. Similar to the frequency shown in FIG. 11B, an analysis of intensity at a given periodicity (eg, with a target object operating at that periodicity) shows how the acoustic signal from the object changes over time. May be used to track the frequency. In general, in certain embodiments, periodicity can be used to perform rate of change analysis of various parameters such as frequency, intensity and the like.

As described elsewhere herein, in one example, different parts of the target scene can be associated with different distances from the acoustic imaging sensor. For example, in certain embodiments, the distance information can include three-dimensional depth information about various parts of the scene. On top of that, or separately, the user may be able to measure distance values associated with multiple locations in the scene (eg, with a laser distance tool) or manually enter them. In one example, such different distance values for different parts of the scene can be used to adjust the backpropagation calculation at such a position to accommodate a particular distance value at that position. ..

On top of that, or separately, if different parts of the scene are associated with different distance values, the proximity from the acoustic sensor array (eg, measured proximity and / or apparent proximity) is such. It can be another distinguishable parameter between the parts. For example, with respect to FIG. 10B, positions 1020, 1050, 1070, and 1090 are each associated with different distance values. In one example, the user may select a particular range of distance to include the acoustic image data in the display, similar to the frequency or periodicity described elsewhere here. For example, the user may choose to display only acoustic image data that represents an acoustic signal that is closer than a predetermined distance, farther than a predetermined distance, or within a predetermined distance range.

Further, in one embodiment, as described for the frequency of FIG. 9C, the acoustic analysis system cyclically displays multiple distance ranges and emits sound from a position in the target scene that meets the current distance range. It may be configured to show only acoustic image data representing a signal. Such a circular display of the various displays can help the user visually distinguish the information between different acoustic signals. For example, in some cases, objects may appear to be in close proximity to each other from the line of sight from the associated electromagnetic imaging tool (eg, a visible light camera module) and are therefore combined with electromagnetic image data of such objects. It is difficult to distinguish the acoustic image data. However, if the objects are separated by a depth difference, a cyclic display of different depth ranges of the acoustic image data can be used to separate each source of acoustic data from other sources.

In general, an acoustic analysis system may be configured to apply various settings to include and / or exclude acoustic image data representing acoustic signals that satisfy one or more predefined parameters. In one example, an acoustic analysis system is used to select multiple conditions that must be met by an acoustic signal, eg, for acoustic image data indicating that the acoustic signal is displayed in a display image. Can be done.

For example, for FIGS. 10A and 10B, only acoustic signals that exceed the threshold intensity in the scene of FIG. 10A are shown in FIG. 10B. However, additional or alternative restrictions are possible. For example, in one embodiment, the user further filters the acoustic image data so that the acoustic image data has frequency content within a predetermined frequency range and / or only for an acoustic signal having a predetermined periodicity. Can be shown. In an exemplary embodiment, acoustic image data may be excluded from additional locations such as 1020 and 1090, limited to a given frequency and / or target periodicity.

In general, users use intensity, frequency, periodicity, apparent proximity, measured proximity, sound pressure, particle velocity, particle displacement, acoustic output, sound energy, sound energy density, acoustic exposure, pitch, amplitude, Any number of acoustic data requirements can be applied to include or exclude acoustic image data, including parameters such as brightness, harmonics, rate of change of such parameters. Further, in certain embodiments, the user may combine the requirements using any suitable logical combination such as AND, OR, XOR and the like. For example, a user may want to display only acoustic signals having an AND (frequency within a predetermined range) (intensity exceeding a predetermined threshold).

On or apart from that, the acoustic analysis system is configured to cyclically display one or more parameter ranges to indicate different parts of the target scene, as shown for the cyclic display of multiple frequencies in FIG. 9C. Can be done. In general, one or more parameters can be cyclically displayed in this way. For example, the parameter (eg, intensity) is divided into multiple ranges (eg, 10 dB to 20 dB and 20 dB to 30 dB), and the acoustic analysis system can cyclically display such ranges, and all acoustic images in the first range. Display the data and then display all the acoustic image data in the second range.

Similarly, in certain embodiments, the acoustic analysis system may be configured to combine parameter requirements by cyclically displaying nested ranges. For example, in an exemplary embodiment, acoustic image data satisfying the first intensity range AND the first frequency range may be displayed. The displayed frequency range may be cyclically displayed while limiting the displayed acoustic image data to acoustic signals that satisfy the first intensity range. After the frequency range is cyclically displayed, the intensity range can be updated to the second intensity range so that the displayed acoustic image data fills the second intensity range and the first frequency range. Similar to the process of incorporating the first intensity range, the frequency range may be similarly cyclically displayed while maintaining the second intensity range. This process can continue until all combinations of frequency and intensity ranges have been met. A similar such process can be run for any of multiple parameters.

Moreover or separately, in certain embodiments, the acoustic analysis system may be configured to identify and distinguish multiple sounds in an acoustic scene. For example, with respect to FIG. 9B, the acoustic analysis system may be configured to identify four individual sounds at positions 910, 920, 930, and 940. The system is configured to cycle through multiple displays showing acoustic image data at a single individual location, although each does not necessarily depend on parameter values, as shown in FIG. 9C. be able to. Similarly, such a cyclic display between acoustic image data at individual locations can be performed after limiting the acoustic image data displaying one or more parameter requirements.

For example, with respect to FIGS. 10A and 10B, before the intensity threshold is applied, the acoustic analysis system contains acoustic image data in which each scene is in a single position (eg, an acoustic image scene with visible light image data). Multiple acoustic image scenes (as display images including) may be cyclically displayed. In one embodiment, according to the illustrated example of FIG. 10A, there is a circular representation of 10 separate images, where each image is in position 1010, 1020, 1030, 1040, 1050, 1060, 1070, 1080, and 1090. Contains image data from one of the different ones. However, according to certain embodiments, after the intensity filter has been applied so that only positions with an intensity above the threshold are displayed (eg, as in FIG. 10B), the acoustic analysis system sets the threshold for filtering. The cyclic display process may be updated to cycle display only the images corresponding to the filled positions. That is, with respect to FIG. 10B, the cyclic display process may be updated to cyclically display only between four images, each showing separate acoustic image data at positions 1020, 1050, 1070, and 1090.

Thus, in various embodiments, each position in the target scene containing the acoustic image data, either before or after applying one or more filters to limit which acoustic image data is displayed. Is shown in one of a plurality of cyclic display images. Such a circular display of individual sound source locations can assist the user viewing the image in identifying a particular sound source. In some embodiments, each image in the circulation contains only a single source of acoustic data, and in such embodiments, one of the acoustic data such as frequency content, intensity, periodicity, apparent accessibility, etc. Or more parameters are included.

In addition to, or instead of, circularly displaying between images showing acoustic image data that meet certain conditions, in one example, the location of the acoustic signal source is detected in the acoustic image data and the acoustic image is separate from the other acoustic signals. Can be displayed in the data. For example, with respect to FIG. 10A, in certain embodiments, acoustic image data representing acoustic signals generated from each of the positions 110-990 may be identified and cyclically displayed. For example, in an exemplary operating process, a display image containing acoustic image data at one of positions 110-990 is automatically or at the user's direction for individual analysis of each source of the acoustic signal. Can be circularly displayed. In various embodiments, the order in which different positions of the acoustic image data are displayed during the cyclic display can depend on various parameters such as position, proximity, intensity, frequency content, and the like.

Above or separately, in one example, the acoustic image data from individual positions is circulated after applying one or more filters to separate only the acoustic image data that meets one or more predetermined conditions. Can be displayed. For example, with respect to FIG. 10B, positions 1020, 1050, 1070, and 1090 are shown as including acoustic image data representing acoustic signals that meet predetermined intensity requirements. In certain embodiments, such display requirements can be applied to the individual cyclic display of the source location of the acoustic signal. For example, further referring to FIG. 10B, a display image containing image data from only one of positions 1020, 1050, 1070, and 1090 satisfying the acoustic intensity condition is cycled for individual analysis at each position. Can be displayed.

In an exemplary process with reference to FIGS. 10A and 10B, acoustic image data collected from the scene is generally shown in FIG. 10A at positions 1010, 1020, 1030, 1040, 1050, 1060, 1070, 1080, and 1090. obtain. Such locations may include acoustic image data representing acoustic signals with different acoustic parameters such as different intensities, frequency content, periodicity, and so on.

As described elsewhere herein, the user may wish to separate acoustic signals with one or more specific acoustic parameters, such as acoustic signals with minimal acoustic intensity. Acoustic image data representing acoustic signals that do not meet such conditions can be excluded from the image and, for example, discard the acoustic image data at positions 1020, 1050, 1070, and 1090 as shown in FIG. 10B. Can be done. However, the user may want to further identify a particular sound source that meets the display conditions (eg, having an intensity above a threshold). Thus, the user may choose to view the acoustic image data associated with positions 1020, 1050, 1070, and 1090 one by one in order to see the location of each sound source and analyze them individually. .. In various embodiments, the user may choose to manually cycle through such positions, or the processor automatically updates the display image to provide acoustic image data for individual positions. It may be displayed sequentially. This is not a goal, but may help the user to further eliminate and ignore acoustic signals that accidentally satisfy one or more filtering parameters applied to the image.

Although described with respect to intensity and FIGS. 10A and 10B, display images containing acoustic image data from a single position selected from multiple positions are generally cyclically displayed one by one for individual analysis. Can be done. The plurality of positions containing typical acoustic image data are the entire set of positions corresponding to the source of the acoustic signal in the acoustic scene, or have, for example, an acoustic signal satisfying one or more conditions. It can be a subset of such positions, including only the positions. Such conditions may depend on any one or more acoustic parameters such as intensity, frequency content, periodicity, accessibility, etc., and may be less than a predetermined value, greater than a predetermined value, or a predetermined value. May be satisfied based on various parameters within the range of.

In various examples, modifying the display image to selectively include acoustic image data in the display image can be done in various ways. In certain embodiments, the display image (including, for example, electromagnetic image data and acoustic image data) can be a real-time image, in which case the electromagnetic image data and the acoustic image data are such to reflect changes in the scene. It will be updated continuously. In one example, when certain conditions are used to determine if acoustic image data is included in the display image, the received acoustic signal is analyzed and the acoustic image at various locations in the updated real-time image. Determine if it contains data. That is, when a new display image is generated based on the newly received acoustic signal and electromagnetic radiation, the construction of the display image sets the specific conditions (eg, intensity threshold, etc.) that the acoustic signal is recognized in the display image. You can rely on the analysis of the acoustic signal to determine if it meets. Display images can then be generated containing acoustic image data only where appropriate according to such conditions.

In another example, the display image may be generated from data stored in memory, such as previously captured acoustic data and electromagnetic image data. In one such example, previously acquired acoustic data is analyzed for various conditions found in the acoustic image data, and the previously captured acoustic data is combined with the electromagnetic image data at positions that meet such conditions. Is done. In such embodiments, a single scene can be viewed in many ways, for example by analyzing different acoustic parameters. The display image representing the previously captured acoustic image data can be updated based on the updated conditions recognized in the display image as to whether the acoustic image data is included at various positions in the display image.

In certain embodiments, one or more acoustic parameters used to selectively include acoustic image data in a display image may be used to modify the display image and / or image capture technique. For example, in a real-time imaging example, various conditions for deciding whether to include acoustic image data in the display may include distance to the target (eg, apparent distance or measured distance) and / or frequency content. .. As described elsewhere herein, certain such parameters can be used in selecting acoustic sensor arrays and / or processing techniques for producing acoustic image data. Thus, in one such example, to generate an acoustic sensor array and / or acoustic image data if the acoustic image data is represented only based on such parameters that meet one or more predetermined conditions. The processing technique may be selected based on such conditions.

For example, in an exemplary embodiment, the acoustic image data is best suited for the first frequency range if the corresponding acoustic signal should only be included in a real-time image at a location containing frequency content within the first frequency range. One or more acoustic sensor arrays may be selected to obtain a good acoustic signal. Similarly, if the acoustic image data should only be included in the real-time image at a location where the source of the acoustic signal is within the first range of distance, then the best acoustic signal for acoustic imaging in the first range of distance. One or more acoustic sensor arrays may be selected to obtain. Moreover or separately, the processing technique for generating the acoustic image data can be selected based on the desired frequency or distance condition, for example, as described with respect to FIG. Such selected acoustic imaging sensor arrays and processing techniques are then used to receive the acoustic signal and generate the acoustic image data of the updated real-time display image in order to optimize the contained acoustic image data. Can be used.

Similarly, in certain embodiments where the display image is generated from historical data previously stored in memory, various conditions that determine where the acoustic image data is included in the display image update the acoustic image data representing the acoustic scene. Can be used to For example, in one embodiment, the data stored in memory includes raw acoustic data received by the acoustic sensor array from the time the acoustic signal is received. Acoustic image data optimized for the desired parameters to display, based on conditions (eg, desired distance and / or frequency range) for determining whether the acoustic image data is contained at various locations in the display image. Processing techniques (eg, backpropagation algorithms) may be selected for use with raw data stored in memory to generate.

Although generally described and shown using visible light and acoustic image data, the process described for FIGS. 9A-9C, 10A and 10B captures any of a variety of electromagnetic image data. It will be understood that it can be used inclusively. For example, in various embodiments, similar processes can be performed using infrared or ultraviolet image data instead of visible light image data. Moreover or separately, combinations of electromagnetic spectra can be used in such processes such as mixed infrared and visible light image data. In general, in various examples, acoustic image data is selectively shown in combination with any combination of electromagnetic image data (eg, included when the corresponding acoustic signal meets one or more predetermined parameters. )Sometimes.

In certain embodiments, the acoustic analysis system stores one or more acoustic signals and / or acoustic image data in, for example, in local memory and / or in a database accessible from an external or remote device. It is composed. Such acoustic signals may include acoustic image data representing the acoustic scene during normal operation and / or other parameters related to the acoustic scene such as frequency data, intensity data, periodicity data. In various examples, the database scene is an acoustic image data representing a wide scene (eg, a factory) and / or a more specific scene (eg, a particular object) and / or other acoustic parameters (eg, intensity, frequency, etc.). Periodicity, etc.) can be included.

In certain embodiments, the database scene can be generic to a particular type of device, such as a particular model of device. On top of that, or separately, database scenes are specific to individual objects, even if different objects are different examples of the same object (eg, two separate machines of the same model). obtain. Similarly, a database scene can be more specific 8spec), for example, including certain operating states of the object. For example, if a particular object has multiple modes of operation, the database may include multiple scenes of such objects, one for each mode of operation.

In various embodiments, the database scene can be a single acoustic image and / or associated acoustic parameters. In another example, the database scene can contain composite data formed from multiple previously captured acoustic images and / or related parameters. In general, a database scene (eg, an acoustic image and / or a parameter) can include an acoustic representation of a normal operating scene. In one example, the database can include other elements related to the scene, such as, for example, the corresponding visible light image, infrared image, ultraviolet image, or a combination thereof. In one embodiment, the generation and / or comparison of the database is delegated to the transferee of this application, which was filed on June 23, 2016, which is incorporated herein by reference in its entirety, and "THERMAL". It can be performed in the same manner as the database generation and comparison of infrared image data described in US Patent Application No. 15 / 190,792 entitled "ANOMALY RECORDION)". In certain embodiments, the database captures the acoustic image data of the scene and / or one or more related acoustic parameters (eg, frequency, intensity, periodicity, etc.) while the object in the scene is operating correctly. It can be generated by capturing. In one such example, the user may tag the captured database image and associate the image with one or more objects, positions, scenes, etc., resulting in the captured acoustic image and /. Or related parameters (s) may be identified in future database analysis and comparison.

The newly generated acoustic image data can be compared with the acoustic image data stored in the database to determine whether the sound profile of the acoustic scene is within typical operating criteria. On top of that or separately, acoustic parameters such as intensity, frequency, periodicity, etc. from live acoustic scenes and / or newly generated acoustic images may be compared to similar parameters in the database.

Current acoustic image data with acoustic image history data stored in a database (eg, previously captured images, composite images generated from multiple previously captured images, expected factory-provided images, etc.) The comparison can be done in multiple ways. 12A-12C show a plurality of exemplary methods for comparing acoustic image data with acoustic image history data stored in a database. FIG. 12A shows an acoustic imaging tool 1200 including an acoustic sensor array 1202 with an acoustic field of view 1212 and an electromagnetic imaging tool 1204 with an electromagnetic field of view 1214. As shown, the electromagnetic field of view 1214 and the acoustic field of view 1212 include a target scene 1220 that includes a target object 1222. In one embodiment, the acoustic imaging tool 1200 is permanently fixed in a position such that the target object 1222 is within the electromagnetic field of view 1214 and the acoustic field of view 1212. In certain embodiments, the acoustic imaging tool 1200 can be powered via inductive or parasitic power, wired to AC main power within the building, or to continuously monitor the object 1222. Can be configured.

The fixed acoustic imaging tool 1200 may be configured to periodically capture the acoustic and / or electromagnetic image data of the object 1222. Since the acoustic imaging tool 1200 is fixed in a substantially fixed position, the images captured at different times are from approximately the same point. In one example, the acoustic image data captured via the acoustic imaging tool 1200 may be compared to a database of acoustic image data representing approximately the same scene, for example, to detect anomalies in the acoustic scene. This can be done, for example, as described in US Patent Application No. 15 / 190,792, which is incorporated by reference.

FIG. 12B shows, for example, an exemplary display on a handheld acoustic imaging tool. The display 1230 includes two sections, 1232 and 1234. In the illustrated example, section 1234 shows a database image 1244 of the target object, while section 1232 contains a live image 1242 of real-time acoustic image data of the object. In such a parallel display, the user compares the live image 1242 with the database image 1244 to see the difference between the typical acoustic signal (eg, shown in the database image 1244) and the current real-time image 1242. can do. Similarly, the user can compare whether the live image 1242 closely matches the database image 1244. If so, the user can capture the live acoustic image for further analysis and / or comparison with the database image 1244.

FIG. 12C shows, for example, another exemplary display on a handheld acoustic imaging tool. Display 1250 in FIG. 12C shows database image 1254 and live image 1256 on the same display 1252. In the example of FIG. 12C, the user can compare the acoustic image data in the live image 1256 as well as the acoustic image data in the database image 1254 to show the difference. In addition, the user may adjust the alignment of the acoustic imaging tool to align the object in the live image 1256 with the object in the database image 1254 for further analysis and comparison.

As a result of the process of FIGS. 12A-12C, live and / or recently captured acoustic images may be compared with previous acoustic image data from a database or the like. In one example, such a process is used to overlay live and / or recently captured acoustic images with database images for automatic comparison. Other processes that can be used to "recapture" acoustic image data from points similar to database images were filed on 20 December 2011 and are "THERMAL IMAGING CAMERA FOR INFRANED". US Patent Application No. 13 / 331,633, entitled "REPHOTOGRAPHY", filed December 20, 2011, entitled "THERMAL IMAGING CAMERA FOR INFLATED REPHOTOGRAPHY". US Patent Application No. 13 / 331,644, as well as US Patent Application No. 13 entitled "THERMAL IMAGING CAMERA FOR INFORMATION REPHOTOGRAPHY" filed on December 23, 2011. / 336,607, each of which has been transferred to the transferee of the present application, which is incorporated by reference in its entirety.

Comparing real-time acoustic image data and / or acoustic signatures with the corresponding acoustic images and / or acoustic signatures of comparable scenes / objects provides a quick and simplified analysis of the operating state of the scene / object. May be used to. For example, comparisons may indicate that a particular location in the acoustic scene emits an acoustic signal with a different intensity or frequency spectrum than during typical operation, which indicates a problem. Sometimes. Similarly, locations within the scene may typically emit silent acoustic signals. Moreover or separately, a comparison of the overall acoustic signatures of the live scene and the historical scene from the database may generally indicate changes in acoustic parameters within the scene such as frequency content, acoustic intensity, etc.

In one example, the acoustic analysis system is configured to compare recent / real-time acoustic scenes with a database. In one embodiment, the acoustic analysis system is configured to characterize the differences between recent / real-time scenes and database scenes and diagnose one or more possible problems in the current scene based on comparisons. .. For example, in one embodiment, the user may preselect a target object or target scene for comparison with an acoustic database. The acoustic analysis system can analyze the scene by comparing the database image and / or other parameters with the recent / current image and / or other parameters based on the selected object / scene. Based on the object / scene selected from the database, the acoustic analysis system identifies one or more differences between the database image / parameter and the recent / current image / parameter, and the identified difference (s). ) Can be associated with one or more possible causes of difference.

In one example, the acoustic analysis system can be pre-programmed with multiple diagnostic information, eg, various differences between database images / parameters and recent / current images / parameters, possible causes. And / or associate with the solution of the cause. On top of that or separately, the user may load such diagnostic information from, for example, a repository of diagnostic data. Such data may be provided, for example, by the manufacturer of the acoustic analysis system, the manufacturer of the target object, and the like. In yet another example, the acoustic analysis system can learn diagnostic information on or off the machine learning process, eg, via one or more machine learning processes. In one such example, the user observes the acoustic deviation of the scene from a typical scene, then diagnoses one or more problems in the target scene, one or more problems and / or one or more. Data representing the above solutions can be input to the acoustic analysis system. Over time, the system resolves various discrepancies between recent / current images and / or parameters and those stored in the database, through multiple data entries, specific problems and / or. It can be configured to learn to associate with a strategy. After diagnosing the problem and / or determining the proposed solution, the acoustic analysis system may be configured to output the suspicious problem and / or the proposed solution to the user, for example, via a display. .. Such a display can be placed on a handheld acoustic inspection tool or remote device (eg, a user's smartphone, tablet, computer, etc.). On top of that or separately, such displays showing potential problems and / or solutions can communicate with remote sites, such as offsite operators / system monitors, over a network, for example.

In some diagnostic characteristics, the acoustic analysis system may observe certain periodic squeaks that indicate that the machine in operation requires additional lubrication. Similarly, a constant treble signal may indicate a gas or air leak in the target scene. Other problems as well may have recognizable acoustic signatures, such as bearing breakage in the object being analyzed, and when viewed through an acoustic imaging system (such as a handheld acoustic imaging tool), the system Or it may be useful in diagnosing abnormalities in objects.

An acoustic analysis system capable of comparing the received acoustic signal with a reference (eg, acoustic image data and / or parameters from a database), performing diagnostic information, and / or proposing corrective action is the acoustic data of the scene. Can be eliminated from the need for experienced professionals to analyze. Rather, acoustic inspection and analysis can be performed by a system operator with limited or no experience in analyzing acoustic data.

FIG. 13 is a process flow diagram showing an exemplary operation of comparing received acoustic image data with a database for object diagnosis. The method comprises receiving a selection of a target target (1380) and obtaining a reference acoustic image and / or acoustic parameters of the target target from a database (1382). For example, a user may wish to perform an acoustic analysis of a particular object of 100 million votes, from a predefined list of objects with reference acoustic images available and / or parameters available in the database. You can choose such an object.

The method further comprises the step (1384) of capturing acoustic image data representing a target target and related parameters using, for example, a handheld acoustic imaging tool. After capturing the acoustic image data and related parameters (1384), the method compares the captured acoustic image data and / or related parameters with the acquired reference image and / or parameters (1386). include.

The method of FIG. 13 further diagnoses behavioral problems of the target of interest based on comparisons (1388) when the captured acoustic image data and / or parameters deviate sufficiently from the criteria (1388). include. The method may further include displaying to the user an indication of possible problems and / or corrective actions (1392). In certain embodiments, a comparative display, eg, a differential image showing the difference between the current acoustic image data and the reference acoustic image data, may be displayed to the user on or separately from it.

In one such example, determining if there is a deviation from the reference (1388) is to compare the acoustic parameters of one or more of the captured data with similar parameters in the reference data and capture. It involves determining whether the difference between a given parameter and a reference parameter exceeds a predetermined threshold. In various examples, different parameters may include different thresholds, such thresholds can be absolute thresholds, statistical thresholds, and the like. In certain embodiments, the comparison can be made position by position and can be performed on a subset of positions in the scene.

For example, with respect to FIG. 9B, only the positions that include acoustic image data and appear on the object (eg, positions 910 and 940) can be analyzed with respect to the movement of the object. In such an example, different acoustic parameters at each position to be compared (eg, 910 and 940) are compared individually between the captured image and the database image. For example, comparing the captured data and / or related parameters with those from the database can refer to FIG. 9B for the frequency, intensity, and periodicity of position 910 in the captured image, respectively. Comparison with the frequency, intensity, and periodicity of position 910 in the image can be included. A similar comparison can be performed at position 940 between the captured image and the database image. As described, each comparison can include different metrics to determine if there is sufficient deviation from the criteria (1388).

Diagnosis of operational problems (1390) and display of possible problems and / or corrective action instructions (1392) are based on a combination of comparisons between captured image data and reference image data and / or parameters. Can be executed. In one example, such a diagnosis can include multidimensional analysis, such as combining comparisons of multiple parameters at a given position. For example, in an exemplary embodiment, a particular condition may be indicated by both a frequency deviation from a reference greater than the first threshold and an intensity deviation from a reference greater than the second threshold.

In one example, the process involves incorporating new acoustic image data and related parameters (1384) and repeating the comparison and diagnostic process, even after displaying possible problems and / or instructions for corrective action (1392). There is. Therefore, the user is whether the corrective action taken is effectively modifying the acoustic signature of the object to correct the identified problem and / or is it consistent with the acoustic signature of the object. Can be observed.

In one embodiment, after comparing the captured data with the reference data (1386), if there is no sufficient deviation from the reference (1388), the process is based on the object's current acoustic signature and the object is normal. Can be terminated with the conclusion that it is working (1394). On top of that or separately, new acoustic image data and related parameters of the target target can be captured (1384) and the comparison and diagnostic process can be repeated. In one example, continuous iterative analysis may be performed using a fixed acoustic analysis system, including, for example, the acoustic imaging tool 1200 of FIG. 12A.

Comparing acoustic data (eg, image data and / or other acoustic parameters) can help the user more easily identify whether an object is functioning properly and, if not, diagnose problems with the object. In one example, the comparison with the criteria allows the user to ignore "normal" sounds in the scene, such as expected operating sounds and floor / background sounds that may be unrelated to object manipulation problems. Useful.

During operation, observation of acoustic image data and / or associated acoustic parameters, or observation of the results of a comparison between the current acoustic scene and the acoustic scene in the database, may indicate the target position to the user for further inspection. For example, a comparative acoustic image showing deviations from a database image may indicate one or more positions in a scene that is behaving abnormally. Similarly, viewing an acoustic image with an acoustic signature at one or more unexpected positions can indicate the target position to the user. For example, referring to FIG. 10B, a user observing FIG. 10B on the display of an acoustic imaging system may notice that a particular location (eg, 1020) emits an unexpected acoustic signal. , Or similarly, the comparison with the reference image indicates unexpected parameters of the acoustic signal at that location (eg, unexpected frequency, intensity, etc.).

In one such example, the user may approach such a position in order to more closely check the position for anomalies. As you approach the object, the distance value to the target may be updated to reflect the new distance between the acoustic array and the target position. The acoustic sensor array and / or backpropagation algorithm can be updated based on the updated distance to the target. On top of that, or separately, updated acoustic analysis from a closer location may result in a different analysis of the acoustic signal from the target. For example, high frequency acoustic signals (eg, ultrasonic signals) tend to be attenuated over a relatively short distance from the source of the acoustic signal. Therefore, additional signals (eg, high frequency signals) may be visible to the acoustic sensor array as the user approaches the target for further inspection. Such obvious changes in observable scenes may also result in adjustments to the acoustic sensor array and / or backpropagation algorithm used for acoustic imaging.

Therefore, the sensor array and / or backpropagation algorithm used for acoustic imaging may be updated one or more times as the user approaches the target object or region. Each update can use a different sensor array and / or backpropagation algorithm to provide additional details about a target object or region that may not have been observable from a distance. For example, closer to a target object or region based on early observations of a wider scene can also increase the acoustic intensity of the target acoustic signal compared to the background sound in the environment.

In certain embodiments, an acoustic analysis system (eg, a handheld acoustic imaging tool) may prompt the user to move closer to a target object or region in the scene. For example, comparing the current acoustic image with the reference database image, the acoustic analysis system can identify one or more positions in the scene that deviate from the reference. The acoustic analysis system may, for example, highlight such one or more positions to the user via a display and suggest that the user approach the specified position for further analysis. In one example, an acoustic analysis system can classify a particular location, such as a sub-component of an object in the environment or a particular object, as having its own reference profile stored in a database. The system may be configured to propose and / or implement such a profile of the classified locations to facilitate further analysis of the identified locations as the user approaches for additional inspection.

The systems and processes described herein can be used to improve the speed, efficiency, accuracy, and thoroughness of acoustic inspections. Various automated actions and / or suggestions (eg, sensor arrays, backpropagation algorithms, etc.) make inspection easier in that inexperienced users can perform a thorough acoustic inspection of the acoustic scene. Allows for improvement. In addition, such processes can be used to analyze a wide range of scenes, including entire systems, individual objects, and subcomponents of individual objects. Predefinition and / or user-generated profiles of reference acoustic data for acoustic scenes can help even inexperienced users identify anomalies in captured acoustic data.

Superimposing acoustic image data on other data streams such as visible, infrared, and / or ultraviolet image data provides additional context and details to the object emitting the acoustic signal represented in the acoustic image data. Can be provided. Combining an acoustic sensor array with a distance measurement tool (such as a laser distance finder) can help the user in quickly and easily determining the distance value to the appropriate target for use in the acoustic imaging process. In various examples, acoustic sensor arrays, distance measurement tools, processors, memory, and one or more additional imaging tools (eg, visible light camera modules, infrared camera modules, etc.) provide efficient acoustics for multiple scenes. It may be supported by a single enclosure of handheld acoustic imaging tools that can provide analysis. Such handheld acoustic imaging tools may be moved from scene to scene in order to quickly analyze multiple target objects. Similarly, handheld tools allow the user to approach the curtain position in the scene for further inspection or analysis.

Various systems and methods for performing acoustic imaging and generating and displaying acoustic image data are described herein. An exemplary system can include an acoustic sensor array that includes a plurality of acoustic sensor elements configured to receive an acoustic signal from an acoustic scene and output acoustic data based on the received acoustic signal.

The system can include an electromagnetic imaging tool configured to receive electromagnetic radiation from the target scene and output electromagnetic image data representing the received electromagnetic radiation. Such imaging tools can include infrared imaging tools, visible light imaging tools, ultraviolet imaging tools, and the like, or a combination thereof.

The system can include a processor that communicates with an acoustic sensor array and an electromagnetic imaging tool. The processor may be configured to receive electromagnetic image data from an electromagnetic imaging tool and acoustic data from an acoustic sensor array. The processor may be configured to generate acoustic image data of the scene, for example, via backpropagation calculations, based on the received acoustic data and the received distance information representing the distance to the target. The acoustic image data can include a visual representation of the acoustic data, such as toning or color scheme, as described elsewhere herein.

The processor may be configured to combine the generated acoustic image data with the received electromagnetic image data to generate a display image containing both the acoustic image data and the electromagnetic image data and to communicate the display image to the display. The combination of the acoustic image data and the electromagnetic image data can include, for example, correcting a parallax error between the acoustic image data and the electromagnetic image data based on the received distance information.

In one example, distance information can be received from a distance measurement tool that communicates with the processor. The distance measuring tool can include, for example, an optical distance measuring device such as a laser distance measuring device and / or an acoustic distance measuring device. On top of that or separately, the user can manually enter distance information, for example via a user interface.

The system includes a laser pointer to help locate a target point such as a sound or sound profile based on selected parameters such as frequency, decibel level, periodicity, distance, etc., or a combination thereof. be able to. Such laser pointers may be used to accurately align and align the field of view of the scene with the appropriate sound visualization displayed on the display. This is useful in environments where the object under inspection is far from the acoustic imager, or when it is not clear where the sound visualization on the display is relative to the actual scene.

In one example, the laser pointer can be visualized on the display. Such visualization can include generating a laser pointer spot (eg, via a processor) on a display that represents a laser pointer in a real scene. In one example, the position of the laser pointer is highlighted on the display by, for example, an icon representing the laser pointer in the scene or another aligned display marker to better determine the position on the display with respect to the actual scene. Can be done.

As described elsewhere herein, thermal imaging systems are false colors (eg, toned) of acoustic data generated by one or more acoustic sensors, such as by creating acoustic image data. It can be configured to create a), symbol, or other non-numerical visual representation. Moreover or separately, the system can provide voice feedback to the user via speakers, headphones, wired or remote communication headsets and the like. The transmission of such audio or heterodyne audio can be synchronized with the visual representation of the detected and displayed sound.

In various examples, acoustic data is visualized in different ways, for example, to facilitate understanding of such data and prevent viewers from making false assumptions about the nature of the sound being visualized. Can be done. In one example, different types of visualizations make it possible to provide an intuitive understanding of the visualized sound.

In certain embodiments, the generated display includes a non-numerical visual representation with numerical and / or alphanumerical contextual data to provide a complete presentation of information about the sound being visualized. Allows users to help determine and / or implement one or more appropriate policies of conduct.

Various display features, including various non-numeric graphic representations (eg, symbols, tonings, etc.) and alphanumerical information, can be combined. In certain embodiments, the display features present in a given scene representation may be customized by the user, for example, from a plurality of selectable settings. Moreover or separately, a preset combination of display features can be selected by the user to automatically include the desired combination of information in the display image. In various embodiments, the aspect of the display image is user adjustable, for example, via a virtual interface (eg, provided via a touch screen) and / or physical control.

FIG. 14 shows the visualization of acoustic data using a gradation toning technique. As shown, acoustic parameters (eg, intensity) are shown via a gradation toning technique. In an exemplary gradation scheme, a parameter value has a unique color associated with it according to the toning technique. A change in a parameter value at a given pixel usually results in a color change associated with that pixel, representing a new parameter value. As shown in the example of FIG. 14, the acoustic parameter values appear to change radially from the center position of the acoustic signal at positions 232, 234, and 236. Other examples of gradation toning are described in US Patent Application No. 15 / 802,153, filed November 2, 2017 and transferred to the transferee of this application.

FIG. 15 shows a visualization of acoustic data using a plurality of shaded concentric circles. In one such example, each identical color of the concentric circles shown in FIG. 15 is the value associated with that color, as opposed to a gradation scheme with a color associated with the parameter value. Can represent pixels with acoustic parameter values within the range of. In the illustrated example, the acoustic parameters (eg, intensity) associated with the acoustic signal at positions 332, 334, 336 vary radially from the center of the acoustic signal. In an exemplary toning technique, the pixels shown in red represent acoustic parameter values within the first range of parameter values, and the pixels shown in yellow represent acoustic parameter values within the second range of parameter values. Represented and represented in green, the pixels represent acoustic parameter values within a third range of parameter values, but other display techniques including additional or alternative colors, patterns, etc. are possible. In various embodiments, the range of values can correspond to an absolute range, such as an intensity value between 10 dB and 20 dB, or a relative range, such as an intensity value between 90% and 100% of maximum intensity. could be.

As described elsewhere herein, in certain embodiments, the display image, including the electromagnetic image data and the acoustic image data, is a visual representation of the acoustic signal and one or more parameters associated with the acoustic signal. It can contain both alphanumeric representations. FIG. 16 shows an exemplary visualization showing both non-numerical information (eg, toning over a parameter value range) and alphanumerical information. In the illustrated example, an alphanumerical sound intensity value label is associated with each of the three positions having toned acoustic image data (eg, intensity data). As illustrated, the acoustic signal has associated acoustic parameters (1602, 1604, 1606) and corresponding visual indicators representing alphanumerical information (1612, 1614, 1616, respectively). In an exemplary embodiment, the alphanumerical information can provide a numerical value, such as a maximum intensity value, associated with the location where the toned acoustic data is displayed. In one example, the user may select one or more positions for displaying toning and / or alphanumerical data. For example, the user may choose to annotate the display image using an alphanumerical representation of the acoustic parameters associated with one or more acoustic signals in the scene.

In one example, the alphanumerical information can represent multiple parameters (eg, acoustic parameters) associated with an acoustic signal at a given location in the scene. FIG. 17 shows an example of a visualization that includes both non-numerical information (eg, toning over a parameter value range) and alphanumerical information. In the illustrated example, toned acoustic data (eg, intensity data) in which a sound intensity value and a corresponding frequency value (eg, average frequency or peak frequency) are indicated via indicators 1702, 1704, 1706, respectively. It is shown in the alphanumeric information 1712, 1714, 1716 associated with each of the three positions it has. As described with respect to FIG. 16, inclusion of various such data at various locations can be initiated by the user. For example, the user may choose to annotate the display image using an alphanumerical representation of one or more acoustic parameters associated with one or more acoustic signals in the scene.

FIG. 18 shows another exemplary visualization showing both non-numerical information (eg, toning over a parameter value range) and alphanumerical information. In the illustrated example, the distance measurements are indicated via indicators 1802, 1804, 1806, respectively, with alphanumerical information 1812, 1814 associated with each of the three positions having toned acoustic data (eg, intensity data). , 1816. The inclusion of various such data in various locations, as described with respect to FIG. 16, may be selected by the user, for example, as part of the display image annotation.

In some examples, non-numerical representations may be used to convey information related to multiple acoustic parameters. For example, FIG. 19 shows an exemplary visualization showing different size and color indicators 1902, 1904, 1906 (in this case, circles) representing different acoustic parameter values. In an exemplary embodiment, the size of the indicator corresponds to the intensity of the acoustic signal at a given position, and the color of the indicator corresponds to the peak or average frequency. In an exemplary embodiment, the indicator size is a relative value, just as comparing the size of one indicator to the size of another indicator represents the relative difference between the acoustic parameter values represented at the location associated with the indicator. Can be shown. On top of that or separately, alphanumerical information can be included to provide absolute or relative acoustic parameter values.

In certain embodiments, a colored indicator may be used to indicate the severity of one or more detected acoustic signals and / or associated acoustic parameters such as deviations from reference parameters. FIG. 20 shows an exemplary visualization showing a plurality of indicators 2002, 2004, 2006 with different colors indicating the severity indicated by the acoustic signal from the corresponding position. For example, in an exemplary embodiment, the red indicator indicates significant severity based on another acoustic parameter (eg, when compared to criteria such as typical operating condition criteria), and the yellow indicator is Medium severity, green indicates a mild severity indicator. In other examples, other color schemes or appearance characteristics (eg, indicator transparency, indicator size, etc.) may be used to visually distinguish the severity of the acoustic signal. In the illustrated example, indicator 2004 represents the highest level of severity, indicator 2006 represents the next highest level, and indicator 2002 represents the least severe acoustic signal.

As described elsewhere herein, in various embodiments, one or more acoustic parameters are displayed in a visual representation of the acoustic scene, for example, by a toned color display or grayscale display. May be done. In certain embodiments, the system may be configured to identify one or more positions in a scene that satisfy one or more acoustic conditions, such as a specified frequency range, intensity range, distance range, and the like. .. In one example, various positions corresponding to a sound profile (eg, satisfying a particular set of conditions or parameters) may be identified. Such identified locations may be presented in a manner different from the acoustic image data toning techniques used in other methods in creating display images. For example, FIG. 21 shows a scene that includes indicators at multiple positions within the scene. Indicators 2101, 2102, 2103, 2104, and 2105 are arranged in the scene. Indicators 2103, 2104, and 2105 include toned acoustic image data representing, for example, the values of one or more acoustic parameters corresponding to scale 2110. Indicators 2101 and 2102 are shown to have a unique presentation technique that distinguishes them from the toning techniques that appear at positions 2103, 2104, and 2105. In such an embodiment, the user can quickly and easily identify their position in an image that meets one or more desired conditions. In one such example, the user may select one or more desired conditions for display in a distinguishable manner, based on the selection of a range of values from a scale, such as 2110.

On top of that, or separately, positions that meet the conditions for a particular sound profile can be presented with icons that represent the conditions, such as the corresponding sound profile. For example, FIG. 22 shows a plurality of icons 2202, 2204, 2206, 2208 arranged in a display image showing a recognized sound profile in the scene. The exemplary profile shown in FIG. 22 includes bearing wear, air leaks, and electric arc discharge. Such profiles may be identified by acoustic signals that satisfy one or more sets of parameters associated with such profiles in order to be classified into such profiles.

FIG. 23 shows another exemplary display image showing acoustic data via a plurality of indicators 2302, 2304 and 2306, using concentric circles representing the acoustic intensity associated with each acoustic signal and alphanumerical information. .. As described elsewhere herein, in one example, the size of the indicator can represent one or more acoustic parameters present at the corresponding position. In certain embodiments, the indicator may be monochromatic, indicating acoustic parameters in one or more other ways, such as by indicator size, line thickness, line type (eg, solid line, dashed line, etc.). Can be done.

In one example, the display may contain alphanumerical information based on the choices made by the user. For example, in one embodiment, the system is located in a particular position (eg, via a processor) in response to a user selection of an indicator on the display at such position (eg, via a user interface). It can contain information that represents one or more acoustic parameters of the acoustic signal. FIG. 24 shows an exemplary display image with indicators and additional alphanumerical information associated with the displayed acoustic signal. In one example, the indicator 2402 on the display may be selected for further analysis (eg, represented via a crosshair, which may indicate a selection, such as via a touch screen input). The display contains alphanumerical information including a list of data related to the position corresponding to the indicator, including the peak intensity and the corresponding frequency, frequency range, measurement distance to the position, and the critical level indicated by the acoustic signal from that position. 2404 is shown.

In one example, the display image can include multiple indicators that represent the corresponding multiple acoustic signals in the scene. In certain embodiments, in such cases, the user may select one or more indicators (eg, via a touch screen or other user interface), and in response to detection of the selection, the processor. , May provide additional information about the acoustic signal. Such additional information may include alphanumericals for one or more acoustic parameters. In one example, such additional information can be displayed simultaneously for multiple acoustic signals. In another example, such additional information about a given acoustic signal is hidden when another acoustic signal is selected.

In some examples, the system can include a laser pointer, as described elsewhere herein. In some examples, the laser pointer can have a fixed orientation or, for example, can have adjustable pointing that can be controlled via a processor. In one example, the system may be configured to point the laser pointer at a position in the target scene associated with the selected position in the image. FIG. 25A shows a system (in one example, embodied as a handheld tool), including a display such as the display shown in FIG. 24, to which indicator 2502 is selected. The laser pointer 2504 emits a laser beam 2506 towards the scene, and the laser creates a laser spot 2508 in the scene corresponding to the position of the selected indicator 2502 in the image. This can help the user visualize the location of the selected and / or analyzed acoustic signal in the environment. In certain embodiments, the laser spot 2508 is detectable by an electromagnetic imaging tool and can be seen on the display along with the displayed indicator 2502 and the alphanumerical information 2512 including acoustic parameter information. In one example, the acoustic imaging system is configured to detect or predict the position of the laser in the scene and provide a visual indication of the laser position 2510. 25B shows a display image as shown in the system view of FIG. 25A.

In embodiments where the laser pointer has a fixed orientation, the user can see as feedback a display image that visually indicates the laser position, so that the user can see the laser to match the selected acoustic signal. You can adjust the aim of.

As described elsewhere herein, in certain embodiments, the acoustic image data can be combined with the electromagnetic image data and presented as a display image. In one example, the acoustic image data can include transparency that can be adjusted so that various aspects of the electromagnetic image data are not completely hidden, and FIG. 26 shows the acoustics represented by the indicator 2602 at some point in the scene. Showing image data, indicator 2602 includes a gradation scheme. The system may include a display device that is integrated with or separate from the acoustic imaging tool and is configured to present display data, including electromagnetic image data and acoustic image data.

In certain embodiments, the device (eg, a handheld acoustic imaging tool) is a physical blend control 2614 (eg, one or more buttons, knobs, sliders, etc. that may be included as part of the user interface) and / or touch. Virtual blend control 2604, such as via a screen or other virtually implemented interface, can be included. In certain embodiments, such functionality may be provided by an external display device such as a smartphone, tablet, computer or the like.

FIG. 27 shows a virtual and / or physical blend control tool for display images, including a partially transparent concentric toning technique. Similar to the description with respect to FIG. 26, the indicator 2702 can represent an acoustic signal in the scene. The acoustic imaging system can include physical blend control 2714 and / or virtual blend control 2704 that can be used to adjust the transparency of the acoustic image data (eg, indicator 2702) in the display image.

On top of that or separately, physical and / or virtual interfaces may be used to adjust one or more display parameters. For example, in one embodiment, one or more filters are applied to selectively display acoustic image data that meets one or more conditions, as described elsewhere herein. obtain. FIG. 28 shows a position in a scene that satisfies one or more filters (eg, has one or more corresponding thresholds or one or more acoustics or other parameters that meet a given condition). It shows a scene containing an indicator 2802 with a color. In various examples, filtering is selected and / or via physical control 2814 (eg, via one or more buttons, knobs, switches, etc.) and / or virtual control 2804 (eg, through a touch screen). Can be adjusted. In the illustrated example, the filter comprises displaying acoustic image data of only acoustic signals having acoustic parameters (eg, frequency) within a predefined range 2806 of acoustic parameters. As shown, the predefined range 2806 is a subset of the possible filter range 2816. In one example, the user may adjust the limits of the predefined range 2806 to adjust the filter effect, for example, via virtual 2804 or physical 2814 control.

FIG. 29 shows a virtual and / or physical filter adjustment of a display image, including a partially transparent concentric toning technique. As shown, the indicator 2902 is shown in the scene based on acoustic parameters within a predetermined range 2906 of values based on the filter. The filter may be adjustable within the range of values 2916, for example via virtual 2904 and / or physical 2914 control.

In certain embodiments, a plurality of filters may be used to customize a display image that includes toned acoustic image data. FIG. 30 shows a display image showing a first indicator and a second indicator. As described elsewhere herein, one or more filters are applied to the display image (eg, via physical filter control and / or virtual filter control) to customize the displayed data. can do. In the illustrated example of FIG. 30, filtering involves establishing a first filter range 3006 and a second filter range 3008. In one example, the filter range may be adjustable within the range of values 3016, for example via virtual 3004 and / or physical 3014 control.

Such a filter range can represent any of a variety of parameters such as frequency, amplitude, and proximity. As shown, the first filter range and the second filter range are each associated with a color (in one example, user adjustable), and the indicators 3002, 3012 have corresponding acoustic signals for each filter. It is placed at a position in the image that meets one or more filter criteria associated with the range. As shown, the first indicator 3002 represents an acoustic signal that fills the first filter range 3006 (indicated by dark shading) and the second indicator 3012 represents a second filter range 3008 (bright shading). Represents an acoustic signal that satisfies). Therefore, the user can quickly identify a position in the scene having acoustic data satisfying various conditions at the same time while specifying which position satisfies which condition.

In one example, a display device such as an acoustic imaging tool or an external display device may include a virtual keyboard as an input device as shown in FIG. FIG. 31 shows a display interface that includes an indicator 3102 that represents one or more acoustic parameters of an acoustic signal in the scene. A virtual keyboard 3110 is included in the display, which can be used to add alphanumerical information 3112 to the display image. Utilizing such a virtual keyboard allows the user to enter customized annotations such as various inspection notes, labels, date / time stamps, or other data that can be stored with the image. In various examples, a virtual keyboard can be used to add text that is included in the image data and / or that is stored in the metadata associated with the display image and is added to the image data. ..

Various devices may be used to present display images containing various combinations of acoustic image data with alphanumerical data, image data from one or more electromagnetic spectra, symbols and other data. .. In one example, the handheld acoustic imaging tool can include a built-in display for presenting display images. In another example, the information to be displayed, or the data processed to generate the display (eg, raw sensor data), may be communicated to an external device for display. Such external devices can include, for example, smartphones, tablets, computers, wearable devices and the like. In certain embodiments, the display image is presented in combination with real-time electromagnetic image data (eg, visible light image data) on an enhanced realty type display.

FIG. 32 shows a display built into eyewear 3210 that can be worn by the user. In one example, the eyewear is "incorporated into a test and measurement tool, or image display from an attached imaging tool (DISPLAY OF IMAGES FROM AN IMAGING TOOL EMBEDDOR), the relevant part of which is incorporated herein by reference. ATTACEED TO A TEST AND MEASUREMENT TOOL) ", which may include one or more built-in imaging tools as described in US Patent Publication No. 20160076937 transferred to the transferee of this application. In one such example, the integrated display can display a real-time display image 3220. For example, the display can display electromagnetic image data (eg, visible light image data) that represents the scene the user is facing, and at the same time (eg, via blends, overlays, etc.) and acoustic image data (eg, via an overlay, etc.). One or more additional data streams such as (including indicator 3202) can be displayed to provide additional information to the user. In certain embodiments, eyewear as shown in FIG. 32 includes a transparent display screen, so that if no display image is provided to the display, the user is not presented with real-time visible light image data. , You can see the scene directly with your own eyes through eyewear. In one such example, additional data, such as acoustic image data, alphanumerical data, etc., is displayed on another transparent display within the user's field of view, allowing the user to add to their field of view of the scene through the display. You can see such data.

As described elsewhere herein, in various examples, the various data presented in the display image is a blend with another data stream (eg, a blend of acoustic and visible light image data). ) Can be combined in various ways. In one example, the strength of the blend can vary between different locations within a single display image. In certain embodiments, the user can manually adjust the blend ratio of each of the plurality of positions (eg, each of the plurality of indicators of the detected acoustic signal). Moreover or separately, the blend can be a function of one or more parameters such as frequency, amplitude, proximity and the like.

In one embodiment, the acoustic imaging tool identifies the extent to which the sensor array points to each of a plurality of locations emitting the detected acoustic signal and provides the corresponding acoustic image data, eg, visible light image data. Can be configured to blend with. FIG. 33A shows an exemplary display including a first indicator 3302 and a second indicator 3304 that represent acoustic signals in an acoustic scene. In FIG. 33A, the acoustic sensor more directly points to the pipe 1 corresponding to the position of the first indicator 3302 when compared to the pipe 2 corresponding to the position of the second indicator 3304. Therefore, in the display technique of FIG. 33A, the first indicator 3302 is displayed more prominently than the second indicator 3304 (eg, has a higher blend factor or lower transparency). Conversely, in FIG. 33B, the acoustic sensor more directly points to the pipe 2 corresponding to the position of the second indicator 3304 when compared to the pipe 1 corresponding to the position of the first indicator 3302. Therefore, in the display technique of FIG. 33B, the second indicator 3304 is displayed more prominently than the first indicator 3302 (eg, has a higher blend factor or lower transparency). In general, in one embodiment, the acoustic imaging system determines a metric that indicates how much the sensor is oriented at a given position (eg, corresponding to an indicator of acoustic image data) and corresponds to the corresponding position. The blending ratio to be applied can be adjusted (for example, the greater the degree of aiming, the higher the blending ratio correspondingly).

In one example, the processor can save the sound profile detected in the scene. For example, in an exemplary embodiment, the user can store the detected acoustic data (eg, displayed as acoustic image data) as a sound profile corresponding to one or more parameters. In certain such examples, such sound profiles may be labeled according to and / or in connection with one or more features of the scene, such as the presence of air leaks. Moreover or separately, a predefined sound profile may be read into the system memory during factory assembly of the acoustic imaging system and / or may be downloaded to the acoustic imaging system or communicated separately. ..

In some examples, the sound profile may include one or more sounds present in the scene. Various sound profiles can be defined by one or more acoustic parameters such as frequency, decibel level, periodicity, distance, etc. (eg, the illustrated sound profile is associated with a frequency value within a predetermined range and that profile. May include periodicity within a given range). A sound profile may be defined by one or more sounds in the scene, in one example where each of the sounds is one or more acoustic parameters such as frequency, decibel level, periodicity, or distance. Can be defined. Multiple sounds of a given profile are specified by similar parameters (eg, each of the two sounds has a frequency range and a periodic range), or by different parameters (eg, a sound). Has a corresponding frequency range, but other sounds may have a corresponding decibel level range and maximum distance value).

In one example, the system may be configured to provide notifications about a sound profile, such as when one or more sounds in an acoustic scene correspond to a known sound profile. For example, the notification can give the user or technician a warning of the recognized sound profile. Moreover or separately, the system may be configured to annotate acoustic image data, electromagnetic image data, and / or display images based on the recognized sound profile. Notifications may include audible sounds, visualizations on the display, LED lights, and so on.

In one example, the system processor may be configured to analyze the criticality of an acoustic signature, taking into account, for example, one or more sound pull files. The correspondence between acoustic data (eg, in relation to sound profiles) and the criticality of the system can be learned, for example, based on machine learning inputs and / or user inputs.

In one example, a processor may be configured to compare data in an acoustic scene with one or more known sound profile patterns, eg, to analyze the scene with respect to the criticality of the detected signature. In various examples, the processor may be configured to notify the user based on potential criticality. This can be, for example, a comparison of the identified acoustic signature with one or more stored criteria, a relative value compared to a user-defined threshold, and / or (eg, past performance, errors, or responses). It can be done based on automatically determined values via machine learning algorithms and / or artificial intelligence programming (based on historical acoustic signatures).

In such various examples, the processor is configured to analyze the acoustic scene with respect to one or more sound profiles to estimate the effect of the discovered sound profile of an object in or within the scene. Sometimes. Estimating the effect may include potential cost or loss of profit with respect to the potential criticality of the sound profile and / or the acoustic signature. In one example, the system (eg, via a processor) has one or more air in the scene (eg, by comparing the acoustic signature to one or more sound profiles for air leaks in the acoustic scene). May be configured to recognize leaks. In one such example, estimates related to the number of air leaks detected, the severity of one or more air leaks, and / or repairing such one or more leaks. Various data such as cost savings can be calculated and / or reported automatically. In one example, cost estimates may be made based on pre-programmed values and / or user input values associated with detected leaks. In the illustrated applications, the system may be configured to determine the cost impact of compressed air leaks per unit time (eg, 1 hour) if not properly repaired.

In one example, the acoustic imaging system may be configured to determine various information about air leaks in the acoustic scene, such as pressure, hole diameter, or leak amount. In some examples, one or more such values may be entered by the user and the remaining values may be produced. For example, in one example, the user can enter a pressure value associated with a particular air line, and with the pressure information entered, the system will use acoustic data from the scene to enter hole diameters and leaks. May be configured to determine. Such a determination can be made, for example, using a look-up table and / or an expression stored in memory.

In an exemplary scenario, the detected sound profile may be associated with a 100 PSIG air leak through a 1/4 inch diameter hole. In certain embodiments, an acoustic imaging system that has access to such a stored sound profile recognizes such profile in an acoustic scene and is associated with such leaks, eg, based on a look-up table. It may be programmed to estimate the cost per hour. In a similar example, the detected sound profile can be associated with a 1/4 inch diameter hole based on an input pressure of 100 PSIG (eg, via manual input). An acoustic imaging system that has access to such a stored sound profile recognizes such a profile in an acoustic scene and, for example, based on a look-up table, the cost per hour associated with such a leak. May be programmed to estimate.

In one example, the acoustic imaging system may be configured to perform cost-saving analysis to clarify multiple detection omissions based on equations and / or look-up tables. In one example, such equations and / or look-up tables can be stored in the memory of the acoustic imaging device or elsewhere accessible by it for cost analysis of the identified leaks.

In one exemplary implementation, the acoustic imaging system may be configured to characterize one or more leaks detected in the environment during a scene or facility inspection. Such characterization may be done, for example, based on a stored sound profile corresponding to such a leak. Detected leaks (eg, of a defined leak amount) may be used to calculate cost savings when such leaks become apparent. For example, an acoustic imaging system may be used to determine the number of leaks present and the amount of leaks associated with such leaks to calculate the cost savings associated with such leaks.

In one example, the cost savings are the number of leaks, the amount of leaks (cfm), the amount of energy associated with the leaks (eg, kW / cfm), and the number of hours of operation, as shown in equation (1) below. , Can be calculated by multiplying the cost per energy.

Cost saving ($) = number of leaks x amount of leaks (cfm) x kW / cfm x number of hours x $ / kWh ... (1)

Acoustic imaging systems may be used to determine the number of leaks and the amount of leaks associated with such leaks (eg, at cfm). Other parameters are either programmed into the system (eg, energy per air generation at kW / cfm), accessed via a database (eg, current energy cost at $ / kWh), or It may be estimated by the system (eg, average number of hours of operation). The system may be programmed to calculate the cost savings associated with clarifying such leaks.

In one example, the system has 100 leaks of 1/32 inch at 90PSIG, 50 leaks of 1/16 inch at 90PSIG, and 10 leaks of 1/4 inch at 100PSIG. Assuming an operating time of 7,000 years, a total electric energy of 0.05 $ / kWh, and compressed air generation conditions of approximately 18 kW / 100 cfm, the cost savings associated with each leak according to the equation (1) are as follows.

Cost savings from 1/32 inch leaks = 100 x 1.5 x 0.61 x 0.18 x 7000 x 0.05 = $ 5,765

Cost savings from 1/16 inch leaks = 50 x 5.9 x 0.61 x 0.18 x 7000 x 0.05 = $ 11,337

Cost savings from 1/4 inch leaks = 10 x 104 x 0.61 x 0.18 x 7000 x 0.05 = $ 39,967

As shown in this example, the savings from removing exactly 10 leaks in a quarter inch account for almost 70% of the total savings. Since the leak has been identified, in one example, the acoustic imaging system may be configured to analyze the leak and identify which leak produces higher cost savings (if obvious). .. In one example, the system can rank or prioritize leaks with higher cost savings and provide such ranks and priorities to users. In addition or separately, the system may be configured to provide the user with notification of cost savings for one or more of the identified leaks.

FIG. 34 shows an exemplary display image that provides a user or technician with instructions relating to the potential criticality of an air leak identified in the scene and the potential cost of loss due to the air leak. There is. In one exemplary embodiment, the acoustic imaging system (eg, via a processor) is within the acoustic scene, for example, through recognition of an acoustic signal similar to a known sound profile associated with a leak present in the acoustic scene. Detects the presence of leaks. The system is configured to identify the criticality of the identified leak by analyzing the detected acoustic signal and, for example, determining one or more sound profiles corresponding to the detected acoustic signal. There is.

As described elsewhere herein, in some cases the acoustic image data may be toned according to a determined criticality. In certain embodiments, the criticality of acoustic data may be determined according to one or more sound profiles. For example, the user can select a leak detection operating mode and the acoustic imaging system can analyze the acoustic data for one or more sound profiles associated with the leak to determine the criticality of the acoustic data in the scene.

As mentioned above, the system may be configured to calculate the numerical cost associated with one or more leaks, such as by one or more equations and / or look-up tables. .. In one exemplary embodiment, the system can identify the size of the leak and the pressure associated with the leak, eg, based on the sound profile associated with such a leak, and then by look-up table and / or equation. The cost per unit time associated with a leak can be calculated.

In the illustrated example of FIG. 34, the criticality and cost per unit time ($ / year) are associated with each of the plurality of positions in the acoustic scene. In the illustrated embodiment, such positions are toned according to criticality and combined with visible light image data to form a display image that provides the user with leak critical information for each position 3410, 3412. , 3414 are shown via acoustic image data. Such criticality can be determined by analysis of the acoustic scene in relation to one or more corresponding sound profiles, as described herein.

In various embodiments, leaks that vary in degree of criticality and / or potential loss cost may be indicated by acoustic image data including various colors, shapes, sizes, opacity, and the like. Similarly, one or more icons may be used to represent a special leak (eg, corresponding to a special sound profile) or the cost or critical range of the leak. On top of that or separately, alphanumerical information such as cost / year may be included closest to the corresponding leak. The display image of FIG. 34 further includes a notification 3420 showing an approximate cost associated with each critical level. Such values may be calculated based on the acoustic parameters for each acoustic signal detected in the scene, for example, taking into account profiles stored in memory representing a particular air leak.

In one example, another contextual information is useful or essential for proper analysis and for reporting the scene. For example, during an operation, when acoustically capturing a scene, the user or technician may wish to record contextual information about the scene being inspected. In older systems, to perform this task, the user or technician would need to take a picture with another camera or device, take document notes, or record notes with another device. Such notes must be manually synchronized with the data from the acoustic imager, potentially leading to collection errors and recollections and data mismatches, and to errors when analyzing and reporting. Potentially guided.

In certain embodiments, the acoustic imaging system of the present disclosure can capture acoustic data of a target scene and associate it with information about the target scene. Such information about the target scene can include details about one or more objects in the scene, around the scene, and / or around the position of the scene. In certain embodiments, the relevant information is captured in the form of an image, audio recording or video recording and associated with acoustic data representing the scene (eg, the acoustic data itself, a display image containing the corresponding acoustic image data, etc.). obtain. In one example, the relevant information is associated with acoustic data to provide a better understanding of what that information represents. For example, relevant information may include details about the target scene or objects within the target scene.

In one example, the system may include one or more devices for gathering information about objects in the scene or the scene in general. One or more such devices may include cameras, positioning devices, clocks, timers and / or various sensors such as temperature sensors, electromagnetic sensors and humidity sensors. In one example, an acoustic imaging system can include a camera (eg, embedded in the housing of an acoustic imaging device) that may be configured to collect annotation information for image and / or video image acquisition. Such embedded cameras can generate photo or video annotations that are added to or otherwise stored with the acoustic image data or other data acquired by the acoustic imaging device.

In one example, such an embedded camera may be configured to produce electromagnetic imaging data that can be combined with acoustic image data for a display, as described elsewhere herein. be. In certain embodiments, the acoustic imaging system stores various information, such as acoustic data, acoustic image data, electromagnetic image data, and annotation data (eg, sensor data, image / video annotation data, etc.), along with a time stamp. It may be configured as follows. In one example, the acoustic imaging system may be configured to display and / or record relevant information both on the device and later in software. The relevant information may be displayed on the display along with the stored display image or acoustic image file and / or stored as metadata, for example.

According to certain embodiments, through the use of an embedded camera, an image acquisition device or a video acquisition device in an acoustic imaging system, such device can be used for primary acoustic data, combined electromagnetic and acoustic image data, and / or audio recording. May be used to generate photo or video annotations that are added to and / or stored with them.

In one example, the user or technician may have been collected by an acoustic image data or acoustic imaging system shown on a display, eg, via a user interface (eg, a touch screen or one or more buttons). Data can be annotated. For example, in one example, a user or technician may use on-display annotations to annotate data during data recording or playback of previously collected data. can.

In one example, the acoustic imaging system can annotate display images (including, for example, acoustic image data and / or electromagnetic image data) via on-screen interactions from system utilization. In one such example, the device stores all relevant annotation information along with the primary acoustic image, either prior to the need for synchronization or for subsequent data matching. Such practices make it possible to reduce or eliminate errors in human memory that result in inaccurate pairing of primary and secondary data.

Various on-display annotations that can be added by the user (eg, via the user interface) are on-display drawings (eg, freehand), text on the display. And description (on-display extend and writing), on-display shape creation, on-display move (movement of object), placement of preconfigured markers on the display (on-display movement of object). On-display precision of pre-configured marks), placement of pre-configured text, instructions or notes on the display (on-display precision of pre-confixed textures, instruments or notes), pre-configured shapes on the display. On-display precision of pre-confixed patterns, on-display precision of pre-confixed drivings or illuminations or pre-configured on the display. It may include, but is not limited to, the placement of icons (on-display precision of pre-configured or programmed icons), or the visualization of one or more acoustic parameters on the display.

In one example, the acoustic imaging device may include a display, which can show collected (eg, live or pre-collected) acoustic and / or electromagnetic image data. The user can annotate such an image via a control unit or touch interface integrated into the display.

FIG. 35 shows an example of a user annotating a display image with annotations on the display. In the example of FIG. 35, three acoustic signals are shown at corresponding positions on the display image via toned indicators representing acoustic data associated with such acoustic signals. As illustrated, in one example, the user can annotate the display image by drawing a freeform shape 3540 to highlight or emphasize a portion of the display image. In this example, the user annotates the display image by surrounding the sound near the center with freeform Figure 3540. The illustrated example is also another in the form of text 3542 that can be used to identify or describe one or more components of the display image, such as the components associated with the circled sound. Shows annotation information. For example, in the illustrated example, a label of a clearly circled sound source (“Mainline 2B-28”) is added to the display image. Such text is either handwritten (eg, via a touch screen interface) or typed (eg, via a virtual or physical keyboard that is communicating with the system).

FIG. 36 shows an example of an annotated display image that includes location information associated with the instruction. In this figure, two sounds are shown on the display in addition to textual information and graphical indications that command the task to be done by the user. The user may annotate the image as shown to order or encourage further users to perform one or more tasks while performing an acoustic check of such a position. ..

When generating an annotated image such as that shown in FIG. 36, the user can annotate the display image by inserting a preconfigured marker (eg, arrow 3640) and / or handwritten text 3642. .. For example, in the illustrated example, the user annotates the display image by inserting an arrow 3640 towards a valve switch in the scene and a textual command 3642 "Close the main valve first".

FIG. 37 shows an example of a user annotating a display image with annotations on the display. In this example, a single acoustic signal is shown via the corresponding indicator on the display image. The illustrated example shows a user annotating a display image by inserting a pre-programmed icon 3740, such as an air leak icon, into the display image. Such icons are recommended by the system and / or based on one or more recognized (eg, historical sound profiles) characteristic sounds selected by the user and / or more. May be placed on the specified sound. The user can also annotate the image to identify possible sound sources for the observed acoustic signal or label them with a freeform label, a predefined shape, or the like.

FIG. 38 shows an example of a user annotating a display image with annotations on the display. In this example, a single acoustic signal is shown via the corresponding indicator on the display image. As shown, the user can annotate the display image by inserting the shape 3840 into the image. In the illustrated example, the user adds a box 3840 around the left-hand component of the display image to show a possible sound source from the acoustic image data. Such a box 3840 may be selected as a predetermined shape (eg, a positionable and adjustable sized rectangle). Other shapes can be used, sized, positioned and annotated on the display image as desired by the user.

In some cases, different types of labels may be combined on the display. For example, FIG. 39 shows an icon label 3942 placed on a display image near an indicator of acoustic image data representing an acoustic signal in the scene and a rectangle 3940 surrounding a component in the scene. Various annotation combinations are possible. In certain embodiments, for example, annotations are automatically added to the display image when a particular sound profile is recognized in the scene. In one example, the system may arouse the user to annotate the display image in light of the perceived sound profile.

On or apart from that, in one example, the user may choose to annotate the image by including visual or textual information that represents the acoustic parameters associated with the acoustic scene. For example, the user may select a particular type of display to show one or more acoustic parameters associated with the scene via (eg, a touch screen or physical control unit). The user can also annotate the image to include information about cost or critical indications associated with parts of the scene, such as detected leaks.

Annotations may include display features contained within the live screen of the display image and / or contained in a single displayed display image captured, eg, stored in memory.

As described elsewhere herein, in various embodiments, the acoustic imaging device displays the detected sound, locates it, indicates it, and analyzes it considerably. Different methods can be adopted. Visualization methods include various colored shapes, icons, which can be accompanied by various levels of transparency adjustments that adjust the visible background in which they are displayed. By simplifying parameter control for acoustic visualization on the device, or making such control more intuitive, users can have less time, less training, and easier and better vision. The result of conversion can be achieved. Various methods of visual parameter control can be performed according to the application and user needs. Many of these methods have been modified and adapted according to the use of sound visualization and location location education and training by special individual levels to provide more adapted equipment in various types of applications and organizations. As described elsewhere herein, in one example, the user may choose to annotate the display image by including a particular data visualization technique in the display image.

FIG. 40 shows an interface that includes a display image 4002 and a multi-parameter data visualization 4040 that includes a plurality of frequency ranges on the right hand side of the display image. In one example, the user may select one or more frequency ranges (eg, via a touch screen and / or other interface) from a plurality of displayed frequency ranges, eg, acoustic image data. It is filtered to display an acoustic image with frequency content associated with the selected frequency range (eg, frequency content above a certain size). In the illustrated example, two frequency ranges 4042 and 4044 are selected. Corresponding indicators 4010 and 4012 of the display image indicate a position in the scene having an acoustic signal satisfying the corresponding frequency range among the plurality of frequency ranges displayed on the screen. In the illustrated example, the frequency range 4042 and the corresponding indicator 4010 are shown with a bright shadow line, while the frequency range 4044 and the corresponding indicator 4012 are shown with a dark shadow line. In general, in certain embodiments, an acoustic signal in a scene that fills a frequency range can be represented via an indicator having a corresponding visualization representation as that frequency range.

In general, the frequency range can be displayed in various ways, such as a right-handed axis, a left-handed axis, a lower-end aligned axis, an upper-end aligned axis, or a central axis. In various embodiments, the frequency range may be divided into resolutions, including every 1 kHz (eg, 1 kHz to 2 kHz, 2 kHz to 3 kHz, etc.). Such frequency ranges do not necessarily have to be the same magnitude or span the same frequency range. In one example, the physical magnitude (eg, width) of the frequency range displayed in the multi-parameter image 4040 is the relative amount of frequency content, the amplitude of such frequencies in the acoustic scene, of such frequencies. It corresponds to one or more parameters such as proximity.

In various embodiments, the frequency range is selected by virtual control through physical control mechanisms such as touch screen interactions and / or directional pads, but the user scrolls up, down, left and right to highlight the desired range bar. Press and hold the button until it is displayed and then selected. In one example, multiple ranges are selected or deselected by the user.

In certain embodiments, the frequency bars (eg, 4042,4044) on the display image 4002 associated with the various frequency ranges move up and down at decibel levels within the range within the real-time display image. In various examples, the range decibel level can be determined in a number of ways, including peak decibel levels, average decibel levels, minimum decibel levels, and time-based average decibel levels. In certain embodiments, the decibel level associated with each frequency range may be tracked over time. Tracking over time can include storing frequency information at each of multiple times, such as at given intervals. Tracking frequency data over time, on or apart from it, involves tracking peak decibel levels observed over time (eg, within a special period, within an operating session) in each frequency range. be able to. The peak level can be calculated by various methods.

The display of frequency information can include peak frequency data in addition to current / current frequency information. FIG. 41 shows an interface comprising a display image 4102 and a multi-parameter display 4140 of frequency information including a plurality of frequency ranges arranged along the lower edge of the display image. As shown in FIG. 41, the frequency information displayed at the bottom edge of the display image contains amplitude information (eg, 4150) in a plurality of frequency ranges (measured in decibels in the example of FIG. 41). ). In one example, the maximum decibel level is displayed in the frequency range of one or more. In some cases, the frequency range indicating the maximum decibel level is selectable by the user. The peak marker 4152 can remain on the multi-parameter display, indicating the maximum decibel level and represented in a significant number or possible ways such as contrasting colors, bar caps, arrows, symbols, numbers, etc. It is possible. In one example, the maximum decibel level shown in the data visualization includes the maximum value detected over a period of time, such as within the last 5 seconds, within the last 30 minutes, etc., from the start of the selected measurement. In one example, the user can reset the maximum value display to ignore the previous maximum value.

In various embodiments, the frequency band contained in the display image can be adjusted by the user or automatically determined by the device with a programmed algorithm or machine learning. In various examples, the frequency band can be determined evenly and distributed, divided into different sizes, is an equalized histogram, or by a considerable combination of methods.

FIG. 42 shows a display image including frequency information of the plurality of frequency bands and peak values of the plurality of frequency bands similar to those shown in FIG. 41. In the example of FIG. 42, a multi-parameter display 4240 containing intensity and frequency information is shown on the right hand portion of the screen and amplitude information (eg, 4250) is shown horizontally with a right-handed axis. Similar to FIG. 41, the multi-parameter display 4240 includes a peak marker 4252 indicating the peak amplitude in one or more frequency ranges within the period.

In one example, the decibel level increases towards the right and left of the central axis, for example in a mirror image. Similarly, decibel level peaks may appear in the mirror image.

Such decibel information mirrored to the central axis is shown in FIG. The display image of FIG. 43 includes a multi-parameter display 4340 showing intensity information for a plurality of frequencies. Such a mirror image axis display allows the user to better identify small changes or low decibel level changes that are important in frequency range selection.

Similar to FIGS. 41 and 42, the multi-parameter display 4340 of FIG. 43 includes a peak marker 4352 indicating the peak amplitude in one or more frequency ranges within the period.

In certain embodiments, one or more frequency ranges may be toned to provide additional information about such frequency ranges, such as decibel levels in such frequency ranges. FIG. 44 shows a multi-parameter display 4440 that includes a toned set of frequency ranges, in which the toning represents the decibel range in which each frequency range is classified. For example, in an exemplary embodiment, the frequency range shown in white (eg, 4450) falls within 0-20 dB, the frequency range shown in light gray (eg, 4450) falls within 21-40 dB, and in dark gray. The frequency range shown falls within 41-100 dB. In one example, toning is, for example, the frequency range shown in white is considered to have "low" intensity, the frequency range shown in light gray is considered to have "normal" intensity, and dark gray. Corresponds to relative values rather than absolute values, as the frequency range indicated by is considered to have "high" intensity. In an exemplary embodiment, the three below the frequency range with respect to intensity are classified as white, the three in the middle of the frequency range with respect to intensity are classified as light gray, and the three above the frequency range with respect to intensity are classified as dark gray. There is. In another example, the frequency range with an intensity of up to one-third of the maximum intensity is shown in white, and the frequency range with an intensity between one-third and two-thirds of the maximum intensity is shown in light gray. And the frequency range with an intensity between two-thirds of the maximum intensity and the maximum intensity is shown in dark gray. In one example, the toning technique shown in the displayed frequency information may also be used for one or more indicators present in the acoustic image data. In general, either colorization techniques or other visualization techniques (eg, via various patterns, transparency, etc.) may be used.

In one example, the frequency range may be toned with respect to the severity of the detected acoustic data in each frequency range. For example, in one example, the frequency range shown in dark gray is considered to be extremely tight, the frequency range shown in light gray is considered to be moderately tight, and the frequency range shown in white is of minor significance. Is considered to indicate. Similar to the above, in some examples, one or more indicators present in the acoustic image data may include similar toned severity indications at one or more positions in the acoustic scene.

FIG. 45 shows an exemplary display image including a multi-parameter display 4540 showing different frequency ranges and indicators 4510, 4512, 4514 toned according to severity. In various examples, colors corresponding to different severity levels may be set automatically by the device or manually by the user.

In some cases, frequency intensity data may be stored or tracked over time, as described elsewhere herein. In certain embodiments, intensity vs. time information may be displayed in each of the frequency ranges of one or more of the multi-parameter displays. FIG. 46 shows the intensity (dB display) vs. time trend in each of the plurality of frequency ranges (eg, 4650) within the multi-parameter display 4640. For example, in one example, in addition to intensity vs. time, peak intensity information is displayed on a multi-parameter display 4640 in one or more frequency ranges, such as a peak marker (eg, 4652). There is. As shown in FIG. 46, the multi-parameter display 4640 includes a time axis and intensity information corresponding to an acoustic frequency or acoustic frequency range over time in each of a plurality of acoustic frequencies or acoustic frequency ranges (eg, 4650). Is displayed.

The various processes described herein can be embodied as non-transitory computer-readable media containing executable instructions for causing such processes to execute one or more processors. The system may include, for example, one or more processors configured to perform such processes, either integrally with the processor or based on instructions stored in memory outside the processor. In some cases, various components can be distributed throughout the system. For example, a system can include multiple distributed processors, each configured processor performing at least a portion of the entire process performed by the system. Further, a single acoustic imaging system in which the various features and functions described herein are embodied, for example, as a handheld acoustic imaging tool or a distributed system with various separate and / or separable components. It will be understood that it can be combined with.

The various functions of the components described herein can be combined. In one embodiment, the feature described in this application is entitled "Systems and Methods for Projecting and Projecting Acoustic Data (SYSTEMS AND METHODS FOR PROJECTING AND DISPLAYING ACOUSTIC DATA)" and is transferred to the transferee of this application. , With reference to attorney reference number 56581.178.2, incorporated herein by reference, which can be combined with the features described in the PCT application filed July 24, 2019. In one embodiment, the feature described in this application is entitled "Systems and Methods for Tagging and Associating Acoustic Images (SYSTEMS AND METHODS FOR TAGGING AND LINKING ACOUSTIC IMAGES)" and is transferred to the transferee of this application. It has attorney reference number 56581.179.2 and is incorporated herein by reference and can be combined with the features described in the PCT application filed July 24, 2019. In certain embodiments, the feature described in this application is entitled "Systems and Methods for Detachable Acoustic Image Sensors (SYSTEMS AND METHODS FOR DETACHANBLE AND ATTACHABLE ACOUSTIC IMAGING SENSORS)" and the transferee of this application. It has attorney reference number 56581.180.2, which has been transferred to and incorporated herein by reference, and can be combined with the features described in the PCT application filed July 24, 2019. In certain embodiments, the feature described in this application is entitled "Systems and Methods for Displaying Acoustic Signatures from Target Scenes (SYSTEMS AND METHODS FOR REPRESENTING ACOUSTIC SIGNATURES FROM A TARGET SCENE)". It has attorney reference number 56581.182.2, which has been transferred to a person and incorporated herein by reference, and can be combined with the features described in the PCT application filed July 24, 2019. ..

Various embodiments have been described. Such examples are non-limiting and do not limit or limit the scope of the invention in any way.

In the illustrated embodiment, the acoustic analysis system 200 includes an electromagnetic imaging tool 203 for generating image data representing a target scene. An exemplary electromagnetic imaging tool may be configured to receive electromagnetic radiation from a target scene and generate electromagnetic image data representing the received electromagnetic radiation. In one example, the electromagnetic imaging tool 203 may be configured to generate electromagnetic image data representing a range of wavelengths within the electromagnetic spectrum, such as infrared radiation, visible light radiation, and ultraviolet radiation. For example, in one embodiment, the electromagnetic imaging tool 203 is one or more cameras configured to generate image data representing a range of wavelengths within the electromagnetic spectrum, such as the visible light camera module 206. Can include modules.

Similarly, an image with corresponding pixels (ie, pixels representing the same part of the target scene) may be referred to as the corresponding image. Thus, in some such arrangements, the corresponding visible light and acoustic images can be superimposed on each other at the corresponding pixels. The operator may contact the user interface 216 to control the transparency or opacity of one or both of the images displayed on the display 214. For example, the operator may contact the user interface 216 to adjust the acoustic image between fully transparent and completely opaque, and the visible light image between completely transparent and completely opaque. Such an exemplary combinational arrangement, which may be referred to as an alpha blend arrangement, adjusts the display 214 to the operator to make the acoustics of any overlapping combination of the two images between the extremes of the acoustic image only and the visible light image only images. It may be possible to display an image of only an image or an image of only a visible light image. Processor 212 can also combine scene information with other data, such as alarm data. In general, alpha-blended combinations of visible and acoustic images can be included anywhere from 100 % acoustic and 0% visible light to 0% acoustic and 100% visible light. In certain embodiments, the amount of blend can be adjusted by the user of the camera. Thus, in some embodiments, blended image can be adjusted between 100% of the visible light and 100% of the acoustic.

The acoustic analysis system of FIG. 3A further includes a distance measuring tool 304 and a camera module 306 disposed within the acoustic sensor array 302. The camera module 306 may represent a camera module of an electromagnetic imaging tool (for example, 203), and may include a visible light camera module, an infrared camera module, an ultraviolet camera module, and the like. Further, although not shown in FIG. 3A, the acoustic analysis system may include one or more additional camera modules of the same type or different type as the camera module 306. In the illustrated example, the distance measuring tool 304 and the camera module 306 are arranged in a grid of acoustic sensor elements of the first array 320 and the second array 322. Although shown to be located between the grid sites in the first array 320 and the second array 322, in certain embodiments, one or more components (eg, camera module 306 and / or distance). The measuring tool 304 ) can be placed at the corresponding one or more grid sites within the first array 320 and / or the second array 322. In certain such embodiments, the components (s) can be placed at the grid site instead of the acoustic sensor elements that would typically be in such positions according to the grid layout.

Similar to the systems of FIGS. 3A and 3B, the system of FIG. 3C includes an acoustic sensor array 392, 394, and a distance measuring tool 314 and a camera array 316 disposed within 396. In one example, additional components (eg, used to image different parts of the electromagnetic spectrum from the camera array 316), such as an additional camera array, are similarly within the acoustic sensor arrays 392, 394, and 396. Can be placed in. Figure 3A~ in Figure 3 C is one or more are shown to be placed in acoustic sensor array, the distance measuring tool and / or one or more of the imaging tool (e.g., a visible light camera module , Infrared camera module, UV sensor, etc.) can be placed outside the acoustic sensor array (s). In one such example, the distance measurement tool and / or one or more imaging tools located outside the acoustic sensor array (s) are provided by the acoustic imaging tool, eg, a housing that houses the acoustic sensor array. Can be supported by or can be placed outside the housing of the acoustic imaging tool.

Filtering acoustic image data by frequency during an exemplary acoustic imaging operation can help reduce image clutter, for example from the background or other non-essential sounds. In an exemplary acoustic imaging procedure, the user may wish to remove background sounds such as floor noise in an industrial environment. In one such example, background noise may primarily include low frequency noise. Therefore, the user may choose to display acoustic image data that represents an acoustic signal that is greater than a predetermined frequency (eg, 10 kHz). In another example, a user wants to analyze a particular object that generally emits an acoustic signal within a particular range, such as a corona discharge from a transmission line (eg, as shown in FIGS. 5A- 5D). There is. In such an example, the user may select a specific frequency range (eg, 11 kHz to 14 kHz for corona discharge) for acoustic imaging.

In an exemplary scenario, 10A may include all of the acoustic image data above the noise floor threshold at each position 1010 to 1090, while FIG 10B shows the same scene as in FIG. 10A, a 40dB Only acoustic image data with greater intensity is shown. This helps the user identify which sound source in the environment (eg, in the target scene of FIGS. 10A and 10B) contributes to a particular sound (eg, the loudest sound in the scene). Enables.

In addition to, or instead of, circular display between images showing acoustic image data that meet certain conditions, in one example, the location of the acoustic signal source is detected in the acoustic image data and the acoustic image is separate from the other acoustic signals. Can be displayed in the data. For example, with respect to FIG. 10A, in certain embodiments, acoustic image data representing acoustic signals generated from each of positions 1010 to 1090 may be identified and cyclically displayed. For example, in an exemplary operation process, the display image including the sound image data at one of the positions 1010 to 1090, for the individual analysis of each source of acoustic signals, at the direction of automatic or user, Can be circularly displayed. In various embodiments, the order in which different positions of the acoustic image data are displayed during the cyclic display can depend on various parameters such as position, proximity, intensity, frequency content, and the like.

In one example, the laser pointer can be visualized on the display. Such visualization can include generating a laser pointer spot on the display display of the laser pointer in a real scene (eg, via a processor). In one example, the position of the laser pointer is highlighted on the display by, for example, an icon representing the laser pointer in the scene or another aligned display marker to better determine the position on the display with respect to the actual scene. Can be done.

FIG. 14 shows the visualization of acoustic data using a gradation toning technique. As shown, acoustic parameters (eg, intensity) are shown via a gradation toning technique. In an exemplary gradation scheme, a parameter value has a unique color associated with it according to the toning technique. A change in a parameter value at a given pixel usually results in a color change associated with that pixel, representing a new parameter value. As shown in the example of FIG. 14, the acoustic parameter values appear to vary radially from the center position of the acoustic signal at positions 1432, 1434, 1436. Other examples of gradation toning are described in US Patent Application No. 15 / 802,153, filed November 2, 2017 and transferred to the transferee of this application.

FIG. 15 shows a visualization of acoustic data using a plurality of shaded concentric circles. In one such example, each identical color of the concentric circles shown in FIG. 15 has a value associated with that color, as opposed to a gradation scheme with a color associated with the parameter value. Can represent pixels with acoustic parameter values within the range of. In the illustrated example, the acoustic parameters (eg, intensity) associated with the acoustic signal at positions 1532, 1534, 1536 vary radially from the center of the acoustic signal. In an exemplary toning technique, the pixels shown in red represent acoustic parameter values within the first range of parameter values, and the pixels shown in yellow represent acoustic parameter values within the second range of parameter values. Represented and represented in green, the pixels represent acoustic parameter values within a third range of parameter values, but other display techniques including additional or alternative colors, patterns, etc. are possible. In various embodiments, the range of values can correspond to an absolute range, such as an intensity value between 10 dB and 20 dB, or a relative range, such as an intensity value between 90% and 100% of maximum intensity. could be.

In certain embodiments, a colored indicator may be used to indicate the severity of one or more detected acoustic signals and / or associated acoustic parameters such as deviations from reference parameters. FIG. 20 shows an exemplary visualization showing a plurality of indicators 2002, 2004, 2006 with different colors indicating the severity indicated by the acoustic signal from the corresponding position. For example, in an exemplary embodiment, a red indicator indicates a critical severity based on one or more acoustic parameters (eg, when compared to criteria such as typical operating condition criteria) and is yellow. indicator showed moderate severity, green indicator represents a mild severity. In other examples, other color schemes or appearance characteristics (eg, indicator transparency, indicator size, etc.) may be used to visually distinguish the severity of the acoustic signal. In the illustrated example, indicator 2004 represents the highest level of severity, indicator 2006 represents the next highest level, and indicator 2002 represents the least severe acoustic signal.

In one example, such an embedded camera may be configured to produce electromagnetic image data that can be combined with acoustic image data for a display, as described elsewhere herein. be. In certain embodiments, the acoustic imaging system stores various information, such as acoustic data, acoustic image data, electromagnetic image data, and annotation data (eg, sensor data, image / video annotation data, etc.), along with a time stamp. It may be configured as follows. In one example, the acoustic imaging system may be configured to display and / or record relevant information both on the device and later in software. The relevant information may be displayed on the display along with the stored display image or acoustic image file and / or stored as metadata, for example.

Claims (39)

An acoustic sensor array composed of a plurality of acoustic sensor elements, wherein each acoustic sensor element receives an acoustic signal from an acoustic scene and outputs acoustic data based on the received acoustic signal.
An electromagnetic imaging tool configured to receive electromagnetic radiation from the target scene and output electromagnetic image data indicating the received electromagnetic radiation.
User interface and
With the display
A processor that communicates with the acoustic sensor array, the electromagnetic imaging tool, the user interface, and the display, receives electromagnetic image data from the electromagnetic imaging tool, receives acoustic data from the acoustic sensor array, and receives the reception. Generates acoustic image data of the scene based on the acoustic data, generates a display image consisting of the combined acoustic image data and electromagnetic image data, provides the display image to the display, and inputs an annotation from the user interface. A processor configured to receive and update the display image on the display based on the received annotation input.
Acoustic analysis system consisting of.
The system of claim 1, wherein the user interface comprises a touch screen and the received annotation input comprises a freestyle annotation received via the touch screen.
Further, a memory that communicates with the processor and is composed of one or more icons, and the received comment input is used to select an icon stored in the memory and to determine the position of the icon on the display image. The system according to claim 1 or 2, which comprises an input to be set.
Claim 1 the annotation input comprises a selection of a predefined shape, an input for setting the size of the predefined shape, and an input for setting the position of the predefined shape in a display image. The system according to any one of 3 to 3.
The processor is configured to detect the position of an acoustic signal present in the acoustic scene.
Updating the display image on the display based on the received comment input includes an indicator located in the position of the acoustic signal in the display image and the comment input received in relation to the transparency of the indicator. The system according to any one of claims 1 to 4, wherein the transparency of the indicator is adjusted accordingly in the display image.
The system according to claim 5, wherein adjusting the transparency of the indicator in the display image comprises adjusting the blend ratio according to the blend of the electromagnetic imaging data and the acoustic image data.
The processor is configured to detect the location of an acoustic signal present in the acoustic scene and determine acoustic parameters in relation to the acoustic signal.
The received annotation input comprises receiving a selection of a range of numbers in relation to acoustic parameters.
Updating the display image on the display based on the received comment input is the position of the acoustic signal only when the acoustic parameter associated with the acoustic signal is within the numerical range selected in relation to the acoustic parameter. The system according to any one of claims 1 to 6, comprising an indicator located in the display image.
The received comment input comprises receiving a selection of a plurality of numerical ranges related to the acoustic parameter, and updating the display image on the display based on the received comment input is acoustic associated with the acoustic signal. Claims configured to include an indicator located at the location of an acoustic signal in the display image only when the parameter is within any of a plurality of numerical ranges selected in relation to the acoustic parameter. Item 7. The system according to Item 7.
The processor is configured to detect the location of an acoustic signal present in the acoustic scene and determine the distance to the target in relation to the acoustic signal.
The display image comprises an indicator located at the position of the acoustic signal.
Updating the display image on the display based on the received annotation input comprises an annotation indicating the distance to the target associated with the acoustic signal near the indicator located at the location of the acoustic signal. The system according to any one of claims 1 to 8.
As the processor determines one or more acoustic parameters associated with an acoustic signal and determines the critical value associated with the acoustic signal based on a comparison of the acoustic parameters against a single predetermined threshold or a plurality of predetermined thresholds. Consists of
Updating the display image on the display based on the received annotation input further indicates a critical value and one or more acoustic parameters in relation to the acoustic signal, near the indicator located at the location of the acoustic signal. The system of claim 9, comprising commentary.
It also comprises a laser distance sensor consisting of a laser pointer configured to fire a laser towards the scene and provide the processor with information indicating the distance to the target scene.
The system of claim 9 or 10, wherein the display image further comprises a laser indicator indicating the position of the laser pointer within the target scene.
The system according to any one of claims 1 to 11, wherein the annotation input comprises an alphanumerical input defining an alphanumerical annotation and an input for setting the position of the alphanumerical annotation in a display image.
12. The system of claim 12, wherein the alphanumerical input comprises selecting an alphanumerical annotation from a predefined list of alphanumerical annotation options.
The processor is configured to detect a plurality of acoustic signals in an acoustic scene and determine a first acoustic parameter in relation to each of the detected acoustic signals.
Receiving an annotation input from the user interface consists of receiving an instruction to update the display image and include information representing the first acoustic parameter.
Claims 1 to include updating the display image based on the received annotation input including instructions on the display image representing a first acoustic parameter determined in relation to each of the detected acoustic signals. 13. The system according to any of 13.
The indication on the display representing the first acoustic parameter determined in relation to each of the plurality of detected acoustic signals consists of an isoacoustic indicator located at the position of each acoustic signal and the first acoustic. The system according to claim 14, which comprises a single color representing a particular numerical value or numerical range associated with a parameter.
15. The system of claim 15, wherein the single color of each isoacoustic indicator represents the critical level of the first acoustic parameter.
The processor is configured to determine a second acoustic parameter in relation to each of the plurality of detected acoustic signals, with each equal acoustic indicator being the first acoustic parameter and the second acoustic parameter. The system of claim 15 or 16, which represents a particular numerical value or range of.
The processor is configured to determine a second acoustic parameter in relation to each of the detected acoustic signals, and the instructions indicate for each of the detected acoustic signals. The system according to any one of claims 14 to 17, comprising an alphanumeric display of a first acoustic parameter and a second acoustic parameter associated with the acoustic signal.
The indication is an indicator placed on the display image at a position in the acoustic scene corresponding to the detected acoustic signal for each of the plurality of detected acoustic signals, the first acoustic parameter of the acoustic signal. An indicator with visualization characters that represent
The system according to any one of claims 14 to 18, comprising a plurality of parameter displays indicating the relationship between the first acoustic parameter and the second acoustic parameter in the acoustic scene.
19. The system of claim 19, wherein the multi-parameter display comprises a time axis showing the evolution of the relationship between the first acoustic parameter and the second acoustic parameter over time.
19. The system of claim 19 or 20, wherein the multi-parameter display comprises a graphical display of acoustic intensity in relation to a plurality of frequencies or frequency ranges within an acoustic scene.
Further comprising a memory containing one or more sound profiles, each of which contains a sound profile defined by one or more acoustic parameters corresponding to one or more sounds.
The processor analyzes the received acoustic data, and if the received acoustic data matches one of the sound profiles stored in the memory, annotates the display image with the annotation associated with the sound profile.
The system according to any one of claims 1 to 21, wherein the received annotation input comprises an instruction to detect a sound profile present in the acoustic scene.
22. The system of claim 22, wherein the annotation comprises instructions for detection conditions of an acoustic scene based on a matched sound profile.
23. The system of claim 23, wherein the detected condition comprises an air leak.
24. The system of claim 24, wherein the instructions consist of alphanumerical annotations of air leaks.
25. The system of claim 25, wherein the processor is further configured to determine a cost associated with an air leak, the alphanumerical annotation comprising displaying the cost associated with the air leak.
The system according to any one of claims 24 to 26, wherein the instruction comprises an icon annotation representing an air leak.
An acoustic sensor array composed of a plurality of acoustic sensor elements, wherein each acoustic sensor element receives an acoustic signal from an acoustic scene and outputs acoustic data based on the received acoustic signal.
A memory containing a sound profile comprising a first acoustic parameter and a first predetermined condition associated with the first acoustic parameter.
With the display
Whether or not the processor communicates with the acoustic sensor array, the display, and the memory, receives acoustic data from the acoustic sensor array, and the received acoustic data matches the sound profile stored in the memory. And if the received acoustic data matches the sound profile stored in memory, the received acoustic data via an instruction on the display that consists of information about the matched sound profile. An acoustic analysis system that shows that is matched with the sound profile stored in memory.
Determining whether the received acoustic data matches the sound profile is whether the received acoustic data contains an acoustic signal that satisfies the first predetermined condition associated with the first acoustic parameter. 28. The system of claim 28.
28. The system of claim 28 or 29, wherein the sound profile comprises a second acoustic parameter and a second predetermined condition associated with the second acoustic parameter.
The received acoustic data is the case where the received acoustic data includes an acoustic signal satisfying the first predetermined condition related to the first acoustic parameter and the second predetermined condition related to the second acoustic parameter. 30. The system of claim 30, wherein the acoustic data matches the sound profile.
The first acoustic parameter consists of frequency, the first predetermined condition consists of frequency range, the second acoustic parameter consists of intensity, the second predetermined condition consists of threshold intensity, and the reception. 31. The system of claim 31, wherein the received acoustic data matches a sound profile when the resulting acoustic data includes a frequency within the frequency range and an acoustic signal having an intensity greater than the threshold intensity.
The received acoustic data includes a first acoustic signal satisfying the first predetermined condition related to the first acoustic parameter and two predetermined conditions second acoustic signal related to the second acoustic parameter. The system according to any one of claims 30 to 32, wherein the received acoustic data matches the sound profile, if any.
An acoustic sensor array composed of a plurality of acoustic sensor elements, wherein each acoustic sensor element receives an acoustic signal from an acoustic scene and outputs acoustic data based on the received acoustic signal.
An electromagnetic imaging tool configured to receive electromagnetic radiation from a target scene and output electromagnetic image data indicating the received electromagnetic radiation, and an electromagnetic imaging tool configured to detect electromagnetic radiation.
With the display
A processor that communicates with the acoustic sensor array, the electromagnetic imaging tool, the front, and the display, receives electromagnetic image data from the electromagnetic imaging tool, receives acoustic data from the acoustic sensor array, and receives the received acoustics. It comprises a processor configured to generate acoustic image data of a scene based on the data, generate a display image consisting of the combined acoustic image data and electromagnetic image data, and provide the display image to the display. ,
The display image is an indicator located on the display at a position within the acoustic scene corresponding to the detected acoustic signal, the indicator having visualization characters representing the acoustic parameters of the acoustic signal, and a plurality of acoustic frequencies or acoustics. An acoustic analysis system that includes a multi-parameter display that contains intensity information that represents the acoustic data received for each of the frequency ranges.
34. The system of claim 34, wherein the multi-parameter display includes a time axis and, for each of a plurality of acoustic frequencies or acoustic frequency ranges, represents intensity information corresponding to the acoustic frequency or acoustic frequency range over time. ..
Further, a user interface for contacting a processor is provided, and the processor receives a selection of one or more acoustic frequencies or acoustic frequency ranges via the user interface, and the selected one or more. It is configured to determine a position in an acoustic scene that has an acoustic signal containing an acoustic frequency or frequency component within the acoustic frequency range of.
30. The system described in.
The system of any of claims 34-36, wherein the multi-parameter display comprises displaying the current intensity value indication and the maximum acoustic intensity value for each of the plurality of acoustic frequencies or acoustic frequency ranges.
37. The system of claim 37, wherein the maximum acoustic intensity value represents the maximum acoustic intensity observed over a predetermined time for each of the plurality of acoustic frequencies or acoustic frequency ranges.
34- 36. The system according to any one of 36.

Images