JP2024055341A - Acoustic inspection method and device - Google Patents

Abstract

[Problem] In an inspection for abnormal sounds such as horn sounds, when an abnormal sound is judged to be present, a post-facto verification can be performed. [Solution] The inspection device has a sound acquisition unit including a microphone, and an image acquisition unit consisting of a video camera that acquires an image seen from the front of the vehicle. If it is judged to be a horn sound, sound data of a length of about several seconds is cut out and frequency analyzed to judge whether the sound is normal or abnormal. If it is an abnormal sound, image data of the same length of several seconds is cut out and saved together with the sound data. On the display, sound data 70b including a time indicator 70b1, a spectrum 70c, a table 70g of the judgment results, an image 70e of the entire vehicle, and an enlarged image 70f of the driver's seat area for checking the operation of the horn switch are all displayed together. The sound is played in response to a playback request, and the enlarged image 70f is played in synchronization with this. [Selected Figure] FIG. 3

Description

This invention relates to a method and device for acoustic inspection, which performs sound-based inspections, for example as part of an automobile inspection process.

For example, in the finished vehicle inspection process, which is the final stage of an automobile production line, an inspector test drives the finished vehicle to be inspected on free rollers and inspects a number of items, including the engine, meters, brakes, horn, and lights. Generally, during this inspection process, the presence or absence of abnormalities is determined for various sounds emitted from various parts of the vehicle through what is called a sensory evaluation based on the inspector's senses. For example, an inspection is conducted in which the inspector sounds the horn and listens to the sound himself to confirm that it is normal. Sounds made from the wiper blades when the wipers are in operation and sounds made from the brakes when the brakes are applied are also subject to sensory inspection.

Instead of such sensory testing, attempts have been made in the past to detect abnormalities by capturing sound with a microphone and analyzing the signal.

Patent Document 1 discloses an abnormal sound diagnosis system that, on a belt-like production line along which inspection objects such as automobile engines are conveyed in a line, acquires the sound generated by trial operation of moving parts of the inspection objects as a measurement sound and compares this measurement sound with a reference sound to determine whether the sound is normal or not. An inspector positioned in front of the production line wears a head-mounted display equipped with a microphone, and is able to see the result of the determination of whether the sound is normal or not on the head-mounted display, as well as visually confirm the measurement range superimposed on the inspection object. When an abnormal sound is displayed on the head-mounted display, the inspector attaches a sticker to the inspection object indicating that it is a defective product.

JP 2010-197361 A

In many types of sound inspections, such as horn inspections during the finished vehicle inspection process, if an abnormality is determined, so-called "traceability" is required, which allows the inspection to be verified after the fact. For example, when inspecting horn sounds, it is necessary to verify whether the inspector sounded the horn correctly. It is possible that the sound of a horn from an adjacent inspection line may be mixed in at a different time from the inspector's horn operation, resulting in the sound being mistakenly determined to be abnormal.

The technology in Patent Document 1 does not take into consideration any post-mortem verification of such test results.

The acoustic inspection method according to the present invention comprises the steps of:
Acquire the sound emitted from the device to be inspected and generate sound data including the target sound;
Acquire video of the inspection target device in parallel with acquiring the sound, and generate video data including the operation or movement that caused the target sound;
Analyzes the sound data to determine whether the target sound is an abnormal sound,
When it is determined that the sound is at least an abnormal sound, in response to a playback request, the sound data is played back through a speaker, and time-series sound data including an indicator synchronized with the playback, the analysis results of the sound data, and a playback image of the video data synchronized with the playback of the sound data are all displayed on a single screen.

For example, while acquiring sound emitted from the equipment to be inspected, such as a vehicle, via a microphone, video of the equipment to be inspected, such as a vehicle, is acquired, for example, via one or more cameras. Sound data including the target sound, such as a horn sound, is generated and analyzed to determine whether it is an abnormal sound. On the other hand, if it is a horn sound, for example, video data is generated that allows visual confirmation of the operation by the inspector that caused the horn sound (pressing the horn switch on the steering wheel).

For example, when playback is requested for post-event verification, sound data containing the target sound (e.g., a horn sound) is played through a speaker, and at the same time, a display of time-series sound data including an indicator synchronized with the playback, an analysis result of the sound data, and a playback image of video data synchronized with the playback of the sound data are all displayed on the screen. For example, in the case of a horn sound, video data showing the pressing of a horn switch is displayed synchronized with the playback of the sound data. The sound data is also visually displayed as a time-series sound pressure waveform, etc., and the display of this sound data includes an indicator synchronized with the playback, allowing the user to visually see where in the time-series sound data the sound currently being heard is.

According to this invention, in response to a playback request, sound data is played through a speaker, and a time-series display of sound data including an indicator synchronized with the playback, the analysis results of the sound data, and a playback image of video data synchronized with the playback of the sound data are all displayed together, making it easy to verify the sound test after the fact.

FIG. 1 is a functional block diagram of a first embodiment in which the present invention is applied to abnormal noise inspection in a finished vehicle inspection process. 4 is a flowchart showing a process flow according to the first embodiment. FIG. 4 is an explanatory diagram showing a display example on a display unit according to the first embodiment. FIG. 11 is a functional block diagram of a second embodiment. 10 is a flowchart showing a process flow according to a second embodiment. FIG. 11 is an explanatory diagram showing a display example on a display unit according to a second embodiment. FIG. 11 is an explanatory diagram of an enlarged image for a wiper sound that is switched and displayed. FIG. 11 is an explanatory diagram showing another display example. FIG. 11 is an explanatory diagram showing still another display example.

Below, an embodiment of the present invention applied to abnormal sound inspection in the finished vehicle inspection process of automobiles is described. Generally, in the finished vehicle inspection process, which is the final stage of an automobile production line, an inspector test drives the finished vehicle to be inspected on free rollers, and inspects a number of items including the engine, transmission, meters, horn, brakes, windshield wipers, etc. according to a predetermined inspection sequence. The inspection device of one embodiment acquires sounds from the vehicle without relying on timing signals from the vehicle, and uses the acquired sounds to extract target sounds and determine whether they are abnormal sounds (also called abnormal sounds), and displays these results together with images.

In one embodiment, the sounds to be inspected include the horn and wipers. For the horn, the inspector presses the horn switch on the steering wheel to sound the horn, and the sound is used to determine whether the horn is normal. Anomalies such as a different horn pitch or strange tone can occur due to anomalies in the horn itself, an incorrect model number for the part (horn), poor wiring contact, anomalies in the horn switch, etc., so the inspection is based on the actual sound of the horn. Similarly, for the wipers, the inspector operates the wipers and inspects whether any abnormal sounds are generated, for example, from near the wiper blades.

Figure 1 shows a functional block diagram of the inspection device of the first embodiment. Note that this inspection device is not an independent device as a horn/wiper inspection device, but is configured as part of a completed vehicle inspection device. The inspection device of the first embodiment is configured to include a sound acquisition unit 1, a video acquisition unit 10, a target sound identification unit 20, an abnormal sound determination unit 30, a sound/video data generation unit 40, a sound/video data storage unit 50, a video display control unit 60, and a display unit 70 including a speaker.

The sound acquisition unit 1 includes a microphone that acquires sounds generated from the vehicle under test and converts them into electrical signals, i.e., sound data, and a recording unit that temporarily stores this sound data. The microphone is placed outside the vehicle so that it can collect sounds from the vehicle, including sounds from the horn, windshield wipers, etc. The microphone's directivity, sampling frequency, frequency band, sensitivity characteristics, etc. are selected according to the measurement target and sound field environment. Typically, a microphone with directionality toward the vehicle is used. Sound data from which noise sounds have been removed may be obtained by localizing the sound source using a microphone array or the like.

The image acquisition unit 10, which is configured, for example, by a video camera, captures images of the vehicle to be inspected in parallel with the acquisition of sounds, and converts them into image data (i.e., video data). A video camera with a large number of pixels, such as the full HD standard (1920 x 1080 pixels), is preferable, in order to enable partial enlargement of the acquired image. The video camera that constitutes the image acquisition unit 10 may be a single unit or multiple units that capture images in sync with each other. In one embodiment, a single video camera is used that captures an overhead image of the vehicle to be inspected from the front (more specifically, diagonally above).

The target sound identification unit 20 extracts the target sound (e.g., a horn sound) from the continuous sound data acquired by the sound acquisition unit 1 in substantially real time, and cuts out sound data including the target sound. For example, if the test target is a horn, it focuses on a sound in a frequency band of, for example, 1 kHz as a frequency band that is characteristic of the horn sound, and identifies the target sound based on whether or not its sound pressure is 100 dBA or more. Then, it generates sound data of a length of, for example, several seconds that includes the sound before and after the horn sound.

The abnormal sound determination unit 30 determines whether the target sound contained in the generated sound data of a few seconds in length is an abnormal sound. For example, the sound data is quantified using frequency analysis techniques such as FFT (Fast Fourier Transform) or wavelet analysis, and the degree of abnormality from a normal sound is calculated.

For example, the data may be converted by frequency analysis into a two-dimensional frequency characteristic (spectrum) with the horizontal axis representing frequency and the vertical axis representing sound pressure or power, or all of the extracted data may be converted into frequency characteristics all at once, or data may be extracted at a predetermined data length (time window) and overlapped in time and slid to calculate the frequency characteristics. Furthermore, the data may be multiplied by A-characteristics or C-characteristics, which are frequency characteristics of human hearing sensitivity, to obtain frequency characteristics that match the hearing sensitivity. Sound data may also be quantified by processing a three-dimensional spectrogram consisting of time, frequency, and sound pressure, or the Mel-cepstrum used in voice recognition, etc.

After performing this processing, the feature quantities that characterize normal and abnormal sound data, such as the XY coordinates of the FFT center of gravity, the spectrum envelope and its inclination angle, and other numerical values, are defined to calculate the degree of abnormality.

The degree of abnormality can be calculated by calculating the average value or variance of the numerical values specified as the features of the abnormal sound to obtain a statistical outlier (= degree of abnormality), or by vectorizing two or more dimensional features and using the cosine similarity, which is indicated by the size of the angle between them.

Then, a threshold is set for determining whether the level of abnormality is normal or abnormal, and the sound is quantitatively determined to be normal or abnormal depending on whether the calculated level of abnormality exceeds the threshold.

Other methods include, for example, linearly transforming multiple values using multivariate analysis to convert the multivariate into a one-dimensional value with a high contribution rate (anomaly level Z = A x feature A + B x feature B + C x feature C, etc.), and then setting a threshold value to determine the degree of anomaly.

When the abnormal sound determination unit 30 determines that a sound is abnormal, the sound/video data generation unit 40 saves the sound data, which includes the target sound (horn sound) and has a length of about several seconds, as described above, in an appropriate format, for example, a WAVE file. Then, video data is cut out to a length range similar to that of the sound data, and saved as a video file of an appropriate format of about several seconds, for example, an AVI file. These WAVE files and AVI files are saved as mutually linked data in the sound/video data storage unit 50, which serves as a database. For example, they are collected into one folder (or may be one compressed file) and stored in the sound/video data storage unit 50. Additional information such as a text file of vehicle information, analysis results of the sound data (spectrum, spectrogram, etc.), and a text file of detailed results when the abnormal sound was determined are added to this folder. Information such as vehicle information and determination results may also be added to the file name when it is saved.

This data allows for proper verification of abnormal sound data after measurement. Furthermore, this data can also be used as learning data and verification data for AI such as deep learning.

The video display control unit 60 generates images or videos to be displayed on the display unit 70 for the results obtained as described above, and further generates sound signals to be emitted from a speaker attached to the display unit 70. The display unit 70 is a display means for displaying the generated images or videos to relevant parties such as the inspector driving the vehicle, the production manager, and the data scientist who utilizes the data. For example, it is composed of an LCD display, an organic EL display, a tablet, an HMD (Head Mounted Display), a smart watch, etc. It also includes a sound source, an amplifier, a speaker, etc. for emitting sound.

When the video display control unit 60 determines that the sound is at least an abnormal sound, as described below, in response to a playback request, it plays back the sound data through the speaker, and displays the time-series sound data including an indicator synchronized with the playback, the analysis results of the sound data, and the playback image of the video data synchronized with the playback of the sound data all together on one screen of the display unit 70. In particular, for the video data, in order to make the operation or movement that caused the target sound more clearly visible, a part of the video is cut out, enlarged or reduced, tilted or rotated, and the image quality (color, sharpness, contrast, brightness) is adjusted or corrected. For example, if the target sound is a horn sound, in the image captured by the video capture unit 10 looking down on the front of the vehicle, the driver's seat area is cut out and enlarged so that the inspector can see that the horn switch on the steering wheel is pressed, and a video for display with high contrast and brightness is created to compensate for the darkness of the interior of the vehicle. Also, to protect the privacy of the inspector, the inspector's face is pixelated or replaced with an avatar image, preventing the individual from being identified. In one embodiment, such a partially enlarged image is displayed together with the overall image of the vehicle, and is displayed in the form of a speech bubble so that the enlarged portion of the overall image can be identified.

Figure 3 is an explanatory diagram showing a display example on the display unit 70 of the first embodiment. This display is generated using data from various files in one folder stored in the sound/video data storage unit 50 described above. In this example, at the top of the display screen, "(1) Sound type" indicates that it is a horn sound, and displays the vehicle model and management number to identify the vehicle (see reference symbol 70a). In the middle, "(2) Sound pressure/frequency characteristics" displays sound data (see reference symbol 70b) including a target sound with time on the horizontal axis and sound pressure on the vertical axis, and frequency characteristics, i.e., spectrum (see reference symbol 70c), obtained by frequency analysis of the sound data, side by side. Here, the display of the sound data (70b), which is time-series data, includes an indicator (see reference symbol 70b1), for example, a red straight line for displaying time that moves in the horizontal axis in synchronization with the playback, so that it is possible to visually confirm which part of the sound data is being played during the playback of the sound.

In addition, at the bottom of the screen, as "(3) Video display", an operation panel section (see reference symbol 70d) including a play button, a rewind button, etc., an overall image of the vehicle seen from above from the front (see reference symbol 70e), and an enlarged image of the driver's seat area superimposed on the overall image in the form of a speech bubble (see reference symbol 70f) are displayed. Note that while the inspector is shown as an illustration in Figure 3, in reality, the inspector is photographed and displayed as is, except for his face.

In the lower right corner of the screen, "(4) Sound Judgment Results" is displayed in the form of a text data table showing the blowing time, sound pressure, abnormal volume, judgment threshold, judgment results, etc. (see symbol 70g).

On the initial screen (e.g. immediately after the end of an inspection), the video data (70e, 70f) are still images or are not displayed. When an administrator verifying the test operates the play button on the operation panel (70d) via an input means such as a mouse, sound data having a length of about several seconds is played through the speaker. At the same time, the indicator (70b1) in the sound data (70b) displayed on the screen starts to move to the right in the figure, allowing the location where the sound is being played to be visually confirmed. At the same time, playback of the overall image (70e) and the driver's seat enlarged image (70f) starts. These videos (70e, 70f) and sounds are played in sync with each other.

In addition, if an abnormal sound is detected, sound playback or video playback (70e, 70f) may be automatically started the first time immediately after the inspection is completed so that the inspector can verify or confirm the sound himself.

According to the display and sound playback as shown in FIG. 3, the person performing the inspection can visually confirm that the sound is indeed an abnormal sound by listening to the actual horn sound while looking at the sound data (70b), spectrum (70c), and the table (70g) of "(4) Sound Judgment Result". The operation of the horn switch by the inspector can be visually confirmed by the driver's seat enlarged image (70f), which is a video played in synchronization with the sound playback, and the sound data (70b) can be visually confirmed and the sound played from the speaker can be listened to while comparing with the timing of the operation. For example, if a sound is mistaken for a horn sound at a time that is clearly different from the timing of the inspector's horn switch operation and judged to be an abnormal sound, it is easy to determine that this abnormal sound judgment is incorrect by playing the video showing the operation of the horn switch in synchronization. Also, for example, if the sound pressure level of the horn sound is extremely low, it is possible to verify after the fact whether the inspector really operated the horn switch.

In the above embodiment, the image acquisition unit 10 is a single video camera, and a portion of the image is cut out and enlarged, making it relatively easy to synchronize with the sound playback. If multiple video cameras are used, it is also possible to add an image taken from the side of the vehicle so that the inspector's hands can be seen, for example.

Although not shown in FIG. 3, it is possible to provide a button for slow or frame-by-frame playback on the operation panel (70d), and configure it so that when slow or frame-by-frame playback is requested by operating this button, the playback of sound data from the speaker, the time display by the indicator (70b1), and the playback of video data (70e, 70f) are all performed at a relatively slow speed while maintaining synchronization with each other. This makes it easier to check the operation timing of the horn switch, for example.

Figure 2 shows the process flow of the inspection device of the first embodiment described above as a flowchart. First, the microphone of the sound acquisition unit 1 collects sounds, including, for example, the sound of a horn, from a vehicle test-driving on the free rollers and acquires them as sound data (step 1). In parallel with this, the video camera of the image acquisition unit 10 captures images of the vehicle test-driving on the free rollers and acquires them as image data (step 2).

Next, proceed to step 3, and determine whether a target sound (a horn sound in this example) has occurred based on the sound pressure characteristics, frequency characteristics, etc. of the acquired sound. If the target sound is detected, proceed from step 3 to step 4, and extract the sound data and video data to a length of, for example, several seconds in the range before and after the target sound, and store these data linked to each other in the sound/video data storage unit 50.

Next, proceed from step 4 to step 5, analyze the extracted sound data, and determine whether it is a normal sound or an abnormal sound. In step 6, generate a screen display as shown in FIG. 3. Then, in step 7, generate a screen display as shown in FIG. 3. As described above, when playback is requested by operating the operation panel unit (70d) for post-mortem verification, playback of the sound data from the speaker, the time display by the indicator (70b1), and playback of the video data (70e, 70f) are performed in synchronization with each other. If it is determined to be an abnormal sound, playback may be started automatically. Note that if it is determined to be a normal sound, generation of a screen for post-mortem verification including the video data (70e, 70f) may be omitted.

Finally, in step 8, it is determined whether the inspector has pressed the stop button to end the inspection. If the stop button has not been operated, the processes of steps 1 to 7 are repeated. For example, if the display indicates that the horn sound is abnormal, the horn switch can be operated again to repeat the horn inspection. When the stop button is pressed, the inspection ends. When the inspection is completed, a message to that effect is displayed on the display serving as the display unit 70. Along with the message, a buzzer or voice may be used to notify the end of the inspection.

Next, the inspection device of the second embodiment will be described with reference to Figures 4 to 6. The following mainly describes the differences from the first embodiment. The inspection device of the second embodiment differs from the first embodiment in that it extracts multiple sounds to be inspected (e.g., horn sounds and wiper sounds) during a series of inspections, judges whether they are abnormal sounds, and switches the display for subsequent verification of each result.

Figure 4 shows a functional block diagram of the inspection display device of the second embodiment. As with the first embodiment, the inspection device of the second embodiment includes a sound acquisition unit 1, a video acquisition unit 10, a target sound identification unit 20, an abnormal sound determination unit 30, a sound and video data generation unit 40, a sound and video data storage unit 50, a video display control unit 60, and a display unit 70 including a speaker, and further includes a priority determination unit 80.

In the second embodiment, while sound and video are continuously acquired by the sound acquisition unit 1 and the video acquisition unit 10, the inspector operates the horn switch and the wiper switch at any timing. The target sound identification unit 20 extracts the target sound (horn sound and wiper sound) from the continuous sound data acquired by the sound acquisition unit 1 substantially in real time, and cuts out sound data including the target sound. For example, if it is determined to be a horn sound, it generates sound data of a length of, for example, several seconds including the time before and after the target sound. Similarly, if it is determined to be a wiper sound (sound mainly generated by the wiper blades), it generates sound data of a length of, for example, several seconds including the time before and after the target sound.

The abnormal sound determination unit 30 determines whether the extracted horn sound and wiper sound are normal or abnormal using an appropriate method, as in the first embodiment described above.

As in the first embodiment, when the abnormal sound determination unit 30 determines that a sound is abnormal, the sound/video data generation unit 40 saves the above-mentioned sound data of a length of about several seconds including the target sound (horn sound or wiper sound) as, for example, a WAVE file. Then, the video data is cut out to a length range similar to that of the sound data and saved as a video file of a length of about several seconds, for example, an AVI file. These WAVE files and AVI files are saved as mutually linked data in the sound/video data saving unit 50, which serves as a database. For example, they are collected in one folder (or one compressed file) and stored in the sound/video data saving unit 50. Additional information such as a text file of vehicle information, analysis results of the sound data (spectrum, spectrogram, etc.), and a text file of detailed results when the abnormal sound is determined is added to this folder. In addition, information such as vehicle information and determination results may be added to the folder name or file name and saved. Here, in the second embodiment, when multiple target sounds are abnormal sounds, they are treated as mutually linked data, and for example, these are collected in one folder.

When multiple target sounds, for example a horn sound and a wiper sound, are abnormal sounds, the priority determination unit 80 determines the priority of which should be displayed first or mainly as a display (including sound playback from the speaker) on the display unit 70. In one example, the priority is determined based on the following four criteria.

1. The degree of impact when the part is used as is 2. The degree of abnormality 3. The magnitude of sound pressure and duration 4. The timing of the occurrence of the abnormal sound These four judgment criteria are prioritized in order of increasing order. In other words, if the magnitude of the impact when the part is used as is without replacement differs, the greater one takes higher priority. If the magnitudes of the impact are the same, the next judgment criterion, the degree of abnormality, is compared. If the magnitudes of abnormality are also the same, the next judgment criterion, the magnitude of sound pressure and duration, is compared. If these are also the same, the next judgment criterion is that the abnormal sound detected first has higher priority.

In the example of the horn sound and the windshield wiper sound, the horn sound has a higher priority based on the first criterion. The horn is an item that requires inspection because its performance is regulated by law, and it is not permitted under the law to use it if it is abnormal. On the other hand, the windshield wipers are a product quality issue, so they have a relatively low priority.

The second criterion is based on the fact that a greater degree of abnormality is more likely to cause problems as an abnormal sound. Similarly, the third criterion takes into consideration that a louder sound pressure or a longer duration is more likely to cause problems as an abnormal sound.

The video display control unit 60 generates a screen display that takes these priorities into account, and displays it via the display unit 70.

Figure 6 is an explanatory diagram showing an example of the display on the display unit 70 of the second embodiment. In this example, since the horn sound has a higher priority, information about the horn sound is displayed preferentially, and information about the wiper sound is displayed by switching the screen. Figure 6 shows the screen when information about the horn sound is displayed, and is basically similar to the display in the first embodiment, but in the "(1) Sound type" column (see symbol 70a) at the top of the display screen, the horn sound and the wiper sound are listed side by side, indicating that the display is about two abnormal sounds. The "horn sound" label has the circled number 1 attached, and the "wiper sound" label has the circled number 2 attached. The "(2) Sound pressure/frequency characteristics" column in the middle (see reference number 70b), the operation panel section (see reference number 70d) and enlarged image (see reference number 70f) in the "(3) Video display" column at the bottom, and the "(4) Sound evaluation results" column at the bottom right (see reference number 70g) each have the number 1 in a circle, indicating that they relate to the horn sound.

When the playback button on the operation panel (70d) is operated in this display state, as explained in the first embodiment above, the acquired horn sound (sound determined to be an abnormal sound) is played through the speaker, and the indicator (70b1) in the sound data (70b) displayed on the screen starts moving to the right in the figure, and at the same time, playback of the overall image (70e) and the enlarged image of the driver's seat (70f) starts. These images (70e, 70f) and sounds are played in sync with each other.

On the other hand, in the overall image (see symbol 70e) of the vehicle seen from above from the front, a circled number 2 is displayed with an arrow to indicate the location where the wiper sound is generated. For example, if a person involved in a post-mortem inspection clicks on the circled number 2 in this overall image (70e) or on the circled number 2 in the "(1) Sound type" column (70a) using an input means such as a mouse, the display will be switched to one related to the wiper sound.

In other words, the sound data (70b) including the target sound, the frequency analyzed spectrum (70c), and the table of "(4) Sound determination results" (70g) are each switched to display data related to the wiper sound, and the circled number 2 is added to indicate that it is a wiper sound.

The driver's seat enlarged image (70f) in FIG. 6 is switched to an enlarged image (70f2) of the wiper portion on the windshield as shown in FIG. 7. At this time, a circled number 1 is displayed with an arrow in the driver's seat portion of the overall image (70e), indicating that the display related to the horn sound is hidden.

When the playback button on the operation panel (70d) is operated in this display state, the acquired wiper sound (sound determined to be an abnormal sound) is played through the speaker, and the indicator (70b1) in the sound data (70b) displayed on the screen starts moving to the right side of the figure, and at the same time, playback of the entire image (70e) and the enlarged image (70f2) of the wiper part starts. These images (70e, 70f2) and sounds are played in synchronization with each other. The enlarged image (70f2) of the wiper part is also a part of the original video data acquired through a video camera that has been cut out and appropriately processed, and the reciprocating movement of the wiper that causes the wiper sound is shown as a video. Therefore, for example, by comparing the movement of the wiper with the sound being played and the sound data (70b) displayed together with the indicator (70b1), it is possible to verify at what timing the abnormal wiper sound was generated, or in what manner the abnormal sound is generated, etc.

In addition, sound and video playback may be started automatically each time the display is switched between the horn sound and the wiper sound.

Figure 5 is a flow chart showing the process flow of the inspection device of the second embodiment. First, the microphone of the sound acquisition unit 1 picks up sounds, including, for example, the sound of a horn, from a vehicle test-driving on the free rollers and acquires them as sound data (step 1). In parallel with this, the video camera of the image acquisition unit 10 captures images of the vehicle test-driving on the free rollers and acquires them as image data (step 2).

Next, proceed to step 3, where it is determined whether a target sound (a horn sound or a wiper sound in this example) has occurred based on the sound pressure characteristics and frequency characteristics of the acquired sound. If the target sound is detected, proceed from step 3 to step 4, and sound data and video data in a length of, for example, several seconds are extracted from the range before and after the target sound, and these are stored in the sound/video data storage unit 50 as linked data. Here, if multiple target sounds (horn sounds and wiper sounds) are detected during the series of tests, sound data and video data of about several seconds are extracted for each.

Then, proceed from step 4 to step 5, analyze the extracted sound data, and determine whether each sound is normal or abnormal.

Next, in step 11, it is determined whether the abnormal sound is two or more types of target sound. If there are two or more types of target sounds, the process proceeds to step 12, where the display priority is determined using the method described above.

Next, in step 6, a screen display like that shown in Figure 6 is generated according to the priority. Then, in step 7, a screen display like that shown in Figure 6 is displayed.

Finally, in step 8, it is determined whether the inspector has pressed the stop button to end the inspection. If the stop button has not been operated, steps 1 to 7 are repeated. If the stop button has been pressed, the inspection ends.

Figure 8 is an explanatory diagram showing different examples of displays on the display unit 70 of the second embodiment. In this example, the display related to the horn sound shown in Figure (a) and the display related to the wiper sound shown in Figure (b) are displayed alternately at regular intervals. For example, the display related to the horn sound with high priority is displayed for 10 seconds, and then the display related to the wiper sound is displayed for 10 seconds. It is preferable to repeat this multiple times. If the playback button on the operation panel unit (70d) is operated while either display is being displayed, the respective abnormal sound is played, and an indicator (70d1) on the sound data (70d) synchronized therewith, as well as the whole image (70e) and the enlarged image (70f, 70f2) are displayed.

In addition, sound and video playback may be automatically started every time the display is switched at a fixed interval.

9 is an explanatory diagram showing yet another example of the display on the display unit 70 of the second embodiment. In this example, information about a target sound with a high priority (in this example, a horn sound) (sound data (70b), a frequency analyzed spectrum (70c), a table of "(4) Sound Judgment Results" (see reference numeral 70g in FIG. 6), and an enlarged image of the driver's seat (70f)) is displayed. Also, as in the example of FIG. 6, the "(1) Sound Type" column (see reference numeral 70a) at the top of the screen lists the horn sound and the wiper sound together, with the circled numbers 1 and 2, respectively. Also, when information about the horn sound is displayed, as explained in FIG. 6, the overall image (70e) of the vehicle viewed from the front with a bird's-eye view has the circled number 2 with an arrow to indicate the location of the wiper sound.

When the playback button on the operation panel (70d) is operated in this display state, the acquired horn sound (sound determined to be an abnormal sound) is played through the speaker, and the indicator (70b1) in the sound data (70b) displayed on the screen starts moving to the right in the figure, and at the same time, playback of the overall image (70e) and the enlarged image of the driver's seat (70f) starts. These images (70e, 70f) and sounds are played in sync with each other.

In this state of display, when the circled number 2 in the overall image (70e) or the circled number 2 in the "(1) Sound Type" column (70a) is clicked with an input means such as a mouse, an enlarged image (70f2) of the wiper portion on the windshield as shown in FIG. 7 is displayed in the area of the "(4) Sound Determination Result" table (see reference numeral 70g in FIG. 6) as a display related to the wiper sound. For example, it is displayed in the form of a speech bubble protruding from the wiper portion of the overall image (70e). At this time, a triangular play button (see reference numeral 70h) labeled "Sound Determination Result Display" is also displayed. When this play button is clicked, the enlarged image (70f2) of the wiper portion disappears and the display returns to the "(4) Sound Determination Result" table.

When the playback operation is performed while the enlarged image (70f2) of the wiper portion is displayed, the wiper sound determined to be an abnormal sound and the enlarged image (70f2) of the wiper portion are played in a synchronized manner. Note that the sound data (70b) and spectrum (70c) continue to display information on the horn sound, which has a high priority.

The above describes an embodiment in which this invention is applied to the inspection of abnormal sounds such as horn sounds and windshield wiper sounds, but this invention is not limited to the above embodiment and can be applied in various ways. For example, it can be applied to the judgment of abnormal sounds such as vehicle brake sounds, and can be configured to capture video data including the operation of the brake calipers or braking operations by an inspector using a video camera and play it back in sync with the braking sounds. It can also be used for sound inspections other than vehicle inspections.

REFERENCE SIGNS LIST 1 sound acquisition unit 10 video acquisition unit 20 target sound identification unit 30 abnormal sound determination unit 40 sound and video data generation unit 50 sound and video data storage unit 60 video display control unit 70 display unit 80 priority determination unit

Claims (10)

Acquire the sound emitted from the device to be inspected and generate sound data including the target sound;
Acquire video of the inspection target device in parallel with acquiring the sound, and generate video data including the operation or movement that caused the target sound;
Analyzes the sound data to determine whether the target sound is an abnormal sound,
when it is determined that the sound is at least an abnormal sound, in response to a playback request, the sound data is played back through a speaker, and a time-series display of the sound data including an indicator synchronized with the playback, an analysis result of the sound data, and a playback image of the video data synchronized with the playback of the sound data are all displayed on one screen.
Acoustic testing methods. The acoustic inspection method according to claim 1, in which the sound data, video data, and analysis result data are stored in a database as linked data. The acoustic inspection method according to claim 1, wherein the equipment to be inspected is a vehicle. The acoustic inspection method according to claim 3, in which the target sound is a vehicle horn sound and video data including the driver's horn operation is played and displayed. The acoustic inspection method according to claim 3, in which the target sound is the sound of a vehicle's wipers operating, and video data including the operation of the wipers on the windshield is played back and displayed. The acoustic inspection method according to claim 1, in which an image of a target range is extracted from a temporarily recorded image of one camera to generate video data. For multiple types of target sounds from the same test target device, the above sound data is played back and displayed on the screen in sequence.
2. The acoustic inspection method of claim 1. Processing the reproduction and screen display of sound data for a plurality of types of target sounds in accordance with a predetermined priority order;
8. The acoustic inspection method of claim 7. when slow playback is requested, playback of the sound data, display of the time by the indicator, and playback of the video data are performed at a relatively slow speed while maintaining synchronization with each other;
2. The acoustic inspection method of claim 1. a sound acquisition unit that acquires a sound emitted from the inspection target device;
an image acquisition unit that acquires an image of the inspection target device in parallel with acquiring the sound;
a sound/video data generating unit that generates sound data including a target sound from the acquired sound and generates video data including an operation or action that causes the generation of the target sound from the acquired video in synchronization with the sound data;
an abnormal sound determination unit that analyzes sound data and determines whether a target sound is an abnormal sound;
A display unit including a speaker;
a display control unit which, when it is determined that the sound is at least an abnormal sound, plays back sound data through the speaker in response to a playback request, and displays, all together on one screen of the display unit, a time-series display of sound data including an indicator synchronized with the playback, an analysis result of the sound data, and a playback image of video data synchronized with the playback of the sound data;
An acoustic inspection device comprising:

Images