JP2023069116A - Information processing apparatus, imaging apparatus, control method thereof, image processing system and program - Google Patents

Abstract

To provide an information processing apparatus, a control method thereof, an imaging apparatus, an image processing system having the imaging apparatus and the information processing apparatus, and a program in which characteristics of vibration detected can be easily understood by a user.SOLUTION: In an image processing system in which a network camera can be connected to a client device through a network, the client device being an information processing apparatus acquires an imaged image of a subject, area information indicating an area of a portion in which vibration occurs in the subject, and vibration sound information representing a sound corresponding to the vibration of the portion from an imaging apparatus, superposes the area indicated by the area information on the imaged image to display on a display unit, and outputs the vibration sound from a sound emitting unit on the basis of the vibration sound information when a selection operation for the area which is displayed on the display unit by a user is accepted.SELECTED DRAWING: Figure 7

Description

The present invention relates to vibration detection by an imaging sensor.

2. Description of the Related Art In the field of factory automation (FA) and the like, vibration sensors are used to detect vibrations in order to analyze vibrations of devices and parts in work processes. Patent Document 1 discloses an event-driven imaging device (event-based image sensor (event-based sensor)) in an imaging device as the vibration sensor.

JP 2019-134271 A

The event-based sensor disclosed in Patent Literature 1 can detect vibration, but when an analyst (user) analyzes the detected vibration, the vibration has what kind of characteristics. It was difficult to determine whether

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and an object of the present invention is to provide a technology that enables a user to easily understand the characteristics of the detected vibration.

As one means for achieving the above object, the information processing apparatus of the present invention has the following configuration. That is, an acquisition means for acquiring, from an imaging device, a photographed image of a subject, region information indicating a region of a portion of the subject where vibration occurs, and vibration sound information representing a sound corresponding to the vibration of the portion; display control means for superimposing the area indicated by the area information on the captured image and displaying the area on a display section; reception means for receiving an operation by a user on the display section; and reception means for receiving a selection operation on the area. and sound control means for outputting the vibrating sound from the sound generating unit when the vibrating sound is generated.

SUMMARY OF THE INVENTION In accordance with the present invention, techniques are provided to facilitate user comprehension of the characteristics of detected vibrations.

1 shows a schematic diagram of an image processing system; FIG. 2 is a block diagram showing an example of the hardware configuration of a network camera; FIG. 3 is a block diagram showing an example of the hardware configuration of a client device; FIG. 4 shows a conceptual diagram of a functional configuration of a sound generator; FIG. FIG. 4 is a diagram for explaining the process of generating vibration sound ((a) is white noise, (b) is the frequency characteristic of the detected vibration, and (c) is the frequency characteristic of the vibration sound). 4 is a flowchart of exemplary processing performed by a network camera; 4 is a flowchart of exemplary processing performed by a client device according to the first embodiment; (a) shows an example of a captured image, and (b) shows an example of a screen displayed on the client device according to the first embodiment. 4 is a flowchart of exemplary processing performed by a client device according to the second embodiment; 10 shows an example of a screen displayed on the client device according to the second embodiment;

Embodiments for carrying out the present invention will be described in detail below with reference to the accompanying drawings. The embodiments described below are examples of means for realizing the present invention, and should be appropriately modified or changed according to the configuration of the apparatus to which the present invention is applied and various conditions. It is not limited to the embodiment. Also, not all combinations of features described in the present embodiment are essential for the solution means of the present invention.

[First embodiment]
(System configuration)
FIG. 1A shows a schematic diagram of the configuration of an image processing system according to this embodiment. This image processing system is composed of a network camera 1 and a client device 2 , and the network camera 1 and the client device 2 are configured to be connectable via a network 3 .

The network camera 1 includes an imaging unit (imaging unit 203) as described later, and is an imaging device capable of generating a captured image from a signal obtained by imaging an arbitrary subject with the imaging unit. is. The client device 2 is any information processing device such as a personal computer (PC), mobile phone, smart phone, PDA, or tablet terminal.

(Configuration of network camera 1)
A configuration example of the network camera 1 will be described with reference to FIG. First, the hardware configuration of the network camera 1 will be described. FIG. 2A is a block diagram showing an example of the hardware configuration of the network camera 1. As shown in FIG. The network camera 1 has a storage unit 201, a control unit 202, an imaging unit 203, a sound collection unit 204, an input unit 205, a display unit 206, a sound generation unit 207, and a communication unit 208 as an example of its hardware configuration.

The storage unit 201 includes memories such as ROM (Read Only Memory) and RAM (Random Access Memory), and stores programs for performing various operations described later and various information such as communication parameters for communication. As the storage unit 201, in addition to memories such as ROM and RAM, storage media such as flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, magnetic tapes, non-volatile memory cards, DVDs, etc. may be used. Also, the storage unit 201 may include a plurality of memories or the like.

The control unit 202 is composed of, for example, one or more CPUs (Central Processing Units), and controls the network camera 1 by executing programs stored in the storage unit 201 .

The imaging unit 203 is composed of a lens and an imaging device, and performs photoelectric conversion for converting a subject image into an electrical signal. An electric signal imaged and photoelectrically converted by the imaging unit 203 is transmitted to the image processing unit 211 (FIG. 2B).
Further, in the present embodiment, the imaging device of the imaging unit 203 is an event-based sensor, and luminance change information of a pixel (amount of change, tendency of change, etc.), address of the pixel (XY coordinates), time information of change ( timing/time of change, etc.) can be output. Accordingly, the imaging unit 203 can output the movement of the subject as a change in luminance. A typical event-based sensor has a sensing rate of about 1 G (1×10E9) Event/sec, which allows sensing up to 50 KHz in an area of 100×100 pixels. The luminance change information of the pixel, the address (XY coordinates) of the pixel, and the time information are transmitted to the vibration detection section 213 (FIG. 2(b)).
Note that the imaging unit 203 may have pixels configured by, for example, avalanche photodiodes (APDs). Specifically, the imaging unit 203 counts the number of photons that have entered each APD (pixel), and further calculates a clock supplied from outside the sensor (or a clock generated within the sensor using that clock). is used to count the time it takes for photons to reach a certain number. The imaging unit 203 calculates the difference between the first time required for the number of photons measured to exceed the threshold for the Nth time and the second time required for the number of photons to exceed the threshold for the N+1th time. A change in luminance of the pixel is detected according to the comparison result. Alternatively, the imaging unit 203 may count photons at a constant cycle and detect a change in luminance according to a change between the number of photons for the Nth time and the number of photons for the N+1th time. By using an avalanche photodiode (APD) as a pixel of the imaging unit 203, readout noise is small, so vibration can be detected with high accuracy.

The sound collector 204 collects sounds existing outside the network camera 1 (external sounds) and converts them into electric signals as audio signals. The audio signal (electrical signal) converted by the sound collector 204 is transmitted to the sound processor 212 (FIG. 2(b)). Specific examples of external sounds include environmental sounds around the network camera 1, human conversation sounds, and the like.

The input unit 205 receives various operations from the user, for example. The display unit 206 outputs various displays. Note that both the input unit 205 and the display unit 206 may be realized by one module like a touch panel.
The pronunciation unit 207 pronounces various audio signals as audible sounds.
A communication unit 208 is an interface that controls wired/wireless communication with an external device.

Next, the functional configuration of the network camera 1 will be explained. FIG. 2B is a block diagram showing an example of the functional configuration of the network camera 1. As shown in FIG. The network camera 1 has an image processing unit 211, a sound processing unit 212, a vibration detection unit 213, an attribute information generation unit 214, a sound generation unit 215, a communication control unit 216, and an output control unit 217 as an example of its functional configuration. .

The image processing unit 211 receives an electrical signal from the imaging unit 203 (FIG. 2A) and performs predetermined image processing on the electrical signal to generate a captured image. The generated captured image is transmitted to communication control section 216 .
The sound processing unit 212 receives an audio signal (external sound) converted into an electric signal from the sound collecting unit 204 (FIG. 2(a)), and performs amplification, band limitation, and analog-digital conversion on the audio signal. Each processing is performed to generate a digital audio signal of the external sound (external sound information). The generated digital audio signal is transmitted to communication control section 216 .

The vibration detection unit 213 detects periodic movements of the subject as vibrations of the subject based on information from the imaging unit 203 (pixel luminance change information, pixel address (XY coordinates), and time information). . The vibration detection unit 213 can detect, for example, a vibration when a pixel with a luminance change stays within a predetermined address range and repeats address displacement.
As described above, the event-based sensor, which is the imaging element of the imaging unit 203, generally has a detection rate of about 1 G (1×10E9) Event/sec. It is possible to detect (sensing) vibrations up to 50 KHz.

The attribute information generation unit 214 converts information (brightness change information, pixel address (XY coordinates), time information) from the imaging unit 203 to a subject (object) for which vibration is detected by the vibration detection unit 213. Attribute information indicating the characteristics (features) of the vibration is generated (derived) by performing arithmetic processing using this. The attribute information includes, for example, any one of fundamental frequency, intensity, modulation degree, and intermittence degree of vibration. The attribute information further includes or adds the address of the pixel where the vibration is detected (hereinafter referred to as the detection address) as area information indicating the position (area) of the portion where the vibration is detected. The generated attribute information is stored in the storage unit 201 .
The fundamental frequency is the frequency of the lowest frequency component that makes up the vibration, the intensity is the intensity of each frequency component that makes up the vibration, and the modulation index is the ratio of the multiple frequency components that make up the vibration. The degree is a degree indicating intermittence of vibration.

For example, FFT (Fast Fourier Transform) calculation is used for the calculation, and the frequency distribution of vibration intensity is calculated by FFT. Then, the fundamental frequency is derived from the low-order frequency components, the intensity is derived from the average signal strength of each frequency component, and the degree of modulation is derived from the ratio of the fundamental frequency component to the other frequency components. The degree of discontinuity is calculated from the degree of discontinuity of data between frames in a unit period (frame) in which the FFT operation is performed. Note that the FFT calculation and the audio processing including the derivation of the degree of modulation and the degree of discontinuity by the FFT calculation are known techniques. The attribute information of the generated vibration is transmitted to the sound generation section 215 and the communication control section 216 .

The sound generation unit 215 generates vibration sound as a sound representing vibration from the attribute information generated by the attribute information generation unit 214 . Information about the generated vibration sound is transmitted to the communication control unit 216 . Here, the generated vibration sound will be described with reference to FIGS. 4 and 5. FIG. FIG. 4 shows a conceptual diagram of the configuration of the sound generator 215, and FIG. 5 is a diagram for explaining the process of generating vibrating sound.

As shown in FIG. 4 , the sound generator 215 is composed of, for example, a white noise generator 41 , a digital filter 42 and a setting section 43 . First, the white noise generator 41 generates white noise, which is a signal containing all frequency components of vibrations that the subject may cause (FIG. 5(a)). Next, the setting unit 43 derives frequency characteristics having frequency characteristics of the detected vibration based on the attribute information, and sets them in the digital filter 42 . The setting unit 43 can derive the frequency characteristic of vibration from the fundamental frequency, the intensity (corresponding to the level), and the degree of modulation derived by the attribute information generation unit 214 and set it in the digital filter 42 . Note that the derivation method is not limited to this.

FIG. 5B shows an example of frequency characteristics derived and set in the setting unit 43. As shown in FIG. The digital filter 42 filters (synthesizes) the white noise generated by the white noise generator 41 with a filter having the frequency characteristics shown in FIG. 5B, thereby generating vibration sound reflecting attribute information. FIG. 5(c) shows the frequency characteristics of the generated (synthesized) vibration sound. Thus, the filtering of the digital filter 42 generates vibratory sound having the same frequency components as the detected vibration. A detection address is added to the vibration sound generated by the sound generation unit 215 and transmitted to the communication control unit 216 .

The communication control unit 216 converts the captured image, audio signal, attribute information, vibration sound information, etc. generated as described above into a network signal (for example, an Ethernet signal) for transmission via the network 3, The network signal is transmitted (distributed) to the client device 2 via the network 3 .
In addition, the communication control unit 216 according to the present embodiment has a server function, and when activated by the control unit 202, connects to the client device 2 via the network 3, and after connection, performs the distribution of captured images. It is configured to transmit initial screen data to the client device 2 .
The output control unit 217 performs display control for the display unit 206 and sound generation control for the sound generation unit 207 .

(Configuration of client device 2)
Next, a configuration example of the client device 2 will be described with reference to FIG. First, the functional configuration of the client device 2 will be described. FIG. 3A is a block diagram showing an example of the hardware configuration of the client device 2. As shown in FIG. The client device 2 has a storage unit 301, a control unit 302, an input unit 303, a display unit 304, a sound generation unit 305, and a communication unit 306 as an example of its hardware configuration. The storage unit 301, the control unit 302, the input unit 303, the display unit 304, the sound generation unit 305, and the communication unit 306 are the storage unit 201, the control unit 202, the input unit 205, and the display unit of the network camera 1 shown in FIG. 206, the sound generating unit 207, and the communication unit 208, the description thereof will be omitted.

Next, the functional configuration of the client device 2 will be described. FIG. 3B is a block diagram showing an example of the functional configuration of the client device 2. As shown in FIG. The client device 2 has a communication control section 311, an information restoration section 312, a display information generation section 313, and an output control section 314 as an example of its functional configuration.

The communication control unit 311 receives network signals transmitted by the network camera 1 via the network 3 . For example, the communication control unit 311 receives signals (information) of captured images, attribute information, vibration sounds, and audio signals (external sounds).
The information restoration unit 312 performs restoration processing on the signal received by the communication control unit 311 . For example, the information restoration unit 312 restores captured images (image signals), audio signals (external sounds), attribute information, and vibration sounds generated by the network camera 1 from the received signals.

The display information generation unit 313 generates information (display information) to be displayed on the display unit 304 from the received vibration attribute information. For example, the display information generator 313 generates text information from the vibration attribute information. The text information includes information indicating what kind of sound the vibration is (for example, "continuous sound", "inaudible", "modulated sound") and information on the fundamental frequency of the vibration (modulated sound multiple frequencies, if any) in the form of text. In addition, the display information generation unit 313 generates, as a vibration region, a figure indicating the region of the vibrating portion from the received detection address (added to the attribute information).

The output control unit 314 performs display control on the display unit 304 of the client device 2 . For example, the output control unit 314 superimposes various information generated by the display information generation unit 313 on the captured image received by the communication control unit 311 and generated by the information restoration unit 312 (that is, on the captured image displayed on the display unit 304 so that the various information is displayed. In addition, the output control section 314 controls sound generation from the sound generation section 305 .

(Processing flow)
Next, the flow of processing by the network camera 1 and client device 2 will be described. First, processing of the network camera 1 will be described. FIG. 6 is a flowchart of exemplary processing performed by the network camera 1 according to this embodiment. The flowchart shown in FIG. 6 can be realized by executing the control program stored in the storage unit 201 by the control unit 202 of the network camera 1 to perform calculation and processing of information and control of each hardware. .

When the network camera 1 is installed at a desired location and the power is turned on (S61), the control unit 202 activates the server function of the communication control unit 216 of the network camera 1. FIG. As a result, the network camera 1 becomes connectable to the client device 2 via the network 3 . Subsequently, the communication control unit 216 transmits initial screen data to the client device 2 via the network 3 .

After transmitting the initial screen data, when the communication control unit 216 receives a setting completion notification from the client device 2 (S62), the network camera 1 starts distributing the captured image. That is, the imaging unit 203 images one or more subjects (objects) within the imaging angle of view of the network camera 1 and outputs an electrical signal obtained by the imaging to the image processing unit 211 . Then, the image processing unit 211 generates a captured image from the electrical signal. The communication control unit 216 transmits (distributes) the captured image generated by the image processing unit 211 to the client device 2 (S63). At this time, the communication control unit 216 may transmit (distribute) the audio signal (external sound) generated by the sound processing unit 212 to the client device 2 . FIG. 8(a) shows an example of a captured image that is transmitted to the client device 2. As shown in FIG.

While the captured image is being distributed to the client device 2 , the vibration detection unit 213 detects vibration of the subject (object) from the information output by the imaging unit 203 . The detection process is as described above. When vibration is detected by the vibration detection unit 213 (Yes in S64), the attribute information generation unit 214 generates vibration attribute information (S65). The process of generating attribute information is as described above. The communication control unit 216 transmits the generated attribute information to the client device 2 via the network 3 (S66).

Also, the attribute information generated in S65 is input to the sound generation unit 215, and the sound generation unit 215 generates vibration sound reflecting the attribute information (S67). The vibration sound generation process is the same as the process described above. The communication control unit 216 transmits the generated vibration sound to the client device 2 via the network 3 (S68).
After that, the control unit 202 determines whether or not to end the delivery of the captured image. If not (No in S69), the process returns to S64. End the process. Whether or not to end distribution of the captured image can be determined by, for example, whether or not the communication control unit 216 receives a distribution end request from the client device 2 . In this case, when the communication control unit 216 receives the request, the control unit 202 can determine to end distribution of the captured image.

Next, processing of the client device 2 will be described. FIG. 7 is a flowchart of exemplary processing performed by the client device 2 according to this embodiment. The flowchart shown in FIG. 7 can be realized by executing the control program stored in the storage unit 301 by the control unit 302 of the client device 2 to perform calculation and processing of information and control of each hardware. . It is assumed that the client device 2 is powered on. Further, here, it is assumed that the communication control unit 216 also has the function of the information restoration unit 312 .

When the initial screen data is transmitted in response to the power-on of the network camera 1 (S61 in FIG. 6), the communication control unit 311 of the client device 2 receives the data, and the display control unit 314 receives the data. , the initial screen is displayed on the display unit 304 (S71). After that, when the user of the client device 2 performs a predetermined operation on the initial screen (for example, accesses a predetermined URL) to open a browser for displaying the captured image on the display unit 304, the communication control unit 311 completes the setting. A notification is sent to the network camera 1 via the network 3 (S72).

After transmitting the setting completion notification, the communication control unit 311 receives (acquires) the captured image (and the external sound), and the display control unit 314 displays the captured image on the display unit 304 (S73). An example of the captured image displayed here is shown in FIG. 8(a).
When the network camera 1 detects the vibration of the subject while the captured image is displayed on the display unit 304, the communication control unit 311 receives the vibration attribute information and vibration sound information generated by the network camera 1. is received (Yes in S74). The received attribute information and vibration sound are stored in the storage unit 301 . Subsequently, the display information generation unit 313 generates information (display information) to be displayed on the display unit 304 based on the received attribute information (S75), and the display control unit 314 causes the display unit 304 to display The generated display information is displayed on the captured image (S76).

The display information generated by the display information generation unit 313 includes, for example, a figure (vibration region) indicating the region of the vibrating portion of the subject and text information of attribute information, as described above. In addition to these pieces of information, the display information generation unit 313 may also generate an icon indicating that the sound can be produced.

FIG. 8B shows an example of a screen displayed on the display unit 304 of the client device according to this embodiment. In the example of FIG. 8B, the network camera 1 detects vibrations of a plurality of parts of one or more subjects, and generates and transmits attribute information and vibration sounds for each vibration. In FIG. 8(b), regions 81, 83, and 85 show figures indicating vibration regions (graphics of boundary lines surrounding vibrating portions), and regions 82, 84, and 86 show the vibrations of the regions 81, 83, and 85. Indicates text information of attribute information.
"1 KHz continuous sound" in the area 82 indicates that the vibrating sound is continuous sound with a frequency of 1 KHz. "22 KHz inaudible" in the area 84 indicates that the vibration sound is a 22 KHz sound outside the audible band (approximately 20 KHz). "2 KHz/600 Hz modulated sound" in area 86 indicates that the vibrating sound is a modulated sound composed of the sum of frequencies of 2 KHz and 600 Hz.
Also shown in FIG. 8(b) is an icon 87, here an icon in the form of a trumpet, indicating that the area 81 can be pronounced. Although not shown, an icon may also be displayed to indicate that the area 85 can be pronounced.

When the user of the client device 2 selects one of the vibration regions displayed on the display unit 304 via the input unit 303 (Yes in S77), the selection output control unit 314 controls the selected vibration region. A vibrating sound is output from the sound generator 305 (S78). It should be noted that, instead of the vibrating area, the user may select text or an icon of the attribute information so that a vibrating sound corresponding to the selected area may be output.

After that, the processing of S74 to S78 is continued until the user of the client device ends the display of the photographed image distributed from the network camera 1. FIG. Then, when the user inputs a distribution end request via the input unit 303 in order to end the display, and the communication control unit 311 transmits the request to the network camera 1 (Yes in S79), the process of FIG. exit. Note that the procedure for ending distribution (display) is not limited to this.

As described above, the display unit of the client device 2 displays the attribute information of the portion where the network camera 1 detects the vibration, and the user of the client device 2 selects the region of the portion where the vibration is detected on the display screen. Vibration sound corresponding to the selected portion is output. As a result, the user can grasp the characteristics of the vibration from the sound, and the characteristics of the vibration can be easily understood.

In this embodiment, as shown in FIG. 8B, the display unit 304 of the client device 2 shows an example of displaying the text information (regions 82, 84, 86) of the attribute information. may be displayed, and vibration sound corresponding to the area may be output according to the selection of the area. In this case, for example, the attribute information transmitted by the network camera 1 includes only the vibration detection address (area information), and the client device 2 displays the vibration area on the display unit 304 based on the detection address. It is superimposed and displayed. Then, when the user selects the area, the client device 2 can output a vibration sound corresponding to the area.

Further, in the present embodiment, in the network camera 1, after the vibration detection unit 213 detects vibration, the sound generation unit 215 generates a vibration sound. , the sound generator 212 may generate the vibrating sound. In this case, information on the selection of the vibration region by the user is transmitted to the network camera 1 via the network 3 , and the control section 202 of the network camera 1 transmits the information to the sound generation section 215 . The information includes an address for specifying the position, such as the address of the pixel corresponding to the selected vibration area. The sound generation unit 215 refers to the address of the transmitted information from the attribute information stored in the storage unit 201, starts calculation for the attribute information to which the detection address corresponding to the address is added, and generates the vibration sound. can be generated.

6, if the vibration sound (synthesized sound) generated by the sound generation unit 215 is outside the audible band, the output control unit 217 outputs a predetermined sound within the audible band to the sound generating unit 207 as a warning sound. You may make it output from.
Also, in such a case, the network camera 1 may send a predetermined notification to the client device 2 . For example, the attribute information generation unit 214 of the network camera 1 includes information indicating that the vibration sound is outside the audible band or frequency information in the attribute information, and the output control unit 314 of the client device 2 that receives the information Therefore, it is determined that the vibration sound is not in the audible band. Then, the output control unit 314 may convert the sound into a predetermined sound within the audible band, and output the sound from the sound generation unit 305 as a warning sound. may be displayed in

In addition, in the client device 2, until the user selects the vibration area in S77 of FIG. 7, or when the vibration sound is not output, the output control unit 314 receives the network External sound of the camera 1 may be output from the sound generator 305 .

Also, on the client device 2, a plurality of vibration areas are displayed as shown in FIG. 8B, and when the vibration sound is output in response to selection of one area in S77, the user selects another area. , the output control unit 314 may output the vibration sound corresponding to the most recently selected area (that is, the other area). Alternatively, the output control unit 314 may output the vibration sound corresponding to the selected area to the claim.

Further, when the vibration region is selected in S77, as shown in FIG. 8B, the output control unit 314 displays on the display unit 304 a plurality of options (selection menu) for the time for outputting the vibration sound. may Here, when the user selects one of the plurality of options via the input unit 303, the output control unit 314 emits a vibration sound for the time corresponding to the selected option. After the time has elapsed, the output of the vibration sound may be stopped.

Furthermore, in the present embodiment, the sound generation unit 215 of the network camera 1 generates vibration sound by applying a digital filter to white noise, but the method of generating vibration sound is not limited to this. For example, the sound generation unit 215 may create and store a plurality of digital sound patterns in advance, and select a digital sound pattern according to attribute information to generate vibration sound.

Further, in the present embodiment, the sound generation unit 215 generates vibration sound having the same frequency characteristics as the detected vibration. A simplification process such as setting may be performed, and the same effect is obtained in that case.

Further, in the present embodiment, the display unit 304 and the sound generation unit 305 are configured separately in the client device 2, but the display unit 304 and the sound generation unit 305 may be integrated.
Further, the sound generator 305 may be configured to generate vibration instead of/in addition to sound. In this case, the display unit 304 and the sound generation unit 305 have a touch panel-like thin film device as one integrated device, and the sound generation unit 305 corresponds to the subject selected by the user on the display unit 304. It may be configured to physically generate vibration due to vibration sound.

[Second embodiment]
Next, an embodiment in which a plurality of network cameras are used will be described as a second embodiment. Differences from the first embodiment will be described below, and descriptions of common features will be omitted.
FIG. 1B shows a schematic diagram of the configuration of the image processing system according to this embodiment. This image processing system is composed of network cameras 1 and 4 and a client device 2 , and the network cameras 1 and 4 and the client device 2 are configured to be connectable via a network 3 .

In the present embodiment, the configurations of the network cameras 1 and 4 and the client device 2 are as described in the first embodiment with reference to FIGS. 2 and 3, respectively, and the description thereof is omitted here.
In this embodiment, the network cameras 1 and 4 are assigned different identification numbers (IDs) to identify each camera. As shown in FIG. 9, it is assumed that network camera 1 has ID1 and network camera 4 has ID2.

Next, the flow of processing according to this embodiment will be described. The network cameras 1 and 4 each execute the processing shown in FIG. 6 described in the first embodiment. FIG. 9 shows a flowchart of an exemplary process performed by the client device 2 according to this embodiment. It should be noted that the same reference numerals are given to the same processing as in FIG. 7, and the description thereof will be omitted.

After the communication control unit 311 of the client device 2 receives the initial screen data and the display control unit 314 displays the initial screen on the display unit 304 based on the data (S71), the user of the client device 2 performs a predetermined operation. Accordingly, the communication control unit 311 transmits a setting completion notification to the network cameras 1 and 4 (S91). Further, the output control unit 314 determines the image arrangement on the display unit 304 of the two captured images distributed from the network cameras 1 and 4 (S91).

FIG. 10 shows an example of a screen displayed on the client device according to this embodiment. In the example of FIG. 10, the images are arranged in the order of network camera identification numbers (for each camera) with the upper left corner of the screen of the display unit 304 as a reference. That is, an image captured by network camera 1 (=ID1) is displayed on the left side, and an image captured by network camera 4 (=ID4) is displayed on the right side. The identification numbers (ID1, ID2) of network cameras 1 and 4 are also displayed.

Subsequently, the communication control unit 311 receives captured images (and external sounds) from the network cameras 1 and 4, and the display control unit 314 integrates the two captured images into one according to the image layout determined in S91. A wide area image is generated and displayed on the display unit 304 (S92).
When the network cameras 1 and 4 detect the vibration of the object while the wide-area image is being displayed on the display unit 304, the communication control unit 311 generates the attribute information of the vibration generated by the network cameras 1 and 4, and Information on the vibration sound is received (Yes in S74). The information is stored and managed in association with the camera identification number and detection address (area information). Subsequently, the display information generation unit 313 generates display information (S75), and the display control unit 314 displays the generated display information on the wide area image displayed on the display unit 304 according to the image layout determined in S91. Information is displayed (S93). In FIG. 10, text information generated from attribute information, a figure (vibration area) indicating the area of the vibrating portion, and the like are displayed on the image captured by the network cameras 1 and 4 .

When the user of the client device 2 selects one of the vibration regions displayed on the display unit 304 via the input unit 303 (Yes in S77), the output control unit 314 controls the selected vibration region and The identification number of the network camera is identified from the determined image layout. Then, the output control unit 314 outputs the vibration sound corresponding to the vibration area and the identification number of the camera from the sound generation unit 305 (S94).

As described above, in a wide-area image composed of images captured by a plurality of network cameras, information such as area information about a portion where vibration has been detected is displayed. A vibration sound corresponding to the region is output.

In this embodiment, the output control unit 314 of the client device 2 assigns the images captured by each network camera to the wide-area image in numerical order of camera identification numbers, but may be determined randomly. In addition, the output control unit 314 may determine the layout according to the user's operation via the input unit 303 .
Also, in this embodiment, an example in which the number of network cameras is two has been described, but the same description can be applied even if the number of network cameras is three or more. Also, at that time, the output control unit 314 can synthesize images for the number of cameras to generate a wide-area image and display it on the display unit 304 .

As described above, according to the embodiments described above, the user can easily understand the characteristics of the vibration of the subject visually and aurally, and can analyze the vibration generated in any device as the subject. Evaluation can be made more quantitatively.

Further, the present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device executes the program. It can also be realized by a process of reading and executing. It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

1; 4: network camera, 2: client device, 3: network

Claims (17)

Acquisition means for acquiring, from an imaging device, a photographed image of a subject, region information indicating a region of a portion of the subject in which vibration occurred, and vibration sound information representing a sound corresponding to the vibration of the portion;
display control means for superimposing the area indicated by the area information on the captured image and displaying the area on a display unit;
receiving means for receiving an operation by a user on the display unit;
sound control means for outputting the vibrating sound from a sound generating unit based on the information of the vibrating sound when the selecting operation for the region is received by the accepting means;
An information processing device comprising: The acquisition means further acquires information on the external sound of the imaging device from the imaging device,
2. The sound control means according to claim 1, wherein when the vibrating sound is not output from the sound generator, the sound controller causes the sound generator to output the external sound based on the information of the external sound. Information processing equipment. The acquisition means further acquires information on the vibration frequency of the portion from the imaging device,
3. The information processing apparatus according to claim 1, wherein the display control means superimposes the information on the frequency on the photographed image and displays it on the display unit. The sound control means determines whether or not the vibration sound is in the audible band based on the frequency information, and converts the vibration sound into sound in the audible band when it is determined that the vibration sound is not in the audible band. 4. The information processing apparatus according to claim 3, wherein the sound is output from the sound generator. The sound control means determines whether or not the vibration sound is in the audible band based on the information on the frequency, and when it is determined that the vibration sound is not in the audible band, the display control means warns the display unit. 5. The information processing apparatus according to claim 3, wherein is displayed. The display control means displays a plurality of options for a time for outputting the vibration sound on the display unit in response to the user selecting the region on the captured image,
When the selection operation of one of the plurality of options is accepted by the acceptance means, the sound control means outputs the vibrating sound from the sound generator for a time period corresponding to the selected option. 6. The information processing apparatus according to any one of claims 1 to 5, wherein the information is output. The acquiring means acquires other area information indicating an area of the part other than the part in which the subject vibrates, and information of another vibration sound representing a sound corresponding to the vibration of the different part. death,
The display control means superimposes the region and the another region indicated by the another region information on the captured image and displays the superimposed image on the display unit;
In a state in which the vibrating sound is being output from the sound generating unit by the sound controlling means, when the receiving means receives a selection operation for the different area, the sound controlling means stops outputting the vibrating sound. 7. The information processing apparatus according to any one of claims 1 to 6, wherein the sound generating unit stops and outputs the different vibration sound based on the information of the different vibration sound. In a state in which the vibrating sound is being output from the sound generating unit by the sound control means, when the receiving means receives a selection operation for the different region, the sound control means controls the vibrating sound and the 8. The information processing apparatus according to claim 7, wherein the different vibration sounds based on information of the other vibration sounds are alternately output from the sound generator. The obtaining means obtains from each of the plurality of imaging devices a photographed image of a subject, area information indicating an area of a portion of the subject where vibration occurs, and vibration sound information indicating a sound corresponding to the vibration of the portion. and are obtained,
9. The display controller according to any one of claims 1 to 8, wherein the display control unit superimposes the area indicated by the area information on the captured image for each of the plurality of imaging devices and displays the superimposed area on the display unit. The information processing device according to the item. a photographing means for photographing a subject and generating a photographed image of the subject;
a detection means for detecting vibration in the subject;
Derivation means for deriving area information indicating the area of the part of the subject where the vibration is detected by the detection means, and attribute information indicating characteristics of the vibration;
generating means for generating a vibration sound representing a sound corresponding to the vibration based on the attribute information;
a transmitting means for transmitting the captured image, the area information, and the vibration sound information to another device;
An imaging device characterized by comprising: The attribute information includes the fundamental frequency that is the frequency of the lowest frequency component that constitutes the vibration, the intensity of each frequency component that constitutes the vibration, the ratio of a plurality of frequency components that constitute the vibration, and the intermittence of the vibration. 11. The imaging device of claim 10, comprising any of the degrees of indicating. The generation means is composed of a white noise generation unit, a digital filter, and a setting unit,
The setting unit sets the frequency characteristic of the digital filter so as to have the frequency characteristic of the vibration based on the attribute information,
12. The imaging apparatus according to claim 10, wherein the digital filter filters white noise with the frequency characteristics to generate the vibration sound. A control method for an information processing device having a display unit and a sound generation unit, wherein an imaging device outputs a photographed image of a subject, area information indicating an area of a portion of the subject in which vibration occurs, and information corresponding to the vibration of the portion. an obtaining step of obtaining information about the vibration sound representing the sound;
a display control step of superimposing the area indicated by the area information on the captured image and displaying it on a display unit;
a sound control step of outputting the vibrating sound from the sound generating unit based on the information of the vibrating sound when a user's selection operation on the region displayed on the display unit is accepted;
A control method characterized by having A control method for an imaging device having imaging means for imaging a subject and generating a captured image of the subject, comprising:
a detection step of detecting vibration in the subject;
a derivation step of deriving area information indicating the area of the part of the subject where the vibration is detected by the detection means, and attribute information indicating characteristics of the vibration;
a generating step of generating a vibration sound representing a sound corresponding to the vibration based on the attribute information;
a transmission step of transmitting the captured image, the area information, and the vibration sound information to another device;
A control method characterized by having An image processing system having an imaging device and an information processing device,
The imaging device is
a photographing means for photographing a subject and generating a photographed image of the subject;
a detection means for detecting vibration in the subject;
Derivation means for deriving area information indicating the area of the part of the subject where the vibration is detected by the detection means, and attribute information indicating characteristics of the vibration;
generating means for generating a vibration sound representing a sound corresponding to the vibration based on the attribute information;
a transmitting means for transmitting the captured image, the area information, and the vibration sound information to another device;
has
The information processing device is
Acquisition means for acquiring the captured image, the area information, and the vibration sound information from the imaging device;
display control means for superimposing the area indicated by the area information on the captured image and displaying the area on a display unit;
receiving means for receiving an operation by a user on the display unit;
sound control means for outputting the vibrating sound from a sound generating unit based on the information of the vibrating sound when the selecting operation for the region is received by the accepting means;
An image processing system comprising: A program for causing a computer to function as each means of the information processing apparatus according to any one of claims 1 to 9. A program for causing a computer to function as each means of the imaging apparatus according to any one of claims 10 to 12.

Images