JP2020041993A - Information display control program, information display control method, and information display control device - Google Patents

Abstract

To provide an information display control program, an information display control method and an information display control device with which it is possible to identify a position with high accuracy.SOLUTION: The information display control program causes a computer to execute a process of displaying the installed positions of a plurality of objects on a display screen. The information display control program specifies the position of a sound source generated by motion of the object and causes a computer to execute a process of determining the attribute of the sound source. The information display control program causes a computer to execute a process of correlating the specified position the attribute of which is determined to the installed position of the object when there is a difference in physical relationship between the specified position the attribute of which is determined and the installed position of the object. The information display control program causes a computer to execute a process of notifying the state of motion of the object on the display screen after correlating.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information display control program, an information display control method, and an information display control device.

  2. Description of the Related Art A system for hearing impaired persons that detects an important sound such as a chime at a front door or a ringtone of a telephone and notifies the detected light or vibration is known. In this system, for example, by installing a microphone in the room, a ringtone of a phone call is acquired, and information indicating that a sound is sounding on a wristwatch-type terminal worn by a user or a terminal installed in a living room is provided. Send. When the terminal receives the information that the sound is sounding, the terminal notifies the user by light or vibration. In a noise environment such as a construction site, a system for displaying and notifying various information to a supervisor at the site is known.

JP-T-2013-522938 JP-T-2015-514239A

  However, for example, in a system for notifying by light, vibration, or the like, the system cannot recognize sounds other than predetermined important sounds, and it is difficult to notify the user. On the other hand, when a sound other than an important sound sounds in a room, the system may notify the user due to erroneous detection. For this reason, the user does not know what the notified sound is and does not know the position of the sound source, so that the user may have to search for the sound source.

  An aspect of the present invention is to provide an information display control program, an information display control method, and an information display control device capable of performing highly accurate position identification.

  In one aspect, the information display control program causes a computer to execute a process of displaying installation positions of a plurality of objects on a display screen. The information display control program causes a computer to execute a process of identifying a position of a sound source generated by the operation of the object and determining an attribute of the sound source. When there is a difference in the positional relationship between the specific position where the attribute is determined and the installation position of the object, the information display control program associates the specific position where the attribute is determined with the installation position of the object. On a computer. The information display control program causes a computer to execute a process of notifying the operating state of the target object on the display screen after the association.

  Accurate position identification can be performed.

FIG. 1 is a diagram illustrating an example of the information display control according to the first embodiment. FIG. 2 is a block diagram illustrating an example of a configuration of the information display control system according to the first embodiment. FIG. 3 is a block diagram illustrating an example of a configuration of the information display control device according to the first embodiment. FIG. 4 is a diagram illustrating an example of an association information storage unit according to the first embodiment. FIG. 5 is a flowchart illustrating an example of the setting process according to the first embodiment. FIG. 6 is a flowchart illustrating an example of a notification process according to the first embodiment. FIG. 7 is a block diagram illustrating an example of a configuration of the information display control device according to the second embodiment. FIG. 8 is a diagram illustrating an example of an association information storage unit according to the second embodiment. FIG. 9 is a diagram illustrating an example of the information display control according to the second embodiment. FIG. 10 is a flowchart illustrating an example of the setting process according to the second embodiment. FIG. 11 is a flowchart illustrating an example of a notification process according to the second embodiment. FIG. 12 is a block diagram illustrating an example of a configuration of the information display control device according to the third embodiment. FIG. 13 is a flowchart illustrating an example of the setting process according to the third embodiment. FIG. 14 is a block diagram illustrating an example of a configuration of the information display control device according to the fourth embodiment. FIG. 15 is a diagram illustrating an example of a correction information storage unit according to the fourth embodiment. FIG. 16 is a diagram illustrating an example of the information display control according to the fourth embodiment. FIG. 17 is a diagram illustrating another example of the information display control according to the fourth embodiment. FIG. 18 is a flowchart illustrating an example of the setting process according to the fourth embodiment. FIG. 19 is a flowchart illustrating an example of a notification process according to the fourth embodiment. FIG. 20 is a diagram showing an example of icon display according to the importance at the time of notification. FIG. 21 is a diagram showing another example of the icon display according to the importance at the time of notification. FIG. 22 is a diagram illustrating an example of a computer that executes an information display control program.

  Hereinafter, embodiments of an information display control program, an information display control method, and an information display control device disclosed in the present application will be described in detail with reference to the drawings. Note that the disclosed technology is not limited by the present embodiment. Further, the following embodiments may be appropriately combined within a consistent range.

  FIG. 1 is a diagram illustrating an example of the information display control according to the first embodiment. As shown in FIG. 1, an information display control device 100 according to a first embodiment, which will be described later, displays the installation positions of a plurality of objects and displays a display screen at the time of setting in which the arrangement positions of the objects are associated with the sound source positions 20 is displayed. The display screen 20 has, for example, display areas such as a bedroom 21a, a living room 21b, and an entrance 21c so as to correspond to a floor plan of the building. In addition, the display screen 20 has a display area 22 for icons that are not arranged or corresponding to outdoors. In the bedroom 21a and the living room 21b, icons corresponding to the microphones 10a and 10b for acquiring sound are arranged.

  For example, when the information display control device 100 starts acquiring sound from the microphones 10a and 10b, the information display control device 100 specifies the position of the sound source generated by the operation of the target object. The information display control device 100 specifies the sound source positions 23 and 24, for example, and displays them on the display screen 20. In the following description, the specified sound source position is also referred to as a specific position. In addition, the information display control device 100 determines the attribute of the sound source for the acquired sound using the learned model. That is, the information display control device 100 determines that the sound source at the sound source position 23 has the attribute “microwave oven”, and determines that the sound source at the sound source position 24 has the attribute “refrigerator”.

  When there is a difference in the positional relationship between the sound source positions 23 and 24 and the installation positions 25 and 26 of the microwave oven and the refrigerator as the object, for example, the information display control device 100 drags and drops the sound source positions 23 and 24 ( Hereinafter, the operation is also simply referred to as drag.) An operation is accepted. The information display control device 100 superimposes the sound source positions 23 and 24 on the installation positions 25 and 26 by the dragging operations 27 and 28, so that the installation position 25 and the sound source position 23 and the installation position 26 and the sound source position 24 are Correspond each.

  When the setting is completed, the information display control device 100 displays a display screen 30 for performing notification, which is used after associating the installation position of the target object with the sound source position. The display screen 30 has a display area corresponding to the floor plan similarly to the display screen 20. The information display control device 100 specifies the sound source positions 23a and 24a from the sound acquired from the microphone 10b, for example. If the information display control device 100 determines that the sound source at the sound source position 23a has the attribute “microwave oven”, the display screen 30 displays an icon indicating that the microwave is emitting sound on the installation position 25 of the microwave oven. 31 is displayed and notified. When the information display control device 100 determines that the sound source at the sound source position 24a has the attribute “refrigerator”, an icon 32 indicating that the refrigerator emits sound is displayed on the refrigerator installation position 26 on the display screen 30. Is displayed and notified. Thereby, the information display control device 100 can perform highly accurate position identification.

  FIG. 2 is a block diagram illustrating an example of a configuration of the information display control system according to the first embodiment. As shown in FIG. 2, the information display control system 1 includes microphones 10a, 10b, 10c and an information display control device 100. The information display control system 1 is a system that specifies and displays the position and type (attribute) of a sound source based on sounds acquired by the microphones 10a, 10b, and 10c. In the following description, when the microphones 10a, 10b, and 10c are not distinguished, the microphone 10 will be described. Further, in the information display control system 1, the number of microphones 10 is not limited, and an arbitrary number of microphones 10 may be provided.

  The microphone 10 and the information display control device 100 are communicably connected via a network N. As the network N, any type of communication network such as a LAN (Local Area Network) and a VPN (Virtual Private Network), such as the Internet, can be adopted regardless of whether it is wired or wireless. Communication via the network N may be encrypted by, for example, TLS (Transport Layer Security) / SSL (Secure Sockets Layer).

  The microphone 10 includes, for example, an omnidirectional microphone and a communication unit that performs communication with the information display control device 100. When receiving the acquisition start instruction from the information display control device 100 to start the acquisition of the sound, the microphone 10 starts transmitting the acquired sound data.

  First, the information display control device 100 executes a setting process for associating the installation position of the target object with the sound source position. The information display control device 100 displays the installation positions of a plurality of objects on a display screen. The information display control device 100 specifies the position of the sound source generated by the operation of the object and determines the attribute of the sound source. When there is a difference in the positional relationship between the specific position where the attribute is determined and the installation position of the target, the information display control device 100 associates the specific position where the attribute is determined with the installation position of the target. Next, the information display control device 100 performs a notification process of notifying the sound acquired by the microphone 10 based on the information in which the installation position of the target object and the sound source position are associated with each other. The information display control device 100 notifies the operating state of the target object on the display screen after the association based on the sound data acquired by the microphone 10.

  Subsequently, the configuration of the information display control device 100 will be described. FIG. 3 is a block diagram illustrating an example of a configuration of the information display control device according to the first embodiment. As shown in FIG. 3, the information display control device 100 includes a communication unit 110, a display unit 111, an operation unit 112, a storage unit 120, and a control unit 130. The information display control device 100 may include various functional units included in a known computer, for example, various input devices and audio output devices, in addition to the functional units illustrated in FIG. 3.

  The communication unit 110 is realized by, for example, a NIC (Network Interface Card) or the like. The communication unit 110 is a communication interface that is connected to the microphone 10 via a network N in a wired or wireless manner, and manages communication of information with the microphone 10. The communication unit 110 receives sound data from the microphone 10. The communication unit 110 outputs the received sound data to the control unit 130. The communication unit 110 transmits the acquisition start instruction input from the control unit 130 to the microphone 10.

  The display unit 111 is a display device for displaying various information. The display unit 111 is realized by, for example, a liquid crystal display or the like as a display device. The display unit 111 displays various screens such as a display screen input from the control unit 130.

  The operation unit 112 is an input device that receives various operations from the user of the information display control device 100. The operation unit 112 is realized by, for example, a keyboard, a mouse, or the like as an input device. The operation unit 112 outputs the operation input by the user to the control unit 130 as operation information. The operation unit 112 may be realized by a touch panel or the like as an input device, or the display device of the display unit 111 and the input device of the operation unit 112 may be integrated.

  The storage unit 120 is realized by, for example, a semiconductor memory device such as a RAM (Random Access Memory) and a flash memory, and a storage device such as a hard disk and an optical disk. The storage unit 120 includes a layout storage unit 121, a learned model storage unit 122, and an association information storage unit 123. The storage unit 120 stores information used for processing in the control unit 130.

  The layout storage unit 121 stores a floor plan of a building (room) as a target of information display to the user of the information display control device 100, and IDs (IDentifiers), arrangement positions, and attributes of objects placed in the room. I do. Here, as the attribute of the target, for example, the name of the target such as a microwave oven or a refrigerator can be used.

  The learned model storage unit 122 stores a learned model obtained by learning the sound emitted from the object and the attribute of the object by machine learning. The trained model is used for determining the attribute of the target that is the sound source, and outputs the attribute of the target with respect to the input sound data, for example. The trained model is one that has been learned using a neural network, for example, using a Mel frequency spectrogram based on the power spectrum of sound data as a feature amount. As the neural network, for example, a CNN (Convolutional Neural Network) can be used. The learned model stores, for example, various parameters (weight coefficients) of the neural network.

  The association information storage unit 123 stores the installation position of the target object and the sound source position in association with each other. FIG. 4 is a diagram illustrating an example of an association information storage unit according to the first embodiment. As shown in FIG. 4, the association information storage unit 123 has items such as “object ID”, “installation position”, “object attribute”, “sound source ID”, “sound source position”, and “sound source attribute”. .

  "Object ID" is an identifier for identifying an object. “Installation position” is information indicating the installation position of the object in the building. As the “installation position”, for example, coordinate information in a floor plan (overhead view) can be used. “Object attribute” is information indicating the attribute of the object. As the "object attribute", for example, a name of an object such as a microwave oven, a refrigerator, a door, and an alarm clock can be used. “Sound source ID” is an identifier for identifying a sound source. “Sound source position” is information indicating the position of the sound source in the building. The “sound source position” is specified using, for example, the loudness and phase difference of sounds acquired from the plurality of microphones 10. The “sound source position” may be information indicating how far the microphone 10 is from the microphone 10 using the volume of one microphone 10. “Sound source attribute” is information indicating an attribute of a sound source. As the “sound source attribute”, similarly to the object attribute, for example, a name of the object such as a microwave oven, a refrigerator, a door, and an alarm clock can be used.

  Returning to the description of FIG. 3, the control unit 130 executes a program stored in an internal storage device using a RAM as a work area by, for example, a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). This is achieved by: The control unit 130 may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control unit 130 includes a display control unit 131, an identification unit 132, a determination unit 133, an association unit 134, and a notification unit 135, and realizes or executes functions and operations of information processing described below. . Note that the internal configuration of the control unit 130 is not limited to the configuration illustrated in FIG. 3, and may be another configuration as long as it performs information processing to be described later.

  The display control unit 131 refers to the layout storage unit 121 and displays the installation positions of a plurality of objects on the display screen at the time of setting for associating the arrangement positions of the objects and the sound source positions. When displaying the installation position of the object on the display screen, the display control unit 131 outputs a setting start instruction to the specifying unit 132.

  The display control unit 131 refers to the layout storage unit 121 and displays the installation positions of a plurality of targets on the display screen when reporting the sound acquired by the microphone 10. When displaying the installation position of the object on the display screen, the display control unit 131 outputs a notification start instruction to the specifying unit 132.

  When the setting start instruction or the notification start instruction is input from the display control unit 131, the specifying unit 132 transmits an acquisition start instruction to the microphone 10 via the communication unit 110 and the network N. The specifying unit 132 starts acquiring data of the sound whose transmission has been started in response to the acquisition start instruction from the microphone 10. That is, the specifying unit 132 starts acquiring sounds.

  The specifying unit 132 specifies the position of the sound source based on the acquired sound data. For example, when the number of microphones 10 is three or more, the specifying unit 132 specifies a point where a circle representing a volume of each microphone 10 intersects (a point based on a phase difference) as the position of the sound source. For example, when the number of the microphones 10 is two, the specifying unit 132 determines any position on a curve connecting two points (two points based on the phase difference) where circles representing the volume of each microphone 10 intersect. Is specified as the position of the sound source. For example, when the number of the microphones 10 is one, the specifying unit 132 specifies any position on the circle representing the volume of the microphone 10 as the position of the sound source. In this case, the specifying unit 132 may specify a concentric circle corresponding to the volume value instead of the position of the sound source. That is, the specifying unit 132 specifies the position of the sound source generated by the operation of the target.

  When a setting start instruction is input from the display control unit 131, the specifying unit 132 assigns a sound source ID to the sound source at the specified sound source position, and associates the sound data, the sound source ID, and the sound source position with the determination unit 133. Output to When the notification start instruction is input from the display control unit 131, the specifying unit 132 outputs the sound data and the specified sound source position to the determination unit 133 in association with each other.

  When the sound data, the sound source ID, and the sound source position associated with the sound are input from the specifying unit 132, the determining unit 133 refers to the learned model storage unit 122 and determines the attribute of the sound using the learned model. . The determination unit 133 associates the sound source attribute, which is the attribute of the determined sound, with the sound source ID and the sound source position, and outputs the sound source attribute to the association unit 134. In addition, the determination unit 133 displays an icon corresponding to the sound source position on the display screen.

  Further, when the sound data and the sound source position associated with each other are input from the specifying unit 132, the determination unit 133 refers to the learned model storage unit 122 and determines the attribute of the sound using the learned model. The determining unit 133 refers to the association information storage unit 123 and acquires a sound source attribute that is the attribute of the determined sound and a sound source ID that matches the sound source position. The determination unit 133 outputs the acquired sound source ID to the notification unit 135 in association with the sound source attribute and the sound source position.

  When the sound source ID, the sound source attribute, and the sound source position are input from the determination unit 133, the associating unit 134 receives a drag operation on the icon of the sound source position. When the icon of the sound source position is superimposed on the installation position of the object by the drag operation, the association unit 134 associates the installation position of the object with the sound source position. The associating unit 134 stores in the association information storage unit 123 the object ID, the installation position, the object attribute, the sound source ID, the sound source position, and the sound source attribute that are the result of the association.

  In other words, when there is a difference in the positional relationship between the specific position (sound source position) where the attribute is determined and the installation position of the target, the associating unit 134 sets the specific position where the attribute is determined to the installation position of the target object. Corresponds to In addition, the association unit 134 associates the specific position whose attribute has been determined with the installation position of the target object by receiving an operation of associating the specific position displayed on the display screen with the installation position of the target object.

  When the sound source ID, the sound source attribute, and the sound source position are input from the determination unit 133, the notification unit 135 refers to the association information storage unit 123, and sets the installation position and the object attribute of the object ID associated with the sound source ID. To get. The notification unit 135 notifies the user of a sound generated by the operation of the target object by displaying an icon corresponding to the target object attribute at the acquired installation position on the display screen. At this time, the notification unit 135 does not display the icon corresponding to the sound source position. In addition, the notification unit 135 can determine the size of the icon to be notified according to the volume of the sound acquired by the microphone 10. Further, the icon representing the microphone 10 represents a volume value and changes its size in accordance with the volume so that the volume increases as the volume increases.

  In other words, the notification unit 135 notifies the operation state of the target object on the display screen after the association. In addition, when detecting the operation state of the target, the notification unit 135 does not display the specific position on the display screen, but displays the installation position of the target after association on the display screen.

  Next, the operation of the information display control device 100 according to the first embodiment will be described. First, the setting process of the information display control process will be described with reference to FIG. FIG. 5 is a flowchart illustrating an example of the setting process according to the first embodiment.

  The display control unit 131 of the information display control device 100 refers to the layout storage unit 121 and displays the installation positions of a plurality of objects on a display screen (Step S1). When displaying the installation position of the object on the display screen, the display control unit 131 outputs a setting start instruction to the specifying unit 132.

  When the setting start instruction is input from the display control unit 131, the specifying unit 132 transmits an acquisition start instruction to the microphone 10. The specifying unit 132 starts acquiring data of the sound whose transmission has been started in response to the acquisition start instruction from the microphone 10 (Step S2). The specifying unit 132 specifies the position of the sound source based on the acquired sound data (Step S3). The specifying unit 132 assigns the sound source ID to the sound source at the specified sound source position, and outputs the sound source, the sound source ID, and the sound source position to the determination unit 133 in association with each other.

  When the sound data, the sound source ID, and the sound source position associated with the sound are input from the specifying unit 132, the determining unit 133 refers to the learned model storage unit 122 and determines the attribute of the sound using the learned model. (Step S4). The determining unit 133 associates the sound source attribute with the sound source ID and the sound source position and outputs the sound source attribute to the associating unit 134. In addition, the determination unit 133 displays an icon corresponding to the sound source position on the display screen (Step S5).

  When the sound source ID, the sound source attribute, and the sound source position are input from the determination unit 133, the associating unit 134 receives a drag operation on the icon of the sound source position (step S6). When the icon of the sound source position is superimposed on the installation position of the target object by the drag operation, the association unit 134 associates the installation position of the target object with the sound source position and stores it in the association information storage unit 123 (Step S7). Thereby, the information display control device 100 can associate the installation position of the target with the sound source position of the sound generated by the operation of the target.

  Next, a notification process in the information display control process will be described with reference to FIG. FIG. 6 is a flowchart illustrating an example of a notification process according to the first embodiment.

  The display control unit 131 of the information display control device 100 refers to the layout storage unit 121 and displays the installation positions of the plurality of objects on the display screen (Step S11). When displaying the installation position of the object on the display screen, the display control unit 131 outputs a notification start instruction to the specifying unit 132.

  When the notification start instruction is input from the display control unit 131, the identification unit 132 transmits an acquisition start instruction to the microphone 10. The specifying unit 132 starts acquiring data of the sound whose transmission has been started in response to the acquisition start instruction from the microphone 10 (step S12). The specifying unit 132 specifies the position of the sound source based on the acquired sound data (step S13). The specifying unit 132 outputs the sound data and the specified sound source position to the determining unit 133 in association with each other.

  When the sound data and the sound source position associated with each other are input from the specifying unit 132, the determination unit 133 refers to the learned model storage unit 122 and determines the attribute of the sound using the learned model (step S14). ). The determination unit 133 acquires a sound source ID that matches the determined sound source attribute and sound source position with reference to the association information storage unit 123. The determination unit 133 outputs the acquired sound source ID to the notification unit 135 in association with the sound source attribute and the sound source position.

  When the sound source ID, the sound source attribute, and the sound source position are input from the determination unit 133, the notification unit 135 refers to the association information storage unit 123, and sets the installation position and the object attribute of the object ID associated with the sound source ID. To get. The notification unit 135 displays an icon corresponding to the target object attribute on the display screen at the acquired installation position (Step S15). Thereby, the information display control device 100 can perform highly accurate position identification on the target object.

  As described above, the information display control device 100 displays the installation positions of the plurality of objects on the display screen. In addition, the information display control device 100 specifies the position of the sound source generated by the operation of the object and determines the attribute of the sound source. Further, when there is a difference in the positional relationship between the specific position (sound source position) where the attribute is determined and the installation position of the target, the information display control device 100 sets the specific position where the attribute is determined to the installation position of the target. Corresponds to In addition, the information display control device 100 notifies the operation state of the target object on the display screen after the association. As a result, the information display control device 100 can perform highly accurate position identification on the target object.

  In addition, when detecting the operation state of the target, the information display control device 100 does not display the specific position on the display screen, but displays the installation position of the target after association on the display screen. I do. As a result, even if the sound source position obtained from the sound generated by the object is shifted from the installation position, the information display control device 100 can notify the installation position of the object on the display screen.

  Further, the information display control device 100 displays the specific position on the display screen. In addition, the information display control device 100 accepts an operation of associating the specific position displayed on the display screen with the installation position of the object, thereby associating the specific position whose attribute is determined with the installation position of the object. As a result, the information display control device 100 can easily associate the position of the sound source with the installation position of the target.

  In the first embodiment, the drag operation to the position of the target object is received for the specified sound source position. However, the operation of dragging the marker representing the attribute of the target object to the target position may be received. An embodiment in this case will be described as a second embodiment. The same components as those of the information display control device 100 according to the first embodiment are denoted by the same reference numerals, and the description of the overlapping configurations and operations will be omitted.

  FIG. 7 is a block diagram illustrating an example of a configuration of the information display control device according to the second embodiment. The information display control device 200 illustrated in FIG. 7 includes a storage unit 220 and a control unit 230 instead of the storage unit 120 and the control unit 130, as compared with the information display control device 100 according to the first embodiment. The storage unit 220 includes an association information storage unit 223 instead of the association information storage unit 123, as compared with the storage unit 120. Further, control unit 230 includes a correspondence unit 234 and a notification unit 235 instead of mapping unit 134 and notification unit 135, as compared with control unit 130.

  The association information storage unit 223 stores information on the marker in association with the information, as compared with the association information storage unit 123. FIG. 8 is a diagram illustrating an example of an association information storage unit according to the second embodiment. As shown in FIG. 8, the association information storage unit 223 stores “object ID”, “installation position”, “object attribute”, “sound source ID”, “sound source position”, “sound source attribute”, and “marker ID”. , "Marker position", and "marker attribute".

  “Target object ID” to “sound source attribute” are the same as those in the association information storage unit 123 of the first embodiment, and a description thereof will be omitted. “Marker ID” is an identifier for identifying a marker. “Marker position” is information indicating a marker position after a drag operation is received. As the “marker position”, for example, coordinate information in a floor plan (overhead view) can be used. As the “marker attribute”, for example, a name of an object such as a microwave oven, a refrigerator, a door, and an alarm clock, that is, an attribute of the object can be used.

  Returning to the description of FIG. 7, upon input of the sound source ID, the sound source attribute, and the sound source position from determination unit 133, mapping unit 234 accepts a drag operation on the marker. In the initial state before accepting the drag operation, the marker is displayed, for example, outside the floor plan on the display screen. When the marker is superimposed on the installation position of the object by the drag operation, the association unit 234 associates the installation position of the object with the marker position. The associating unit 234 stores the object ID, the installation position, the object attribute, the marker ID, the marker position, and the marker attribute, which are the results of the association, in the association information storage unit 223. When the association between the marker and the installation position of the object is completed, the association unit 234 associates the sound source position closest to the marker arranged on the floor plan with the marker and the object, for example. This is stored in the attachment information storage unit 223.

  In other words, the associating unit 234 accepts an operation of associating a marker representing the attribute of the target object displayed on the display screen with the installation position of the target object having the same attribute as the marker, thereby changing the marker to the target object. Correspond to the installation position.

  When the sound source ID, the sound source attribute, and the sound source position are input from the determination unit 133, the notification unit 235 refers to the association information storage unit 223 and acquires the marker position and the marker attribute of the marker ID associated with the sound source ID. I do. The notification unit 235 notifies the user of a sound generated by the operation of the target object by displaying an icon corresponding to the marker attribute at the acquired marker position on the display screen. At this time, the notification unit 235 does not display the icon corresponding to the sound source position.

  Here, information display control according to the second embodiment will be described with reference to FIG. FIG. 9 is a diagram illustrating an example of the information display control according to the second embodiment. As shown in FIG. 9, the information display control device 200 displays the installation positions of a plurality of objects, and displays a display screen 40 at the time of setting for associating the arrangement positions of the objects with the marker positions. The display screen 40 has a display area such as a bedroom 21a, a living room 21b, and an entrance 21c, for example, corresponding to the floor plan of the building, similarly to the display screen 20 of the first embodiment. In addition, the display screen 40 has a display area 22 for icons corresponding to non-arranged and outdoor locations. On the display screen 40, for example, a marker 41 of a microwave oven and a marker 42 of a refrigerator which are not arranged in the display area 22 are arranged. In the bedroom 21a and the living room 21b, icons corresponding to the microphones 10a and 10b for acquiring sound are arranged.

  For example, when the information display control device 200 starts acquiring sounds from the microphones 10a and 10b, the information display control device 200 specifies the position of the sound source generated by the operation of the target object. Unlike the information display control device 100, the information display control device 200 does not display the specified sound source position. The information display control device 200 determines the attribute of the sound source for the acquired sound using the learned model.

  The information display control device 200 accepts the drag operation 43 of the marker 41 to the installation position 25 of the microwave oven as the target. In addition, the information display control device 200 accepts the drag operation 44 of the marker 42 to the refrigerator installation position 26 as the target. The information display control device 200 allows the markers 41 and 42 to be superimposed on the installation positions 25 and 26 by the drag operations 43 and 44, respectively, so that the marker positions of the installation position 25 and the marker 41 and the markers of the installation position 26 and the marker 42. The positions are associated with each other.

  When the setting is completed, the information display control device 200 displays a display screen 45 for performing notification, which is used after associating the installation position of the target object with the marker position. The display screen 45 has a display area corresponding to the floor plan similarly to the display screen 40. The information display control device 200 specifies the sound source positions 23a and 24a from the sound acquired from the microphone 10b, for example. When the information display control device 200 determines that the sound source at the sound source position 23a has the attribute “microwave oven”, the display screen 45 corresponds to the marker attribute on the marker position associated with the microwave oven installation position 25. The icon 31 is displayed and notified. The icon 31 is an icon indicating that the microwave is emitting a sound.

  When the information display control device 200 determines that the sound source at the sound source position 24a has the attribute “refrigerator”, the display screen 45 corresponds to the marker attribute on the marker position associated with the refrigerator installation position 26. The icon 32 is displayed and notified. The icon 32 is an icon indicating that the refrigerator emits a sound. Thereby, the information display control device 200 can perform highly accurate position identification based on the marker position corresponding to the installation position of the target object.

  Next, the operation of the information display control device 200 according to the second embodiment will be described. First, the setting process of the information display control process will be described with reference to FIG. FIG. 10 is a flowchart illustrating an example of the setting process according to the second embodiment. In the following description, the processing of steps S1 to S4 of the setting processing is the same as that of the first embodiment, and thus the description thereof will be omitted.

  The associating unit 234 of the information display control device 200 executes the following processing following the processing of step S4. When the sound source ID, the sound source attribute, and the sound source position are input from the determination unit 133, the association unit 234 receives a drag operation on the marker (step S21). When the marker is superimposed on the installation position of the object by the drag operation, the association unit 234 associates the installation position of the object with the marker position and stores the association in the association information storage unit 223 (Step S22). When the association between the marker and the installation position of the object is completed, the association unit 234 stores the sound source position closest to the marker in the association information storage unit 223 in association with the marker and the object. Thereby, the information display control device 200 can associate the installation position of the target object with the marker position of the marker representing the attribute of the target object.

  Subsequently, a notification process in the information display control process will be described with reference to FIG. FIG. 11 is a flowchart illustrating an example of a notification process according to the second embodiment. In the following description, the processes in steps S11 to S14 of the notification process are the same as those in the first embodiment, and thus description thereof will be omitted.

  The notification unit 235 of the information display control device 200 executes the following processing following the processing of step S14. When the sound source ID, the sound source attribute, and the sound source position are input from the determination unit 133, the notification unit 235 refers to the association information storage unit 223 and acquires the marker position and the marker attribute of the marker ID associated with the sound source ID. I do. The notification unit 235 displays an icon corresponding to the marker attribute on the acquired marker position on the display screen (Step S23). Thereby, the information display control device 200 can perform highly accurate position identification based on the marker position corresponding to the target object.

  As described above, the information display control device 200 receives the operation of associating the marker representing the attribute of the target object displayed on the display screen with the installation position of the target object having the same attribute as the marker, and thereby targets the marker. Correspond to the installation position of the object As a result, the information display control device 200 can perform highly accurate position identification based on the marker position corresponding to the target object.

  Note that the information display control device 200 does not need to specify the sound source position and determine the sound source attribute at the time of setting. In this case, steps S2 to S4 of the setting process are omitted. In the notification process in this case, the notification unit 235 refers to the association information storage unit 223 and, for example, among the markers having the same attribute as the input sound source attribute, corresponds to the marker position closest to the input sound source position. The marker position and marker attribute of the marker ID to be acquired are acquired. The notification unit 235 displays an icon corresponding to the marker attribute at the acquired marker position on the display screen.

  As described above, the information display control device 200 displays the installation positions of the plurality of objects on the display screen. In addition, the information display control device 200 accepts an operation of associating a marker representing the attribute of the target object displayed on the display screen with the installation position of the target object having the same attribute as the marker, thereby setting the marker to the target object. Correspond to the installation position. In addition, the information display control device 200 specifies the position of the sound source generated by the operation of the object and determines the attribute of the sound source. In addition, the information display control device 200 notifies the operating state of the target object having the same attribute as the determined attribute on the display screen after the association. As a result, highly accurate position identification can be performed based on the marker position corresponding to the object.

  In the second embodiment, the sound source position closest to the marker is stored in association with the marker and the object when the association between the marker and the installation position of the object is completed. The sound source position may be associated with the installation position of the object using the association of the trigger as a trigger. An embodiment in this case will be described as a third embodiment. The same components as those of the information display control devices 100 and 200 according to the first and second embodiments are denoted by the same reference numerals, and the description of the overlapping configurations and operations will be omitted. Further, the display screen of the information display control of the third embodiment is the same as that of the second embodiment, and the description thereof is omitted.

  FIG. 12 is a block diagram illustrating an example of a configuration of the information display control device according to the third embodiment. The information display control device 300 illustrated in FIG. 12 includes a control unit 330 instead of the control unit 230 as compared with the information display control device 200 according to the second embodiment. Further, control unit 330 has an associating unit 334 instead of associating unit 234 as compared to control unit 230.

  When the sound source ID, the sound source attribute, and the sound source position are input from the determination unit 133, the associating unit 334 receives a drag operation on the marker. In the initial state before accepting the drag operation, the marker is displayed, for example, outside the floor plan on the display screen. When the marker is superimposed on the installation position of the object by the drag operation, the association unit 334 associates the installation position of the object, the sound source position, and the marker position. The associating unit 334 stores the object ID, the installation position, the object attribute, the sound source ID, the sound source position, the sound source attribute, the marker ID, the marker position, and the marker attribute, which are the results of the association, in the association information storage unit 223. .

  In other words, the associating unit 334 associates the specific position (sound source position) with the installation position of the target, using the association of the marker with the installation position of the target as a trigger.

  Next, the operation of the information display control device 300 according to the third embodiment will be described. FIG. 13 is a flowchart illustrating an example of the setting process according to the third embodiment. In the following description, the processing of steps S1 to S4 and S21 of the setting processing is the same as that of the second embodiment, and thus the description thereof will be omitted. In the third embodiment, the notification process is the same as in the second embodiment, and a description thereof will not be repeated.

  The associating unit 334 of the information display control device 300 performs the following processing following the processing of step S21. When the marker is superimposed on the installation position of the object by the drag operation, the association unit 334 associates the installation position of the object, the sound source position, and the marker position, and stores the association in the association information storage unit 223 (Step S31). Thereby, the information display control device 300 can associate the installation position of the target object, the sound source position, and the marker position of the marker representing the attribute of the target object.

  As described above, the information display control device 300 associates the specific position (sound source position) with the installation position of the target, triggered by the association of the marker with the installation position of the target. As a result, the information display control device 300 can perform highly accurate position identification based on the marker position corresponding to the target object.

  In the first embodiment, the sound source position for which the attribute of the sound source has been determined is associated with the installation position of the target. However, the position of another sound source that is not the determination target of the attribute of the sound source is specified, and the sound source position and the installation position are determined. The position of another sound source may be corrected using the correction information based on the association. An embodiment in this case will be described as a fourth embodiment. The same components as those of the information display control device 100 according to the first embodiment are denoted by the same reference numerals, and the description of the overlapping configurations and operations will be omitted.

  FIG. 14 is a block diagram illustrating an example of a configuration of the information display control device according to the fourth embodiment. The information display control device 400 illustrated in FIG. 14 includes a storage unit 420 and a control unit 430 instead of the storage unit 120 and the control unit 130 as compared with the information display control device 100 of the first embodiment. The storage unit 420 further includes a correction information storage unit 424 as compared with the storage unit 120. The control unit 430 is different from the control unit 130 in that the identification unit 132, the determination unit 133, the association unit 134, and the notification unit 135 are replaced with the identification unit 432, the determination unit 433, the association unit 434, and the notification unit. 435.

  The correction information storage unit 424 stores correction information for correcting the position of another sound source that is not a sound source attribute determination target. FIG. 15 is a diagram illustrating an example of a correction information storage unit according to the fourth embodiment. As shown in FIG. 15, the correction information storage unit 424 has items such as “other sound sources”, “sound sources to be associated”, and “correction information”. Here, the “sound source to be associated” is a sound source whose attribute has been determined, for example, a sound source whose attribute belongs to the same group when other sound sources are grouped with the sound source whose attribute has been determined. The “other sound sources” and “sound source to be associated” have items such as “sound source ID”, “sound source position”, and “sound source attribute”. Further, the “correction information” includes items such as “correction value” and “corrected sound source position”.

  “Sound source ID” is an identifier for identifying a sound source. The “sound source position” is specified using, for example, the loudness and phase difference of sounds acquired from the plurality of microphones 10. “Sound source attribute” is information indicating an attribute of a sound source. As the “sound source attribute”, in the case of other sound sources, for example, names such as other A and other B can be used. As the “sound source attribute”, in the case of a sound source to be associated, for example, the name of an object such as a microwave oven, a refrigerator, a door, and an alarm clock can be used. The “correction value” is a value for correcting the position of another sound source based on the movement amount received by the drag operation of the sound source to be associated. “Sound source position after correction” is information indicating the position of another sound source after correction using the correction value.

  Returning to the description of FIG. 14, when the setting start instruction or the notification start instruction is input from the display control unit 131, the specifying unit 432 starts acquiring sound data, similarly to the specifying unit 132. The specifying unit 432 specifies the position of a sound source including other sound sources based on the acquired sound data. That is, the specifying unit 432 specifies the position of the sound source generated by the operation of the target object and the position of another sound source that is not a determination target of the attribute of the sound source.

  When the setting start instruction is input from the display control unit 131, the specifying unit 432 assigns a sound source ID to the sound source at the specified sound source position, and associates the sound data, the sound source ID, and the sound source position with the determination unit 433. Output to When the notification start instruction is input from the display control unit 131, the specifying unit 432 outputs the sound data and the specified sound source position to the determination unit 433 in association with each other.

  When the sound data, the sound source ID, and the sound source position associated with the sound are input from the specifying unit 432, the determining unit 433 refers to the learned model storage unit 122 and determines the attribute of the sound using the learned model. . The determination unit 433 associates the sound source attribute, which is the attribute of the determined sound, with the sound source ID and the sound source position, and outputs the sound source attribute to the association unit 434. The determination unit 433 also sets the sound source attribute to, for example, “Other A” for the sound source (other sound source) for which the attribute of the sound could not be determined, and similarly associates the sound source attribute with the sound source ID and the sound source position. Output to the attachment unit 434.

  In addition, the determination unit 433 performs sound source grouping. For example, when there are three microphones 10 in the living room, the determination unit 433 performs grouping by dividing the microphones into three groups. That is, the living room is divided into areas according to the number of microphones 10. The determination unit 433 groups the sound source positions of the sound sources whose attributes existing in the divided area have been determined, and the sound source positions of the other sound sources. The determination unit 433 stores in the correction information storage unit 424 the sound source positions of other grouped sound sources and the sound source positions of the sound sources whose attributes have been determined in association with each other. If the sound source positions of the other sound sources are near the boundary between the two areas, the determination unit 433 stores the groups on both sides of the boundary in association with the sound source positions of the sound sources whose attributes have been determined. The determination unit 433 displays an icon corresponding to the sound source position including the other sound sources on the display screen. At this time, the determination unit 433 may display the grouped areas on a display screen.

  Further, when the sound data and the sound source position associated with each other are input from the specifying unit 432, the determination unit 433 refers to the learned model storage unit 122 and determines the attribute of the sound using the learned model. The determination unit 433 refers to the association information storage unit 123 and acquires a sound source attribute that is the attribute of the determined sound and a sound source ID that matches the sound source position. The sound source attributes here include the sound source attributes of other sound sources. The determining unit 433 outputs the acquired sound source ID to the notifying unit 435 in association with the sound source attribute and the sound source position.

  When the sound source ID, the sound source attribute, and the sound source position are input from the determination unit 433, the associating unit 434 accepts a drag operation on the icon of the sound source position of the sound source whose attribute has been determined. When the icon of the sound source position of the sound source whose attribute has been determined is superimposed on the installation position of the target object by the drag operation, the association unit 434 associates the installation position of the target object with the sound source position. The associating unit 434 stores the object ID, the installation position, the object attribute, the sound source ID, the sound source position, and the sound source attribute, which are the results of the association, in the association information storage unit 123.

  When the association is completed for the sound source position of the sound source whose attribute has been determined, the associating unit 434 calculates the difference between the sound source position before and after the drag operation. The associating unit 434 calculates the corrected sound source positions of the sound source positions of the other sound sources belonging to the same group as the sound source positions of the sound sources whose attributes have been determined based on the calculated difference. That is, the associating unit 434 calculates the correction information of the sound source position of the other sound sources. The associating unit 434 stores the calculated correction information in the correction information storage unit 424. When there are a plurality of sound source positions of sound sources whose attributes can be determined in the same group, the associating unit 434 calculates correction information based on the sound source positions of sound sources whose one attribute can be determined. The sound source position of the determined sound source may be corrected. In this case, the associating unit 434 can reduce the burden of associating the sound source position of the user with the installation position of the target object.

  When the sound source ID, the sound source attribute, and the sound source position are input from the determination unit 433, the notification unit 435 refers to the association information storage unit 123, and the target object ID associated with the sound source ID of the sound source whose attribute has been determined. Acquires the installation position and object attributes. The notification unit 435 notifies the user of a sound generated by the operation of the target object by displaying an icon corresponding to the target object attribute at the acquired installation position on the display screen. At this time, the notification unit 435 does not display the icon corresponding to the sound source position.

  When the input sound source ID is not stored in the association information storage unit 123, the notification unit 435 refers to the correction information storage unit 424 and acquires the correction information associated with the input sound source ID. . The notifying unit 435 notifies the user of the sound of the other sound source by displaying an icon corresponding to the other sound source at the sound source position of the other sound source on the display screen based on the acquired correction information. At this time, the notification unit 435 does not display the icon corresponding to the sound source position, as in the case of the sound source whose attribute has been determined. When the position of the other sound source has two sound sources associated with each other, the notification unit 435 corrects the position of the other sound source based on the respective correction information and displays the icon.

  In other words, the notification unit 435 corrects the position of another specified sound source based on the correction information when the specific position (the sound source position) is associated with the installation position of the target object. In addition, the determination unit 433 divides the area including the installation position of the target into a plurality. The notification unit 435 corrects, for each divided area, the position of another sound source in the same area based on the correction information of the specific position in the area. In addition, when there is a position of another sound source near the boundary between the two areas, the notification unit 435 corrects the position of the other sound source based on the correction information of the specific position in the area on both sides of the boundary.

  Here, information display control according to the fourth embodiment will be described with reference to FIGS. 16 and 17. FIG. 16 is a diagram illustrating an example of the information display control according to the fourth embodiment. As shown in FIG. 16, the information display control device 400 displays the installation positions of a plurality of objects, and displays a display screen 50 at the time of setting for associating the arrangement positions of the objects with the marker positions. The display screen 50 has, for example, display areas such as a bedroom 21a, a living room 21b, and an entrance 21c so as to correspond to a floor plan of the building. In addition, the display screen 50 has a display area 22 for icons that are not arranged or corresponding to outdoors. In the bedroom 21a and the living room 21b, icons corresponding to the microphones 10a, 10b, and 10c for acquiring sound are arranged. The living room is divided into groups 51, 52, and 53 corresponding to the microphones 10a, 10b, and 10c, respectively.

  For example, when the information display control device 400 starts acquiring sound from the microphones 10a to 10c, the information display control device 400 specifies the position of the sound source generated by the operation of the target object. The information display control device 400 specifies, for example, the sound source positions 54 and 55 of the group 52 and the sound source positions 56 and 57 of the group 53 and displays them on the display screen 50. In addition, the information display control device 400 determines the attribute of the sound source for the acquired sound using the learned model. That is, the information display control device 400 determines that the sound source at the sound source position 54 has the attribute “fridge” and determines that the sound source at the sound source position 55 has the attribute “faucet”. Further, the information display control device 400 determines that the sound source at the sound source position 56 has the attribute “air conditioner”. Furthermore, since the information display control device 400 could not determine the attribute of the sound for the sound source at the sound source position 57, the information display control device 400 sets, for example, the attribute “other A” as another sound source.

  When there is a difference in the positional relationship between the sound source positions 54 and 55 and the installation positions 26 and 58 of the refrigerator and the faucet, which are the objects, the information display control device 400 drags the sound sources to the sound source positions 54 and 55, for example. Accept the drop operation. The information display control device 400 superimposes the sound source positions 54 and 55 on the installation positions 26 and 58 by the dragging operations 59 and 60, thereby setting the installation position 26 and the sound source position 54 and the installation position 58 and the sound source position 55, Correspond each.

  When there is a difference in the positional relationship between the sound source position 56 and the installation position 61 of the air conditioner as the target for the group 53, the information display control device 400 accepts, for example, a drag-and-drop operation on the sound source position 56. The information display control device 400 associates the installation position 61 with the sound source position 56 by superimposing the sound source position 56 on the installation position 61 by the drag operation 62. The information display control device 400 calculates a difference between the sound source position 56 before and after the drag operation. The information display control device 400 calculates the sound source position 63 of the sound source position 57 of the other sound sources belonging to the same group 53 as the sound source position 56 after the correction based on the calculated difference. That is, the information display control device 400 calculates the correction information 64 of the sound source position 57. Note that the person at the sound source position 63 is a display example in the case where a talking voice is assumed as an example of a sound that is not a target of the attribute determination. The sound source position 57 and the corrected sound source position 63 include a picture of another object, For example, it may be a question mark or the like.

  When the setting is completed, the information display control device 400 displays a display screen 70 for performing notification, which is used after associating the installation position of the target object with the sound source position. The display screen 70 has a display area corresponding to the floor plan similarly to the display screen 50. The information display control device 400 specifies the sound source positions 54a and 55a from the sound acquired from the microphone 10b, for example. If the information display control device 400 determines that the sound source at the sound source position 54a has the attribute “refrigerator”, the display screen 70 displays an icon 71 on the refrigerator installation position 26 indicating that the refrigerator emits sound. Notify. When the information display control device 400 determines that the sound source at the sound source position 55a has the attribute “faucet”, the icon 72 indicating that the faucet is emitting sound is displayed on the installation position 58 of the faucet on the display screen 70. Is displayed and notified.

  In addition, the information display control device 400 specifies the sound source positions 56a and 57a from the sound acquired from the microphone 10c, for example. When determining that the sound source at the sound source position 56a has the attribute “air conditioner”, the information display control device 400 displays an icon 73 indicating that the air conditioner is emitting a sound on the installation position 61 of the air conditioner on the display screen 70. Notify. If the information display control device 400 determines that the sound source at the sound source position 57a has the attribute “Other A”, the information display control device 400 refers to the correction information, and displays the corrected sound source position 63 on the display screen 70 as the attribute determination target. An icon 74 corresponding to a no sound, for example, a talking voice is displayed and notified. Thus, the information display control device 400 can perform highly accurate position identification even for a sound whose attribute is not to be determined.

  FIG. 17 is a diagram illustrating another example of the information display control according to the fourth embodiment. As shown in FIG. 17, the information display control device 400 displays the installation positions of a plurality of objects, and displays a display screen 80 at the time of setting for associating the arrangement positions of the objects with the marker positions. The display screen 80 has, for example, display areas such as a bedroom 21a, a living room 21b, and an entrance 21c so as to correspond to a floor plan of the building. In addition, the display screen 80 has a display area 22 for icons that are not arranged or corresponding to outdoors. In the bedroom 21a and the living room 21b, icons corresponding to the microphones 10a, 10b, and 10c for acquiring sound are arranged. The living room is divided into groups 51, 52, and 53 corresponding to the microphones 10a, 10b, and 10c, respectively.

  For example, when the information display control device 400 starts acquiring sound from the microphones 10a to 10c, the information display control device 400 specifies the position of the sound source generated by the operation of the target object. The information display control device 400 specifies, for example, the sound source positions 54 of the group 52 and the sound source positions 56 and 57 of the group 53 and displays them on the display screen 80. In addition, the information display control device 400 determines the attribute of the sound source for the acquired sound using the learned model. That is, the information display control device 400 determines that the sound source at the sound source position 54 has the attribute “fridge”. Further, the information display control device 400 determines that the sound source at the sound source position 56 has the attribute “air conditioner”. Furthermore, since the information display control device 400 could not determine the attribute of the sound for the sound source at the sound source position 57, the information display control device 400 sets, for example, the attribute “other A” as another sound source.

  When there is a difference in the positional relationship between the sound source position 54 and the installation position 26 of the refrigerator, which is the target, for the group 52, the information display control device 400 accepts, for example, a drag-and-drop operation on the sound source position 54. The information display control device 400 associates the installation position 26 with the sound source position 54 by superimposing the sound source position 54 on the installation position 26 by the drag operation 59. Further, the information display control device 400 calculates a difference between the sound source position 54 before and after the drag operation in order to calculate the correction information of the sound source position 57 near the boundary between the group 52 and the group 53. The information display control device 400 calculates the corrected sound source position X of the sound source position 57 of the other sound sources based on the calculated difference.

  When there is a difference in the positional relationship between the sound source position 56 and the installation position 61 of the air conditioner as the target for the group 53, the information display control device 400 accepts, for example, a drag-and-drop operation on the sound source position 56. The information display control device 400 associates the installation position 61 with the sound source position 56 by superimposing the sound source position 56 on the installation position 61 by the drag operation 62. The information display control device 400 calculates a difference between the sound source position 56 before and after the drag operation. The information display control device 400 calculates the corrected sound source position Y of the sound source position 57 of the other sound sources belonging to the same group 53 as the sound source position 56 based on the calculated difference.

  The information display control device 400 calculates the correction information 81 of the sound source position 57 based on the difference based on the sound source position 54 and the corrected sound source position X, and the difference based on the sound source position 56 and the corrected sound source position Y. . In FIG. 17, the sound source position after the correction based on the correction information 81 is shown as a sound source position 82. That is, in the example of FIG. 17, the information display control device 400 determines the sound source position 82 after the correction of the sound source position 57, which is another sound source, by receiving the drag operations 59 and 62. It is to be noted that the person at the sound source position 82 is a display example in the case where a talking voice is assumed as an example of the sound that is not the attribute determination target, as in the case of FIG. 16, and the sound source position 57 and the corrected sound source position 82 May be a picture of another thing, such as a question mark.

  When the setting is completed, the information display control device 400 displays a display screen 85 for performing notification, which is used after associating the installation position of the target object with the sound source position. The display screen 85 has a display area corresponding to the floor plan similarly to the display screen 80. The information display control device 400 specifies the sound source positions 54a and 57a from the sound acquired from the microphone 10b, for example. When the information display control device 400 determines that the sound source at the sound source position 54a has the attribute “refrigerator”, the display screen 85 displays an icon 71 indicating that the refrigerator emits sound on the refrigerator installation position 26 on the display screen 85. Notify.

  In addition, the information display control device 400 specifies the sound source positions 56a and 57a from the sound acquired from the microphone 10c, for example. When determining that the sound source at the sound source position 56a has the attribute “air conditioner”, the information display control device 400 displays an icon 73 indicating that the air conditioner is emitting a sound on the installation position 61 of the air conditioner on the display screen 85. Notify. When the information display control device 400 determines that the sound source at the sound source position 57a has the attribute “others A”, the information display control device 400 refers to the correction information and displays the display target on the display screen 85 on the corrected sound source position 82. An icon 86 corresponding to a no sound, for example, a talking voice is displayed and notified. Thereby, the information display control device 400 can perform highly accurate position identification even when the sound source position is near the boundary of the group.

  Next, the operation of the information display control device 400 according to the fourth embodiment will be described. First, the setting process of the information display control process will be described with reference to FIG. FIG. 18 is a flowchart illustrating an example of the setting process according to the fourth embodiment. In the following description, the processing of steps S1, S2, S4, S6, and S7 of the setting processing is the same as that of the first embodiment, and thus the description thereof will be omitted.

  The specifying unit 432 of the information display control device 400 executes the following processing following the processing of step S2. The specifying unit 432 specifies the positions of sound sources including other sound sources based on the acquired sound data (step S41). The specifying unit 432 assigns the sound source ID to the sound source at the specified sound source position, outputs the sound data, the sound source ID, and the sound source position to the determination unit 433 in association with each other, and proceeds to step S4.

  The determination unit 433 performs the following processing following the processing in step S4. The determination unit 433 associates the sound source attribute, which is the attribute of the determined sound, with the sound source ID and the sound source position, and outputs the sound source attribute to the association unit 134. In addition, the determination unit 433 associates the sound source attribute indicating that it is the other sound source with the sound source ID and the sound source position and outputs the other sound sources to the association unit 134. The determining unit 433 groups other sound sources with sound sources whose attributes have been determined based on the area corresponding to the microphone 10 (step S42). The determining unit 433 stores the sound source positions of the other grouped sound sources in the correction information storage unit 424 in association with the sound source positions of the sound sources whose attributes have been determined (step S43). The determination unit 433 displays an icon corresponding to the sound source position including the other sound sources on the display screen (step S44), and proceeds to step S6.

  The associating unit 434 performs the following process following the process of step S7. When the association is completed for the sound source positions of the sound sources whose attributes have been determined, the associating unit 434 calculates correction information of the sound source positions of the other sound sources based on the difference between the sound source positions before and after the drag operation. (Step S45). The associating unit 434 stores the calculated correction information in the correction information storage unit 424 (Step S46). Accordingly, the information display control device 400 can also associate the corrected sound source position with respect to the sound whose attribute is not to be determined.

  Subsequently, the notification process in the information display control process will be described with reference to FIG. FIG. 19 is a flowchart illustrating an example of a notification process according to the fourth embodiment. In the following description, the processes in steps S11, S12, S14, and S15 of the notification process are the same as those in the first embodiment, and thus the description thereof will be omitted.

  The specifying unit 432 of the information display control device 400 executes the following processing following the processing of step S12. The specifying unit 432 specifies the positions of sound sources including other sound sources based on the acquired sound data (step S47). The identification unit 432 outputs the sound data and the identified sound source position to the determination unit 433 in association with each other, and proceeds to step S14.

  The notification unit 435 performs the following processing following the processing in step S15. When the input sound source ID is not stored in the association information storage unit 123, the notification unit 435 refers to the correction information storage unit 424 and acquires the correction information associated with the input sound source ID. The notification unit 435 displays an icon corresponding to the other sound source at the sound source position of the other sound source on the display screen based on the acquired correction information (step S48). Thus, the information display control device 400 can perform highly accurate position identification even for a sound whose attribute is not to be determined.

  As described above, the information display control device 400 specifies the position of another sound source that is not a determination target of the attribute of the sound source. In addition, the information display control device 400 corrects the position of another specified sound source based on the correction information when the specific position is associated with the installation position of the target object. As a result, the information display control device 400 can perform highly accurate position identification on the sound of another sound source that is not the target of the attribute determination of the sound source.

  In addition, the information display control device 400 divides the area including the installation position of the target object into a plurality of areas, and for each of the divided areas, based on the correction information of the specific position in the area, the other sound sources in the same area. Correct the position. As a result, the information display control device 400 can perform more accurate position identification when there are a plurality of microphones 10.

  Further, when there is a position of another sound source near the boundary between the two areas, the information display control device 400 corrects the position of the other sound source based on the correction information of the specific position in the area on both sides of the boundary. As a result, the information display control device 400 can perform more accurate position identification for the sound of another sound source located near the boundary of the area.

  In each of the above embodiments, the size of the icon to be notified is determined according to the volume of the sound acquired by the microphone 10, but the present invention is not limited to this. For example, the size of the icon may be changed according to the degree of importance, and the icon display in this case will be described with reference to FIGS. 20 and 21, the information display control device 100 according to the first embodiment will be described as an example of the information display control device according to each embodiment.

  FIG. 20 is a diagram showing an example of icon display according to the importance at the time of notification. As shown in FIG. 20, the information display control device 100 displays a display screen 90 for performing notification. The display screen 90 has, for example, display areas such as a bedroom 21a, a living room 21b, and an entrance 21c so as to correspond to a floor plan of the building. In the bedroom 21a and the living room 21b, icons corresponding to the microphones 10a and 10b for acquiring sound are arranged.

  For example, when the sound source of the sound acquired from the microphone 10b has the attribute “faucet”, the information display control device 100 assumes that the faucet is left open and the importance is high. In this case, the information display control device 100 displays, for example, an icon 91 having a size that covers not only the faucet but also other objects on the installation position of the faucet.

  FIG. 21 is a diagram showing another example of the icon display according to the importance at the time of notification. As shown in FIG. 21, the information display control device 100 displays a display screen 92 for performing notification. The display screen 92 has, for example, display areas such as a bedroom 21a, a living room 21b, and an entrance 21c so as to correspond to a floor plan of the building. In the bedroom 21a and the living room 21b, icons corresponding to the microphones 10a and 10b for acquiring sound are arranged.

  For example, when the sound source of the sound acquired from the microphone 10b has the attribute “knock sound”, the information display control device 100 assumes that the visitor is visiting the entrance and the importance is medium. In this case, the information display control device 100 displays, for example, an icon 93 indicating a knock sound having a loudness that does not hinder the display of other icons at the entrance 21c and notifies the user. Accordingly, the information display control device 100 changes the size of the icon to be displayed according to the importance of the acquired sound source of the sound, so that the importance of the sound can be easily determined.

  In each of the above embodiments, the setting and the notification are displayed using the floor plan, but the present invention is not limited to this. For example, a sound source position of a sound source, an installation position of a target object, and the like may be displayed on a photograph of a room to perform setting or notification.

  Further, in each of the above embodiments, the history of notification may be stored and, for example, may be displayed as a history list.

  Further, in each of the above embodiments, the installation position of the target object is described as one, but the present invention is not limited to this. For example, when the same object (for example, a mobile phone or a smartphone) is located at two or more installation positions (for example, a bedroom and a living room), registration is performed at the time of setting, and at the time of notification, registration is performed based on a sound source position. An icon can be displayed at any of the installed positions to notify the user.

  Further, in each of the above embodiments, the determination of the attribute of the sound is performed using the learned model, but the present invention is not limited to this. For example, an embedding code may be added to the sound of the microwave oven, and the attribute of the sound may be determined based on the embedding code included in the sound acquired by the microphone 10. Further, for example, a communication module may be installed in a microwave oven, and the attribute may be acquired via communication at the timing when the microwave sounds.

  In the fourth embodiment, the living room is divided into a plurality of areas and grouped, but the present invention is not limited to this. For example, they may be grouped by sound frequency band. In this case, it is possible to perform grouping using differences in parameters that vary depending on environmental conditions such as reflection, absorption, and diffraction depending on the frequency of the sound source.

  In the above embodiments, the CNN has been described as an example of the neural network, but the neural network is not limited to this. For example, various neural networks such as RNN (Recurrent Neural Network) can be used. Further, as a learning technique, various known techniques other than the error back propagation can be adopted. The neural network has a multi-stage configuration including, for example, an input layer, an intermediate layer (hidden layer), and an output layer, and each layer has a structure in which a plurality of nodes are connected by edges. Each layer has a function called “activation function”, edges have “weight”, and each node value is calculated from the value of the node of the previous layer, the value of the weight of the connection edge, and the activation function of the layer. Is calculated. In addition, as a calculation method, various known methods can be adopted. As the machine learning, various methods such as SVM (Support Vector Machine) may be used in addition to the neural network.

  In addition, each component of each unit illustrated does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution / integration of each unit is not limited to the one shown in the figure, and all or a part thereof is functionally or physically distributed / integrated in arbitrary units according to various loads and usage conditions. Can be configured. For example, the specifying unit 132 and the determining unit 133 of the information display control device 100 may be integrated. Further, the illustrated processes are not limited to the above-described order, and may be performed simultaneously or may be performed in a different order as long as the processing contents are not contradictory.

  Further, various processing functions performed by each device may be entirely or arbitrarily executed on a CPU (or a microcomputer such as an MPU or an MCU (Micro Controller Unit)). The various processing functions may be entirely or arbitrarily executed on a program analyzed and executed by a CPU (or a microcomputer such as an MPU or an MCU) or on hardware by wired logic. It goes without saying that it is good.

  The various processes described in the above embodiments can be realized by executing a prepared program on a computer. Therefore, an example of a computer that executes a program having the same functions as the above embodiments will be described below. FIG. 22 is a diagram illustrating an example of a computer that executes an information display control program.

  As shown in FIG. 22, the computer 500 includes a CPU 501 for executing various arithmetic processing, an input device 502 for receiving data input, and a monitor 503. Further, the computer 500 includes a medium reading device 504 for reading a program or the like from a storage medium, an interface device 505 for connecting to various devices, and a communication device 506 for connecting to another information processing device or the like by wire or wirelessly. Having. Further, the computer 500 includes a RAM 507 for temporarily storing various information, and a hard disk device 508. Each of the devices 501 to 508 is connected to a bus 509.

  The hard disk device 508 stores an information display control program having functions similar to those of the display control unit 131, the specifying unit 132, the determination unit 133, the association unit 134, and the notification unit 135 shown in FIG. . The hard disk device 508 stores a layout storage unit 121, a learned model storage unit 122, an association information storage unit 123, and various data for implementing an information display control program. The hard disk device 508 stores an information display control program having the same functions as those of the display control unit 131, the specifying unit 132, the determination unit 133, the association unit 234, and the notification unit 235 illustrated in FIG. May be performed. The hard disk device 508 may store the layout storage unit 121, the learned model storage unit 122, the association information storage unit 223, and various data for realizing the information display control program.

  In addition, the hard disk device 508 stores an information display control program having the same functions as those of the display control unit 131, the specifying unit 132, the determination unit 133, the association unit 334, and the notification unit 235 illustrated in FIG. May be performed. The hard disk device 508 may store the layout storage unit 121, the learned model storage unit 122, the association information storage unit 223, and various data for realizing the information display control program. Further, the hard disk device 508 stores an information display control program having the same functions as those of the display control unit 131, the specifying unit 432, the determination unit 433, the association unit 434, and the notification unit 435 illustrated in FIG. May be performed. The hard disk device 508 stores a layout storage unit 121, a learned model storage unit 122, an association information storage unit 123, a correction information storage unit 424, and various data for implementing an information display control program. You may do so.

  The input device 502 receives input of various information such as operation information from the user of the computer 500, for example. The monitor 503 displays various screens such as a display screen for the user of the computer 500, for example. The interface device 505 is connected to, for example, a printing device. The communication device 506 has a function similar to that of the communication unit 110 shown in FIGS. 3, 7, 12, and 14, is connected to the network N, and exchanges various information with the microphone 10 and other information processing devices.

  The CPU 501 performs various processes by reading out each program stored in the hard disk device 508, developing it in the RAM 507, and executing it. Further, these programs can cause the computer 500 to function as the display control unit 131, the specifying unit 132, the determining unit 133, the associating unit 134, and the notifying unit 135 illustrated in FIG. Further, these programs may cause the computer 500 to function as the display control unit 131, the specifying unit 132, the determination unit 133, the association unit 234, and the notification unit 235 illustrated in FIG. In addition, these programs may cause the computer 500 to function as the display control unit 131, the specifying unit 132, the determining unit 133, the associating unit 334, and the notification unit 235 illustrated in FIG. Further, these programs may cause the computer 500 to function as the display control unit 131, the specifying unit 432, the determining unit 433, the associating unit 434, and the notification unit 435 illustrated in FIG.

  Note that the above information display control program does not necessarily need to be stored in the hard disk device 508. For example, the computer 500 may read out and execute a program stored in a storage medium readable by the computer 500. The storage medium readable by the computer 500 corresponds to, for example, a portable recording medium such as a CD-ROM, a DVD (Digital Versatile Disc), a USB (Universal Serial Bus) memory, a semiconductor memory such as a flash memory, and a hard disk drive. . Alternatively, the information display control program may be stored in a device connected to a public line, the Internet, a LAN, or the like, and the computer 500 may read out and execute the information display control program therefrom.

1 information display control system 10, 10a, 10b, 10c microphone 100, 200, 300, 400 information display control device 110 communication unit 111 display unit 112 operation unit 120, 220, 420 storage unit 121 layout storage unit 122 learned model storage unit 123, 223 associating information storage unit 130, 230, 330, 430 control unit 131 display control unit 132, 432 specifying unit 133, 433 determining unit 134, 234, 334, 434 associating unit 135, 235, 435 reporting unit 424 correction Information storage N network

Claims (13)

Display the installation positions of multiple objects on the display screen,
While specifying the position of the sound source generated by the operation of the object, determine the attribute of the sound source,
When there is a difference in the positional relationship between the specific position where the attribute is determined and the installation position of the object, the specific position where the attribute is determined is associated with the installation position of the object,
Notifying the operation state of the object on the display screen after the association,
An information display control program for causing a computer to execute processing. The notifying process does not display the specific position on the display screen when detecting the operation state of the target object, and displays an installation position of the target object on the display screen after the association. To inform,
The information display control program according to claim 1, wherein: The determining process displays the specific position on the display screen,
The associating process, by accepting an operation of associating the specific position displayed on the display screen with the installation position of the object, associating the specific position with the determined attribute with the installation position of the object,
The information display control program according to claim 1 or 2, wherein The associating process includes: accepting an operation of associating a marker representing an attribute of the target object displayed on the display screen with an installation position of the target object having the same attribute as the marker, thereby setting the marker to the target object. Corresponding to the installation position of
The information display control program according to claim 1 or 2, wherein The associating process is to associate the specific position with the installation position of the target object by using the association of the marker with the installation position of the target object as a trigger,
The information display control program according to claim 4, characterized in that: Identify the position of another sound source that is not the determination target of the attribute of the sound source,
Based on the correction information in the case where the specific position is associated with the installation position of the target, the specified position of the other sound source is corrected,
The information display control program according to claim 1, wherein the information display control program causes a computer to execute a process. The correcting process divides the area including the installation position of the target object into a plurality of areas, and for each of the divided areas, the other sound source in the same area based on the correction information of the specific position in the area. To correct the position of
The information display control program according to claim 6, wherein: The process of correcting the position of the other sound source based on correction information of the specific position in the area on both sides of the boundary when the position of the other sound source is located near the boundary between the two areas. to correct,
The information display control program according to claim 7, wherein: Display the installation positions of multiple objects on the display screen,
By accepting an operation for associating a marker representing the attribute of the object displayed on the display screen with the installation position of the object having the same attribute as the marker, the marker corresponds to the installation position of the object. Attached
While specifying the position of the sound source generated by the operation of the object, determine the attribute of the sound source,
Notifying the operation state of the target object that is the same attribute as the determined attribute on the display screen after the association,
An information display control program for causing a computer to execute processing. Display the installation positions of multiple objects on the display screen,
While specifying the position of the sound source generated by the operation of the object, determine the attribute of the sound source,
When there is a difference in the positional relationship between the specific position where the attribute is determined and the installation position of the object, the specific position where the attribute is determined is associated with the installation position of the object,
Notifying the operation state of the object on the display screen after the association,
An information display control method, wherein the processing is executed by a computer. Display the installation positions of multiple objects on the display screen,
By accepting an operation for associating a marker representing the attribute of the object displayed on the display screen with the installation position of the object having the same attribute as the marker, the marker corresponds to the installation position of the object. Attached
While specifying the position of the sound source generated by the operation of the object, determine the attribute of the sound source,
Notifying the operation state of the target object that is the same attribute as the determined attribute on the display screen after the association,
An information display control method, wherein the processing is executed by a computer. A display control unit that displays the installation positions of the plurality of objects on a display screen,
A determination unit that specifies the position of the sound source generated by the operation of the target object and determines the attribute of the sound source.
When there is a difference in the positional relationship between the specific position where the attribute is determined and the installation position of the object, an associating unit that associates the specific position where the attribute is determined with the installation position of the object,
A notification unit that notifies the operation state of the target object on the display screen after the association,
An information display control device comprising: A display control unit that displays the installation positions of the plurality of objects on a display screen,
By accepting an operation of associating a marker representing the attribute of the object displayed on the display screen with the installation position of the object having the same attribute as the marker, associating the marker with the installation position of the object A mapping unit;
A determination unit that specifies the position of the sound source generated by the operation of the target object and determines the attribute of the sound source.
A notification unit that notifies the operation state of the target object that is the same attribute as the determined attribute on the display screen after the association,
An information display control device comprising:

Images