JP6457700B2 - Display control system, display control method, and display control program - Google Patents

Description

  The present invention relates to a display control system, a display control method, and a display control program.

  A video call system that transmits video images and audio is known (for example, see Patent Document 1). In such a system, a moving image obtained by imaging a caller and a voice uttered by the caller are transmitted and received between terminals.

JP 2011-29732 A

  In a system that transmits images and sounds to the other party, such as a video call system, it may be difficult for the other party to hear the sound when all the sounds are transmitted to the other party. In addition, since all audio information is included in the data transmitted to the other party, the total amount of data to be transmitted increases and the communication load increases. On the other hand, when the data from which the sound is deleted is transmitted, information such as a sense of reality recognized from the sound is lost in the other party.

  Therefore, an object of the present invention is to reduce the amount of data required for transmission while preventing loss of information that can be recognized from voice in a system for transmitting images and sounds.

  In order to solve the above-described problem, a display control system according to an aspect of the present invention includes an acquisition unit that acquires image information associated with sound information, and an acquisition unit from among a plurality of preset categories related to sound. A category determination unit that determines a category to which sound information associated with the image information acquired by the method corresponds, and a selection generation that selects or generates display data corresponding to the sound information based on the category determined by the category determination unit And means for associating display data with image information instead of sound information.

  A display control method according to an aspect of the present invention is a display control method in a display control system, and includes an acquisition step of acquiring image information associated with sound information, and a plurality of preset categories related to sound. A category determination step for determining a category to which the sound information associated with the image information acquired in the acquisition step corresponds, and display data corresponding to the sound information is selected or generated based on the category determined in the category determination step And a selection generating step for associating and associating the display data with the image information instead of the sound information.

  A display control program according to an embodiment of the present invention is a display control program that causes a computer to function as a display control system, and relates to a computer that obtains image information associated with sound information, and preset sound. A category determination unit that determines a category corresponding to sound information associated with the image information acquired by the acquisition unit from among a plurality of categories, a display corresponding to the sound information based on the category determined by the category determination unit Selection selection means for selecting or generating data and association means for associating display data with image information instead of sound information.

  According to said form, the image information containing sound information is acquired and the category to which sound information corresponds is determined. Then, display data selected or generated based on the determined category is associated with the image information. Therefore, by replacing the sound information representing the sound with the display data, the data amount of the image information is reduced without deficient in the amount of information obtained from the sound in the image information. Thereby, for example, the communication load when transmitting image information is reduced.

  The display control system according to another aspect further includes a determination unit that determines whether or not to select or generate display data based on predetermined determination material information, and the selection generation unit selects or displays display data by the determination unit. When it is determined to generate, display data may be selected or generated.

  According to the above aspect, the sound information associated with the image information is selectively replaced with display data. Therefore, for example, sound information inappropriate to be transmitted as sound is replaced with display data, and sound information to be transmitted as sound remains associated with the image information. Thereby, the other party can be made to recognize the information regarding a sound appropriately.

  In the display control system according to another aspect, the image information includes an image representing the user's facial expression, and the determination unit selects or generates display data based on the user's facial expression in the image included in the image information. It may be determined whether or not.

  According to said form, the characteristic of the sound which generate | occur | produced in the space captured by the image represented by image information is acquired indirectly via a user's facial expression. Thereby, it is appropriately determined whether or not the sound information associated with the image information should be replaced with display data.

  In a display control system according to another embodiment, a plurality of display data is associated with a category, and the selection generation unit is configured to display a plurality of display data associated with each characteristic based on a sound characteristic indicated by sound information. Of these, at least one or more display data may be selected.

  According to said form, the display data according to the characteristic of sound information are selected by previously associating the display data from which a mode differs for every characteristic of sound information, for example. Thereby, the display data reflecting the characteristics of the sound information can be associated with the image information. Therefore, when the image information is displayed on the transmitted counterpart, it is possible to express the characteristics of the sound information.

  In the display control system according to another aspect, the selection generation unit may generate at least one or more display data based on a sound characteristic indicated by the sound information.

  According to the above aspect, the display data can be associated with the image information without requiring the display data to be associated with the category in advance.

  In the display control device according to another aspect, a plurality of display data is associated with the category, and the selection generation unit identifies an object corresponding to the category determined by the determination unit from the image, and the mode of the object Further, based on at least one of the positions, at least one or more display data among a plurality of display data associated with each category may be selected.

  The characteristics of the sound information may be reflected in the form of the object associated with the sound information through the determined category. According to the said form, the display data according to the aspect etc. of an object are selected by previously associating different display data for every aspect etc. of the object corresponding to a category, for example. Thereby, the display data reflecting the characteristics of the sound information can be associated with the image information. Therefore, when the image information is displayed on the transmitted counterpart, it is possible to express the characteristics of the sound information.

  In the display control system according to another aspect, the selection generation unit specifies an object corresponding to the category determined by the determination unit from the image, and at least one or more based on at least one of an aspect and a position of the object The display data may be generated.

  According to the above aspect, the display data can be associated with the image information without requiring the display data to be associated with the category in advance.

  The display control system according to another aspect may further include display control means for causing the display device to display the display data associated with the image information by the association means together with the image represented by the image information.

  According to said form, sound information is expressed by the display data displayed on the display apparatus. Therefore, it becomes easy to recognize sound information on the image information receiving side.

  In the display control system according to another aspect, the display data may be image data that visually represents the sound indicated by the sound information, and the display control means may display the image data on a display device.

  According to the above aspect, since the image data is displayed on the display device, the sound information is visually expressed on the display device.

  In the display control system according to another aspect, the display data is identification data associated in advance with image data representing the sound indicated by the sound information, and the display control means is configured in a terminal that receives the image information, The image data associated with the identification data may be displayed on the display device.

  According to the above aspect, the identification data associated in advance with the image data representing the sound is associated with the image information, and the image data is displayed on the display device at the terminal that has received the image information. As a result, the identification data for identifying the image data is transmitted in association with the image information, so that the communication load is reduced compared to the case in which the image data representing the image is transmitted in association with the image information.

  In the display control apparatus according to another aspect, the display control unit may change the display mode of the display data selected or generated by the selection generation unit based on the sound characteristics indicated by the sound information.

  According to the said form, the display data changed according to the characteristic of sound information can be displayed on a display apparatus. Therefore, it is possible to express the characteristics of the sound information by the display data displayed on the display device. In addition, according to this embodiment, it is not necessary to store display data corresponding to characteristics in advance, so that processing resources can be reduced.

  In the display control device according to another aspect, the display control unit specifies an object corresponding to the category determined by the determination unit from the image, and selects and generates based on at least one of the form and position of the object. The display mode of the display data selected or generated by the means may be changed.

  The characteristics of the sound information may be reflected in the form of the object associated with the sound information through the determined category. According to the said form, the display data which have the display mode changed according to the mode etc. of the object can be displayed on a display apparatus. Therefore, it is possible to express the characteristics of the sound information by the display data displayed on the display device.

  In the display control device according to another aspect, the display control means dynamically changes the display mode of the display data selected or generated by the selection generation unit based on at least one change in the mode and position of the object in the image. It is good also as changing to.

  According to the said form, according to the change of the aspect etc. of the object contained in the moving image represented by image information, it can display on a display apparatus, changing display data dynamically. Therefore, it is possible to express a dynamic change in the characteristics of the sound information by the display data displayed on the display device.

  In the display control device according to another aspect, the display control means may display the display data in association with a position in the image corresponding to the category determined by the determination means.

  According to the above aspect, for example, by associating information on an object that is a source of sound represented by the sound information with a category of sound information, display data is displayed at a position where the object is represented in the image. Can be displayed. Thereby, it is possible to more easily recognize what the sound represented by the sound information is.

  According to an aspect of the present invention, in a system for transmitting images and sound, it is possible to reduce the amount of data required for transmission while preventing loss of information that can be recognized from sound.

It is a figure which shows the apparatus structure of the display control system which concerns on this embodiment. It is a block diagram which shows the function structure of a display control apparatus. It is a figure which shows the hardware constitutions of a display control apparatus. It is a figure which shows the example of the structure of the display data memory | storage part, and the data memorize | stored. It is a figure which shows the example of the screen displayed by a display control part. It is a figure which shows the example of the screen displayed by a display control part. It is a figure which shows the example of the screen displayed by a display control part. It is a figure which shows the example of the screen displayed by a display control part. It is a figure explaining the example of the display control which changes display data dynamically. It is a flowchart which shows the processing content of the display control method implemented in a display control apparatus. It is a figure which shows the structure of a display control program.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the description of the drawings, the same or equivalent elements are denoted by the same reference numerals, and redundant description is omitted.

  FIG. 1 is a diagram illustrating a device configuration of a display control system according to the present embodiment. The display control system 1 includes a display control device 10 and terminals TA and TB, and these devices can communicate with each other via a network N.

  A camera TA and a microphone M are connected to the terminal TA. The terminal TA acquires a moving image from the camera C and acquires sound from the microphone M. The terminal TA may incorporate a camera and a microphone. A display device D and a speaker SP are connected to the terminal TB. The terminal TB can output an image to the display device D. The terminal TB can output sound to the speaker SP. Note that the terminal TB may incorporate a display device and a speaker.

  The display control apparatus 10 according to the present embodiment acquires image information from the terminal TA, and causes the display device D and the speaker SP of the terminal TB to output an image represented by the acquired image information and audio included in the image information. That is, in the present embodiment, the terminal TA is described as a transmitting side for image information and the terminal TB is described as a receiving side (the other party). Note that both the terminal TA and the terminal TB may include a camera, a microphone, a display device D, and a speaker, and may constitute a system that transmits and receives images and sounds in both directions. Examples of the system that transmits and receives images and sound in both directions include a video chat system and a conference system.

  The terminal T is configured by, for example, a stationary or portable personal computer, a high-function mobile phone (smart phone), or the like, but an apparatus that configures the terminal T is not limited, and for example, a mobile phone or a personal digital assistant (PDA) It may be a portable terminal such as. In FIG. 1, two terminals T are shown as an example, but the number of terminals T is not limited to this.

  FIG. 2 is a block diagram illustrating a functional configuration of the display control apparatus 10. The display control device 10 is configured by a server, for example.

As shown in FIG. 2, the display control apparatus 10 of the present embodiment is functionally an acquisition unit 11 (acquisition unit), a category determination unit 12 (category determination unit), and a selection generation unit 13 (selection generation unit).
A determination unit 14 (determination unit), an association unit 15 (association unit), and a display control unit 16 (display control unit) are provided.

  The display control apparatus 10 can access storage means such as the display data storage unit 20. The display data storage unit 20 may be provided in the display control device 10, or may be configured as an external storage unit provided so as to be accessible from the display control device 10.

  FIG. 3 is a hardware configuration diagram of the display control apparatus 10. As shown in FIG. 3, the display control device 10 is physically composed of a CPU 101, a main storage device 102 constituted by memories such as a RAM and a ROM, an auxiliary storage device 103 constituted by a hard disk, a communication control device 104, and the like. It is comprised as a computer system containing. The display control device 10 may further include an input device 105 such as a keyboard, a touch panel, and a mouse which are input devices, and an output device 106 such as a display.

  The functions shown in FIG. 2 operate the communication control device 104 and the like under the control of the CPU 101 by reading predetermined computer software on the hardware such as the CPU 101 and the main storage device 102 shown in FIG. In addition, it is realized by reading and writing data in the main storage device 102 and the auxiliary storage device 103. Data and databases necessary for processing are stored in the main storage device 102 and the auxiliary storage device 103. The terminal T is also configured as a computer system having a hardware configuration as shown in FIG.

  Subsequently, each functional unit of the display control apparatus 10 will be described. The acquisition unit 11 acquires image information representing an image. Sound information is associated with the image information. The sound information is information representing, for example, environmental sound generated in the space represented by the image of the image information, sound generated by the object represented by the image, and the like.

  In the present embodiment, the acquisition unit 11 acquires image information including moving images acquired at the terminal TA from the terminal TA. The image information is associated with sound information representing the sound acquired at the terminal TA. Specifically, the image information includes a moving image obtained by capturing the user of the terminal TA and the background of the user. Further, the sound information associated with the moving image can include a sound uttered by the user and an environmental sound uttered in the environment represented by the moving image. The environmental sound includes, for example, sounds other than the voice uttered by the user and generated by the user and sounds generated around the user.

  The category determination unit 12 determines a category to which the sound information associated with the image information acquired by the acquisition unit 11 corresponds, from among a plurality of categories related to sounds set in advance. The plurality of categories related to the sound information are concepts for classifying the sound that can be associated with the sound generation source and the cause of the sound, and are preset in the display data storage unit 20.

  The display data storage unit 20 is a storage unit that stores display data. The display data stored in the display data storage unit 20 is data that visually represents the sound represented by the sound information. For example, the display data stores the onomatopoeia text data, the onomatopoeia image data, and the sound. Symbols, images, etc., which are data to be displayed on the display device.

  FIG. 4 is a diagram illustrating a configuration of the display data storage unit 20 and an example of stored data. As shown in FIG. 4, the display data storage unit 20 stores display data in association with a category related to sound information and the volume of sound represented by the sound information / the magnitude of movement of the corresponding object. In addition, a corresponding object is associated with each category. In the corresponding object, an arbitrary object corresponding to the category is stored, but an object that is a sound source of sound information corresponding to the category is mainly stored.

  For example, in the category “mastication sound”, “mouth” which is a sound source of mastication sound is associated as a corresponding object. Further, display data “mogu-mogu”, “paku-paku”, and “baku-baku” corresponding to the volume / size of movement are associated with the category “chewing sound”. It has been. That is, “mogumogu”, “pakupaku”, and “bakubaku” are Japanese expressions representing masticatory sounds, and the volume or movement of the mouth increases in this order.

  Further, in the category “wind sound”, a “tree branch” in which the strength of the wind is reflected in the movement is associated with the corresponding object. The category “wind sound” is associated with display data “soyo-soyo”, “view-byu”, and “gou” corresponding to the volume / size of movement. . That is, “Soyoyo”, “View View”, and “Go” are Japanese expressions representing wind sounds, and the volume or the movement of the tree branches increases in this order.

  Further, the category “automobile engine sound” is associated with “automobile” which is a sound source of the engine sound of the automobile as a corresponding object. Further, display data “BRRR” and “BRRRRRRR” corresponding to the volume / size of motion are associated with the category “automobile engine sound”.

  Specifically, the category determination unit 12 first acquires sound information associated with the image information acquired by the acquisition unit 11. Subsequently, the category determination unit 12 may extract and acquire a predetermined sound from the sound information by a sound processing technique well known to those skilled in the art. For example, the category determination unit 12 may obtain an environmental sound by removing a human voice from the sound information. Moreover, the category determination part 12 may acquire the voice uttered by the user. The category determination unit 12 may extract all sounds included in the sound information. The sound extracted and acquired from the sound information may be set in advance, for example. The process of extracting a predetermined sound from the sound information can be performed by those skilled in the art using a well-known sound processing technique, and thus detailed description thereof is omitted. Subsequently, the category determination unit 12 performs Fourier transform on the acquired sound to acquire each frequency component.

  The category determination unit 12 normalizes the frequency component, inputs it to an SVM (Support Vector Machine), and classifies it into a category. SVM is one of pattern recognition models using supervised learning. In SVM, for example, chewing sounds, wind sounds, wave sounds, rain sounds, broken glass sounds, falling objects, sirens, intercom chimes, children's crying sounds, opening sounds of cans and bottles Is classified and stored as a class. The category determination unit 12 calculates the belonging probability of each class corresponding to the category of the frequency components of the acquired sound, and classifies the sound into a class for which a certain probability or more is calculated. Note that sound classification by SVM is possible based on techniques well-known to those skilled in the art and knowledge that is commonly possessed, and thus further detailed description is omitted. The category determination unit 12 determines that the acquired sound is, for example, a mastication sound (see FIG. 4).

  The selection generation unit 13 selects or generates display data corresponding to the sound information based on the category determined by the category determination unit 12. In the present embodiment, the selection generation unit 13 selects at least one or more display data from one or more display data stored in the display data storage unit 20. For example, when the category determination unit 12 determines that the category to which the sound information corresponds is “mastication sound”, the selection generation unit 13 associates with the category “mastication sound” in the display data storage unit 20. Of the displayed display data “mogumogu”, “pakupaku”, “bakubaku”, “mogumogu” is selected. Here, as an example, “mumbling” is selected as the default of the three display data, but selection processing for selecting one or a plurality of display data from a plurality of display data will be described later. .

  In addition, the selection production | generation part 13 is good also as producing | generating the display data corresponding to sound information similar to the display data memorize | stored in the display data memory | storage part 20 based on a category. The selection generation unit 13 is, for example, character image data having text, character size, display mode, and the like determined based on the size of the sound extracted from the sound information, the size of the movement of the object corresponding to the sound, etc. Can be generated as display data.

  The determination unit 14 determines whether to select or generate display data based on predetermined determination material information. Specifically, the determination unit 14 does not determine to select or generate display data corresponding to sound information that should be transmitted as sound to the other party's terminal TB. On the other hand, the determination unit 14 determines to select or generate display data corresponding to sound information that is inappropriate to be transmitted as sound to the other party's terminal TB. The selection generation unit 13 selects or generates display data when the determination unit 14 determines to select or generate display data.

  The predetermined determination material information regarding the determination of selection or generation of display data is information preset by the user, for example. For example, the display control device 10 stores determination material information in which the necessity of selection or generation of display data is preset for each category (for example, stored in the display data storage unit 20). The necessity of selection or generation of display data may be determined with reference to predetermined determination material information.

  When the image information includes an image representing the facial expression of the user, the determination unit 14 determines whether to select or generate display data based on the facial expression of the user in the image included in the image information. Also good. That is, the predetermined determination material information may be an image representing the user's facial expression. Specifically, for example, the determination unit 14 determines to select or generate display data when the user's facial expression is analyzed to determine that the user is uncomfortable. That is, when determining that the generated sound is an unpleasant sound based on the user's facial expression, the determining unit 14 determines to select or generate display data corresponding to the sound.

  Whether or not the user is uncomfortable can be determined based on, for example, distortion of the shape of the user's mouth, eyebrows, eyes, and the like. For example, a well-known face image analysis method (A Method for Measuring Comfortableness / uncomfortableness of Human by Analyzing Facial, http://ci.nii.ac.jp/naid/110004820683/) can be applied.

  In addition, when the determination unit 14 matches the time when a certain sound is generated in the sound information and the time when the facial expression indicating that the user has an uncomfortable feeling in the image information matches a certain time or more, It is determined that the sound is an unpleasant sound, and the display data storage unit 20 stores that the category corresponding to the sound is a category corresponding to the unpleasant sound. If sound information classified into the same category is acquired after the determination, the determination unit 14 determines to select or generate display data corresponding to the sound.

  The associating unit 15 associates display data with image information instead of sound information. Specifically, the associating unit 15 associates the display data selected or generated by the selection generating unit 13 with the image information instead of the previously associated sound information. That is, the sound information associated with the image information is deleted. By associating the display data with the image information and deleting the sound information in this way, the data amount of the image information is reduced without deficient in the amount of information obtained from the sound in the image information. Thereby, for example, the communication load when transmitting image information is reduced.

  The display control unit 16 causes the display device to display the display data associated with the image information by the association unit 15 together with the image represented by the image information. For example, the display control unit 16 may cause the display device D to display the display data associated with the image information by the associating unit 15 on the image represented by the image information. FIG. 5 is a diagram illustrating an example of a screen displayed by the display control unit 16.

  As illustrated in FIG. 5, the image information acquired by the acquisition unit 11 includes a moving image obtained by capturing the user of the terminal TA. Therefore, the display control unit 16 displays the moving image of the user of the terminal TA on the terminal TB. Display on display device D. Then, the display control unit 16 causes the display data T1 “mogumogu” selected by the selection generation unit 13 to be displayed superimposed on the moving image of the user of the terminal TA.

  In this way, the display data is displayed on the display device D, whereby the environmental sound acquired at the terminal TA is visually expressed at the terminal TB. Therefore, it becomes easy for the user of the terminal TB to recognize the environmental sound. Further, compared to transmitting the audio signal acquired at the terminal TA to the terminal TB in real time, it is possible to reduce resources for transmission.

  In the example described with reference to FIG. 5, the display data is image data that visually represents the sound indicated by the sound information, and the display control unit 16 transmits the image data (display data) to the counterpart terminal. The image data is transmitted to the TB and displayed on the display device D of the terminal TB. On the other hand, in the mode described below, display of image data on the display device D is realized by transmitting identification data associated with the image data to the terminal TB instead of the image data. It is good as well. That is, the display data associated with the image information may be identification data associated in advance with the image data instead of the image data.

  Specifically, in this aspect, the display control unit 16 is configured in the terminal TB. The associating unit 15 corresponds to the display data selected by the selection generating unit 13 based on the category determined by the category determining unit 12, and the identification data associated in advance with the image data representing the sound indicated by the sound information. Associate with image information. Then, the associating unit 15 transmits the image information associated with the identification data to the terminal TB. The terminal TB stores an association between identification data and image data in advance. Then, the display control unit 16 configured in the terminal TB causes the display device D to display image data associated with the identification data.

  According to this aspect, the identification data previously associated with the image data that visually represents the sound is associated with the image information, and the image data is displayed on the display device at the terminal that has received the image information. As a result, the identification data for identifying the image data is transmitted in association with the image information, so that the communication load is reduced compared to the case in which the image data representing the image is transmitted in association with the image information.

  Next, an example of selection of display data will be described. The selection generation unit 13 may select at least one or more display data from among a plurality of display data associated with each characteristic based on the characteristic of the sound information acquired by the acquisition unit 11. The characteristics of the sound information are, for example, volume and frequency height.

  Specifically, the selection generation unit 13 is a plurality of display data associated with the category determined by the category determination unit 12, and the sound information is selected from the plurality of display data associated with the volume. One or more display data is selected on the basis of the characteristics such as the volume. In the display data storage unit 20 shown in FIG. 4, for example, three display data are stored in association with each other according to the volume of the mastication sound. The selection generation unit 13 determines the volume level of the sound information based on a predetermined threshold. And the selection production | generation part 13 selects the display data linked | related with the determined sound volume. Here, it is assumed that the selection generation unit 13 determines that the volume of the sound information is medium.

  In this case, the selection generation unit 13 selects the display data “Pakupaku”. Then, the associating unit 15 associates the display data “Pakupaku” with the image information instead of the sound information. FIG. 6 is a diagram illustrating an example of a screen displayed in such a case. As shown in FIG. 6, the display control unit 16 superimposes the display data T2 “Pakpaku” associated with the image information by the associating unit 15 on the moving image of the user of the terminal TA, and displays it on the display device D of the terminal TB. Display.

  In this way, display data corresponding to the characteristics of the sound information is selected by associating display data having a different aspect for each characteristic of the sound information in advance. Thereby, the display data in which the characteristic of the sound information is reflected can be displayed on the display device. Therefore, it is possible to express the characteristics of the sound information by the display data displayed on the display device. In addition, when the characteristic of sound information changes dynamically, according to the change, the display data which the selection production | generation part 13 selects are changed dynamically, and the display control part 16 displays the changed display data. It may be displayed on the device D. The selection generation unit 13 generates display data corresponding to the sound characteristics instead of selecting the display data stored in the display data storage unit 20 based on the sound characteristics indicated by the sound information. It is good as well.

  Next, an example of display control of display data by the display control unit 16 will be described with reference to FIG. The display control part 16 is good also as changing the display mode of the display data selected by the selection production | generation part 13 based on the characteristic of sound information.

  Specifically, for example, the display control unit 16 determines the volume of the sound information acquired by the acquisition unit 11. The display control unit 16 may determine the volume level stepwise based on a predetermined threshold value, or may measure the volume value. Here, assuming that the display data “mogumogu” is selected by the selection generation unit 13, the display control unit 16 displays one of the display modes of the display data based on the volume of the determined sound information as shown in FIG. The display data T3 “mogumogu” is displayed by changing the size of the display data T3 more greatly than in FIG.

  In addition, how to change the display mode based on the characteristics is set in advance, for example. Moreover, the display control part 16 may change the display mode of display data based not only on a sound volume but on a frequency. In addition to changing the size of the display data as in the example shown in FIG. 7, the display control unit 16 may change the font, color, and display position of the display data. Further, the display control unit 16 may repeatedly display the display data based on the characteristics of the sound information.

  As described above, in the present embodiment, the display data changed according to the characteristics of the sound information can be displayed on the display device. Therefore, it is possible to express the characteristics of the sound information by the display data displayed on the display device. Further, according to this embodiment, it is not necessary to store display data changed in accordance with characteristics in advance, so that processing resources can be reduced. In addition, when the characteristic of sound information changes dynamically, the display control part 16 is good also as changing the display mode of display data dynamically according to the change.

  Next, an example of selection of display data based on an object in an image will be described. The selection generation unit 13 identifies an object corresponding to the category determined by the category determination unit 12 from the image, and a plurality of displays associated with each category based on at least one of the form and position of the object It is good also as selecting at least 1 or more display data of data. FIG. 8 is a diagram illustrating an example of display of an image and display data for explaining the display data selection processing.

  Specifically, when the category determination unit 12 determines that the category to which the sound information corresponds is “wind sound”, the selection generation unit 13 displays the category in the display data storage unit 20 (see FIG. 4). The object “tree branch” corresponding to “wind sound” is acquired, and the object Ob4 representing the tree branch is specified from the image V4. The predetermined object can be specified from the image by a known image processing technique such as pattern matching.

  The selection generation unit 13 determines at least one of the aspect and position of the object Ob4. Here, the selection generation unit 13 acquires the magnitude of the movement of the object Ob4 (tree branch) as the form of the object. Note that the magnitude of the movement of an object in an image can also be acquired by a known image processing technique. The selection generation unit 13 determines the magnitude of the movement of the object Ob4 based on a predetermined threshold. And the selection production | generation part 13 selects the display data linked | related with the magnitude | size of the determined motion in the display data memory | storage part 20. FIG. Here, for example, the selection generation unit 13 determines that the magnitude of the movement of the object Ob4 is medium, and selects the display data T4 “view view” corresponding to the magnitude of the movement. Then, the associating unit 15 associates the display data T4 “view view” with the image information instead of the sound information. As shown in FIG. 8, the display control unit 16 superimposes the display data T4 “view view” associated with the image information by the association unit 15 on the moving image of the user of the terminal TA, and displays the display device D of the terminal TB. To display.

  As described above, in consideration of the fact that the characteristics of the sound information may be reflected in the aspect of the object corresponding to the category, different display data is associated in advance for each aspect of the object corresponding to the category. Thus, display data corresponding to the form of the object is selected. Thereby, the display data reflecting the characteristic of the environmental sound can be displayed on the display device. Therefore, it is possible to express the characteristics of the environmental sound by the display data displayed on the display device. Further, the selection generation unit 13 specifies an object corresponding to the category determined by the category determination unit 12 from the image, and is stored in the display data storage unit 20 based on at least one of the form and position of the object. Instead of selecting the displayed display data, display data corresponding to at least one of the form and position of the object may be generated.

  In addition, the display control unit 16 identifies an object corresponding to the category determined by the category determination unit 12 from the image, and is selected by the selection generation unit 13 based on at least one of the form and position of the object. The display mode of the display data may be changed.

  Specifically, for example, when the category determination unit 12 determines that the category to which the environmental sound corresponds is “wind sound” and the selection generation unit 13 selects the display data “view view”, the display control is performed. The unit 16 acquires an object “tree branch” corresponding to the category “wind sound” in the display data storage unit 20 (see FIG. 4), and identifies an object representing a tree branch from the image represented by the image information. To do. Then, the display control unit 16 determines at least one of the aspect and position of the object. Here, the display control unit 16 acquires the magnitude of the movement of the object (tree branch) as the form of the object. Then, the display control unit 16 superimposes the display data “view view” in which the size, which is one of the display modes, is changed based on the acquired size of the movement of the object on the moving image of the user of the terminal TA. Display on the display device D of the terminal TB.

  In addition, how to change the display mode based on the characteristics is set in advance, for example. In addition to changing the size of the display data, the display control unit 16 may change the font, color, and display position of the display data. The display control unit 16 may repeatedly display the display data based on the environmental sound characteristics.

  Thus, in consideration of the fact that the characteristics of sound information may be reflected in the object mode corresponding to the category, display data having a display mode changed according to the object mode or the like By displaying on the display device, the characteristics of the sound information can be expressed by the display data displayed on the display device.

  Further, the display control unit 16 may dynamically change the display mode of the display data selected by the selection unit based on at least one change in the mode and position of the object corresponding to the category in the image. Good.

  FIG. 9 is a diagram illustrating an example of display control that dynamically changes display data. FIG. 9 shows a moving image MV5 represented by the image information acquired by the acquisition unit 11, and shows a state in which an object Ob51 representing a car moves to a position indicated by the object Ob52.

  In this example, first, the category determination unit 12 determines that the environmental sound included in the image information representing the image shown in FIG. 9 corresponds to the category “automobile engine sound”. Then, the selection generation unit 13 selects the display data “BRRR” associated with the category “automobile engine sound”. In this selection, the selection generation unit 13 may select one display data among a plurality of display data based on the volume of the environmental sound, or may select predetermined default display data. Also good. The associating unit 15 associates the display data “BRRR” with the image information instead of the sound information.

  The display control unit 16 specifies the object “car” corresponding to the category “car engine sound” from the MV 5. That is, the display control unit 16 specifies the object Ob51. Here, the display control unit 16 displays the display data in association with the position corresponding to the category determined by the category determination unit 12 in the image. Specifically, the display control unit 16 displays the display data T51 “BRRR” in association with the display position of the object Ob51, which is the position where the object “car” corresponding to the category “car engine sound” is represented. .

  Subsequently, when the automobile moves to the position indicated by the object Ob52 in the moving image MV5, the display control unit 16 displays the display data T52 “BRRR” in association with the display position of the object Ob52. In addition, the display control unit 16 displays the display data T52 “BRRR” in which the display size is largely changed based on the change in the position and size of the objects Ob51 and Ob52.

  As described above, according to the example described with reference to FIG. 9, the display data is dynamically changed in accordance with the change in the aspect of the object included in the moving image represented by the image information. Can be displayed. Therefore, it is possible to express a dynamic change in the characteristics of the sound information by the display data displayed on the display device. Further, by associating the sound information category with the information of the object that is the source of the sound indicated by the sound information, the display data can be displayed at the position where the object is represented in the image. Thereby, it is possible to more easily recognize what the sound of the sound information is.

  The change in the position of the object in the moving image, in particular, the change in the position of the object in the depth direction of the image can be obtained by configuring the camera C for acquiring an image in the terminal TA by, for example, an RGB-D camera. Can be acquired easily. The RGB-D camera can acquire an image with depth information.

  In addition, when the position of the object in the moving image moves forward from the back position in the image, even when the volume does not change, the display control unit 16 does not change when the object moves forward. The display data may be displayed with great emphasis.

  Further, the display mode of the display data may be corrected based on whether the position of the object in the moving image is a back position in the image or is troublesome. For example, when an object is located in the back of the image, the volume of sound using the object as a sound source is relatively low. In such a case, the display control unit 16 may display the display data with a relatively large display size so as to correct the volume level.

  Next, the operation of the display control apparatus 10 of this embodiment will be described with reference to FIG. FIG. 10 is a flowchart showing the processing contents of the display control method implemented in the display control apparatus 10.

  First, the acquisition unit 11 acquires image information representing an image and associated with sound information (S1). Subsequently, the category determination unit 12 determines a category to which the sound information associated with the image information acquired by the acquisition unit 11 falls within a plurality of categories related to the sound set in advance (S2).

  Next, the selection generation unit 13 selects or generates display data corresponding to the sound information based on the category determined in step S2 (S3). Then, the associating unit 15 associates the display data selected by the selection generating unit 13 in step S3 with the image information instead of the sound information (S4).

  Next, a display control program for causing a computer to function as the display control apparatus 10 will be described with reference to FIG. The display control program p1 includes a main module m10, an acquisition module m11, a category determination module m12, a selection generation module m13, a determination module m14, an association module m15, and a display control module m16.

  The main module m10 is a part that performs overall control of display control processing. Functions realized by executing the acquisition module m11, the category determination module m12, the selection generation module m13, the determination module m14, the association module m15, and the display control module m16 are each an acquisition unit of the display control apparatus 10 illustrated in FIG. 11, the functions of the category determination unit 12, the selection generation unit 13, the determination unit 14, the association unit 15, and the display control unit 16.

  The display control program p1 is provided by a storage medium d1 such as a magnetic disk, an optical disk, or a semiconductor memory, for example. The display control program p1 may be provided via a communication network as a computer data signal superimposed on a carrier wave.

  According to the display control system 1, the display control method, and the display control program p1 of the present embodiment described above, image information including sound information is acquired, and a category to which the sound information corresponds is determined. Then, display data selected or generated based on the determined category is associated with the image information. Therefore, by replacing the sound information representing the sound with the display data, the data amount of the image information is reduced without deficient in the amount of information obtained from the sound in the image information. Thereby, for example, the communication load when transmitting image information is reduced.

  The present invention has been described in detail based on the embodiments. However, the present invention is not limited to the above embodiment. The present invention can be variously modified without departing from the gist thereof.

  For example, in the present embodiment, the display control apparatus 10 includes the acquisition unit 11, the category determination unit 12, the selection generation unit 13, the determination unit 14, the association unit 15, and the display control unit 16. ˜16 may be configured as any one of the terminals TA and TB. That is, the display control system 1 may be realized as a system in which the terminal TA and the terminal TB communicate directly without having the display control device 10 configured by, for example, a server. In that case, all of the functional units 11 to 16 may be configured in the terminal TB. Further, all of the functional units 11 to 16 may be configured in the terminal TA. Further, the functional units 11 to 16 may be configured to be distributed to the terminal TA and the terminal TB.

DESCRIPTION OF SYMBOLS 1 ... Display control system, 10 ... Display control apparatus, 11 ... Acquisition part, 12 ... Category determination part, 13 ... Selection production | generation part, 14 ... Determination part, 15 ... Association part, 16 ... Display control part, 20 ... Display data storage Part, D ... display device, d1 ... storage medium, m10 ... main module, m11 ... acquisition module, m12 ... category determination module, m13 ... selection generation module, m14 ... determination module, m15 ... association module, m16 ... display control module, N ... Network, p1 ... Display control program, T, TA, TB ... Terminal.

Claims (18)

Acquisition means for acquiring image information associated with sound information;
The display data corresponding to the sound information is selected or generated based on a category to which the sound information associated with the image information acquired by the acquisition unit is selected from among a plurality of categories related to a preset sound. Selection generating means to
Association means for associating the display data selected or generated by the selection generation means with the image information instead of the sound information corresponding to the display data;
A display control system comprising: The sound information is information representing a sound generated in a space represented in the image of the image information, and is acquired by a microphone.
The display control system according to claim 1. The association unit deletes the sound information corresponding to the display data selected or generated by the selection generation unit from the image information;
The display control system according to claim 1 or 2. A judgment means for judging whether to select or generate the display data based on predetermined judgment material information;
The selection generation unit selects or generates the display data when the determination unit determines to select or generate the display data.
The display control system as described in any one of Claims 1-3. The image information includes an image representing a user's facial expression,
The determination means determines whether to select or generate the display data based on a user's facial expression in an image included in the image information.
The display control system according to claim 4. A plurality of display data is associated with the category,
The selection generation unit selects at least one or more display data from among a plurality of display data associated with each characteristic based on a category to which the sound information corresponds and a sound characteristic indicated by the sound information. ,
The display control system as described in any one of Claims 1-5. The selection generation means generates at least one or more display data based on a sound characteristic indicated by the sound information.
The display control system according to any one of claims 1 to 6. A plurality of display data is associated with the category,
The selection generation unit specifies an object corresponding to the category to which the sound information corresponds from the image represented by the image information, and associates the object for each category based on at least one of an aspect and a position of the object. Selecting at least one or more display data from a plurality of display data being displayed,
The display control system as described in any one of Claims 1-7. The selection generation unit identifies an object corresponding to a category to which the sound information corresponds from an image represented by the image information , and displays at least one or more displays based on at least one of an aspect and a position of the object Generate data,
The display control system as described in any one of Claims 1-8. Further comprising display control means for causing the display device to display the display data associated with the image information by the association means together with the image represented by the image information.
The display control system according to any one of claims 1 to 9. The display data is image data that visually represents the sound indicated by the sound information,
The display control means causes the display device to display the image data.
The display control system according to claim 10. The display data is identification data associated in advance with image data representing the sound indicated by the sound information,
The display control means includes
A terminal configured to receive the image information;
Displaying the image data associated with the identification data on the display device;
The display control system according to claim 10. The display control means changes a display mode of display data selected or generated by the selection generation means based on a sound characteristic indicated by the sound information.
The display control system according to any one of claims 10 to 12. The display control means includes
An object corresponding to the category corresponding to the sound information is identified from the image, and the display mode of the display data selected or generated by the selection generation unit is changed based on at least one of the mode and position of the object. Let
The display control system according to any one of claims 10 to 13. The display control means includes
Dynamically changing the display mode of the display data selected or generated by the selection generation unit based on at least one change in the mode and position of the object in the image;
The display control system according to claim 14. The display control means includes
In the image, the display data is displayed in association with a position corresponding to the category to which the sound information corresponds.
The display control system according to any one of claims 10 to 15. A display control method in a display control system, comprising:
An acquisition step of acquiring image information associated with sound information;
The display data corresponding to the sound information is selected or generated based on a category to which the sound information associated with the image information acquired in the acquisition step corresponds among a plurality of categories related to a preset sound. A selection generation step to
Associating the display data selected or generated in the selection generation step with the image information instead of the sound information corresponding to the display data;
A display control method. A display control program for causing a computer to function as a display control system,
The computer,
Acquisition means for acquiring image information associated with sound information;
The display data corresponding to the sound information is selected or generated based on a category to which the sound information associated with the image information acquired by the acquisition unit is selected from among a plurality of categories related to a preset sound. Selection generating means, and associating means for associating the display data selected or generated by the selection generating means with the image information instead of the sound information corresponding to the display data,
Display control program to function as.

Images