JP2012054897A - Conference system, information processing apparatus, and information processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a conference system, an information processing apparatus, and an information processing method, capable of suppressing unpleasantness and discomfort to other conference participants by suppressing transmission and reception of image data and voice data based on an operation analysis by image recognition in the conference system.SOLUTION: A conference server apparatus 3 receives images photographed by cameras 15 of respective terminal devices 1, 1, ..., performs image recognition processing for the received images and detects an inappropriate operation from an image recognition result. When the inappropriate operation is detected, the conference server apparatus 3 controls to stop transmission or process a part thereof so that the image indicating the inappropriate operation cannot be confirmed through the other terminal devices 1, 1, ....

Description

  The present invention relates to a conference system that transmits images picked up by a camera or a sound collected by a microphone between a plurality of information processing apparatuses, and can realize a conference between conference participants even if they are remote. In particular, a conference system capable of detecting the motion of a conference participant, controlling transmission / reception of images or sounds according to the detected motion status of the participant, and suppressing discomfort and discomfort to other conference participants The present invention relates to an information processing apparatus and an information processing method.

  Along with the development of communication technology, image processing technology, etc., a video conference system has been realized that enables a conference between a plurality of information processing apparatuses respectively installed at two remote sites or a plurality of three or more sites via a network. . Since large-capacity data can be transmitted and received, not only is the voice data collected by the terminal device transmitted to other terminal devices to share the speech of the speaker, but each terminal device A conference system (so-called Web conference system) that can realize a conference with facial expressions, gestures, etc. by photographing a conference participant with a terminal device and transmitting the captured video data to another terminal device has been put into practical use. Yes.

  In a conventional conference system, each information processing device designates a telephone number or an IP (Internet Protocol) address and establishes a connection directly with another information processing device, and the two information processing devices have one-to-one audio data. And by exchanging image data. When realizing a conference system between three or more information processing devices, one information processing device is a parent device, other information processing devices are child devices, and a plurality of child devices are each a parent device. And the parent device relays data exchange between the child devices.

  In order to realize a conference system between more bases, a plurality of information processing devices are connected to an MCU (Multipoint Control Unit) in a star shape, and data exchange between the information processing devices is performed by the MCU. There is a configuration that relays. In a conference system using an MCU, the number of information processing devices (bases) that can participate in the conference system depends on the performance of the MCU, that is, the number of information processing devices that can be connected (for example, the number of communication ports).

  In order to realize a conference system between many sites, an information processing device used by a conference participant connects to a server device as a client device via a communication network such as a LAN (Local Area Network) or the Internet. There is also a configuration in which data exchange is relayed by the server device. In such a server / client system configuration, although there are limitations on the processing capacity of the server device and the network communication speed (usable bandwidth), the number of sites participating in the conference system compared to the configuration using the MCU. There is an advantage that (the number of information processing devices) can be easily increased or decreased.

  In this way, a conference can be realized by using each information processing apparatus individually (or by a small number of people) for each information processing apparatus, regardless of the configuration using the MCU or the configuration of the server / client system. At this time, in each information processing apparatus, a liquid crystal panel that displays a shared screen, a display that uses an organic EL panel, a camera that shoots a conference participant who uses the apparatus, and a conference participant who uses the apparatus are collected. A microphone for sounding and a speaker for outputting sound are provided. The information processing apparatus transmits and receives captured video (image) data and collected audio data via the MCU or the server apparatus. Thereby, it is possible to hold a conference by sharing the remarks, facial expressions, gestures, etc. of the conference participants.

  In Patent Document 1, when using a conference system for exchanging audio and images and conducting a conference between conference participants in different cultural spheres, the motion of each conference participant is animated using a three-dimensional avatar or the like. An invention is disclosed in which the gestures of conference participants in a certain cultural sphere are partly changed so that they can be regarded as socially appropriate in other cultural spheres.

  Patent Document 2 discloses an invention in which, in a conference system, a speech by a conference participant is recognized by voice recognition, converted into character information, and displayed on a common screen by deleting unnecessary words from the character information. It is disclosed.

JP 2009-77380 A JP-A-10-301927

  The conference system that can realize a conference while confirming the image of the other party while being in a remote place and its peripheral technology play a large role in improving communication between people of different cultures and languages. Conversely, in a conference system that transmits and receives images of conference participants, the operations of the conference participants are transmitted to other conference participants in real time. This may cause various problems and concerns that did not exist in communication via other media.

  For example, in a conference system that transmits and receives video of conference participants between a plurality of sites, video signals of conference participants are transmitted to other sites in real time. In this case, the non-verbal movements and facial expressions of the conference participants are important. In particular, it may be a problem if conference participants are in the background of diverse cultures and languages. Naturally, the appropriateness of whether or not the language is appropriate for the situation will vary greatly depending on the culture. Actions that are appropriate and allowed in one culture are often considered inappropriate and not allowed in another culture, which can be misleading.

  In a business place where international exchanges are performed, the appropriateness of the operation of the conference participants when using the conference system is important. Depending on the behavior, it is possible to build trust over language-only communication. Trust can be expressed by appropriate gaze at the right time, proper gestures and facial expressions, and it determines the success or failure of the transaction. Conversely, improper gestures at improper timing can greatly reduce trust, even if done unconsciously.

  Conference participants can learn appropriate behaviors in different cultures as background knowledge, but it is very difficult to maintain their suitability even with unpatterned behaviors. It is also unrealistic for a conference participant to have all the differences in customs and traditions in the culture of other conference participants in all conferences that may participate.

  Furthermore, it is unrealistic to correct in advance an action that may be unconsciously performed by a conference participant and that may make other conference participants uncomfortable.

  By using the technology disclosed in Patent Document 1, when a conference participant performs gestures that are perceived as inappropriate culturally or socially by other conference participants in the conference system. Even if it exists, it substitutes for another gesture using an avatar, or switches to images other than a meeting participant's image, for example, a shared screen, such as a meeting material. Thereby, it is possible to avoid making other conference participants uncomfortable.

  However, in the technique disclosed in Patent Document 1, in order to determine whether or not the operation is inappropriate, it is necessary to acquire image data obtained by photographing a conference participant and perform an operation analysis according to the culture model data. In Patent Document 1, it is possible to use data obtained from a motion sensor or an acceleration sensor or information from voice for motion analysis. However, the actual analysis target and analysis using the information are described. Examples of are not described. Further, in the technique disclosed in Patent Document 1, motion analysis is performed according to culture model data. In order to deal with various cultures, a huge amount of culture model data for each culture is stored in a database in advance. It is necessary to set the load on the analysis process, and the load of the analysis process is enormous.

  Further, by using the technique disclosed in Patent Document 2, it is also possible to delete words / phrases that are regarded as inappropriate from the content of the speech of the conference participant and display them on the common screen. This also makes it easier for conference participants to see the common screen, and also facilitates the creation of minutes because it has already been converted into text information.

  However, with the technique disclosed in Patent Document 2, although it is possible to delete the information received as inappropriate, it can be deleted from the character information to display, but the audio and the video being spoken are already shared and other conference participants Has been communicated to. Therefore, it is impossible to avoid discomfort and discomfort to other conference participants other than the speaker.

  The present invention has been made in view of such circumstances, and the conference system controls transmission / reception of image data and audio data based on an operation analysis based on image recognition, thereby causing discomfort and discomfort to other conference participants. It is an object of the present invention to provide a conference system that can be suppressed, an information processing apparatus that controls the conference system, and an information processing method.

  The conference system according to the present invention includes a first imaging device or a sound collecting device, a means for acquiring an image from the imaging device or a sound from the sound collecting device, and a transmission / reception means for transmitting and receiving the acquired image or sound. A plurality of information processing devices, and a plurality of second information processing devices connected to the plurality of first information processing devices via communication media and relaying images or sounds transmitted and received by each first information processing device, In a conference system that realizes a conference by displaying or outputting a common image or sound between the first information processing devices to realize a conference, recognizes the action of a person shown in the image acquired by the first information processing device Recognizing means, detecting means for detecting presence / absence of an inappropriate operation based on the recognition result by the recognizing means, and reception of an image or sound from the first information processing apparatus according to the detection result of the detecting means, Properly it is characterized by having a transmission control means for controlling the processing of the possibility of transmission to another first information processing apparatus, increase or decrease of the transmission rate, or to a part of the image.

  In the conference system according to the present invention, when the detection unit detects an inappropriate operation, the transmission control unit receives an image or sound from the first information processing apparatus in which the inappropriate operation is detected, or Transmission to another first information processing apparatus is prohibited, a transmission rate is reduced, or processing of a part of the image is performed.

  The conference system according to the present invention includes means for measuring a duration of the operation after the detection unit detects an inappropriate operation, and a determination unit for determining whether the duration is a predetermined time or more. And the transmission control means receives the image or sound from the first information processing apparatus in which the inappropriate operation is detected, or the image or Transmission of audio to the first information processing apparatus is prohibited, a transmission rate is reduced, or processing of a part of the image is performed.

  The conference system according to the present invention includes a table that is a list of image recognition results registered in advance as inappropriate operations, and the detection unit corresponds to the recognition result included in the table by the recognition result by the recognition unit. The presence or absence of an inappropriate operation is detected depending on whether or not it is performed.

  In the conference system according to the present invention, when the detection unit detects an operation accompanied by inappropriate voice or utterance, the image or voice is received from the first information processing apparatus in which the inappropriate operation is detected, or Transmission of the image or sound to another first information processing apparatus is prohibited.

  In the conference system according to the present invention, the transmission control unit may include another image on a part of the image corresponding to the inappropriate operation among the images from the first information processing apparatus in which the inappropriate operation is detected. It is characterized by processing to superimpose an image.

  In the conference system according to the present invention, the recognition unit and the detection unit are provided in a second information processing apparatus.

  In the conference system according to the present invention, when the detection unit detects an inappropriate operation, the imaging direction of the imaging device of the first information processing apparatus or the sound collection direction of the sound collection device in which the inappropriate operation is detected is changed. It has the means to instruct | indicate.

  The information processing apparatus according to the present invention recognizes the motion of a person appearing in a received image in an information processing apparatus that is connected to a plurality of other apparatuses via a communication medium and includes means for transmitting and receiving an image or sound to and from each apparatus. Means for detecting the presence or absence of an inappropriate operation based on the recognized result, and receiving an image or sound from the other device, or other image or sound based on the detected result. And means for controlling whether transmission to the apparatus is possible, increase / decrease in transmission rate, or processing of a part of the image.

  An information processing method according to the present invention includes: an imaging device or a sound collecting device; a means for acquiring an image from the imaging device or a sound from the sound collecting device; and a transmission / reception means for transmitting and receiving the acquired image or sound. A system including a plurality of one information processing device and a second information processing device that connects the plurality of first information processing devices via a communication medium and relays an image or sound transmitted / received by each first information processing device. In the information processing method for controlling transmission / reception of an image or sound to / from the first information processing apparatus, the person's movement shown in the image acquired by the first information processing apparatus is recognized, and based on the recognized result, inappropriate The presence / absence of an operation is detected, and depending on the detection result, whether or not the image or sound is received from the first information processing apparatus, or whether the image or sound is transmitted to the first information processing apparatus, the transmission rate of Reduced, or and controls the processing into a part of the image.

  In the present invention, it is detected by image recognition whether or not an improper operation is reflected in an image captured by the first information processing apparatus used by the conference participant, that is, an image that should capture the conference participant. Depending on the detection result, control is performed such as whether the acquired image or sound can be transmitted / received, the transmission rate is increased or decreased, or a part of the image is processed.

  Specifically, in the present invention, when an inappropriate operation is detected from an image that is to capture a conference participant, an image or sound from the first information processing device to another first information processing device is detected. Even if transmission is prohibited, even if transmission is performed, the transmission rate is reduced and the frame is dropped, or a part of the screen is processed. Alternatively, reception of an image or sound from the first information processing apparatus at the second information processing apparatus is prohibited. As a result, it becomes impossible to view an image showing an inappropriate operation on the other first information processing apparatus, or it becomes difficult to confirm clearly, or a part of the image is hidden. It becomes impossible to watch.

  In the present invention, when an inappropriate operation is detected continuously for a predetermined time or longer, transmission / reception is prohibited and viewing is impossible, or the transmission rate is reduced and frames are dropped, or part thereof Is processed and hidden, making it impossible to see. By monitoring whether the duration time is equal to or longer than the predetermined time, the progress of the conference is made smoother than when transmission is limited by only one detection. In addition, it is possible to give a warning to a person who has performed an inappropriate operation before the predetermined time has elapsed.

In the present invention, an inappropriate operation is detected based on a comparison reference with a table of image recognition results registered in advance as an inappropriate operation. By defining in advance, it is possible to reduce the load of detection processing of inappropriate motion based on image recognition. In addition, by using a table, it is easy to change the content detected as an inappropriate operation by appropriately updating the table without revising the content of the inappropriate operation detection process.
Inappropriate actions include, for example, “telephone, snooze, chat, quarrel, cry, laugh, leave, look away, smoke, eat, put out tongue”, and image recognition results corresponding to these actions. The comparison result is performed. It becomes possible to exclude the image or sound of the conference participant who performs those operations from the conference system.

In the present invention, an inappropriate action involving voice or utterance is detected by image recognition based on an opening / closing action of the mouth, etc., regardless of voice recognition, so that an inappropriate voice or utterance is transmitted to other conference participants. It can be detected before and excluded from the conference system.
Inappropriate actions involving voice or speech are, for example, actions such as phone calls, chats, fights, crying, or laughing, and in these cases, movements of the mouth, hands or There is a possibility that it can be detected by image recognition combined with arm movements.

  In the present invention, when an inappropriate motion is detected, another image is superimposed as a part of the image in which the inappropriate motion is reflected, and the other conference participants do not touch it. The other image to be superimposed may be a monochrome image such as white or black, or a mosaic image.

  In the present invention, the image recognition means and the detection of an inappropriate operation are performed by the second information processing apparatus that receives the image from the first information processing apparatus. As a result, the processing load on each first information processing apparatus can be reduced, and the first information processing apparatus that does not have a specific function such as image recognition or motion detection is used, and other conference participants are not made uncomfortable. A conference system can be realized.

  In the present invention, when an inappropriate operation is detected, by controlling not to image a conference participant who performs the inappropriate operation, or to not collect sound from the conference participant, It can be avoided that an image or sound that makes other conference participants feel uncomfortable is transmitted from the second information processing apparatus to the other first information processing apparatus.

  According to the present invention, an inappropriate operation by a conference participant operating the first information processing apparatus is detected from a recognition result by image recognition, and the conference participant for realizing the conference system is imaged according to the operation state. Transmission / reception of the collected image obtained by collecting the captured image or the speech of the conference participant is controlled.

  Thus, for example, the operation state of a conference participant who operates a specific first information processing apparatus is “telephone, snooze, chat, quarrel, cry, laugh, leave, look away, smoking, meal, tongue out” In some cases, processing such as prohibiting (stopping) transmission or reception of images or audio, or both, or superimposing other images is performed to suppress discomfort and discomfort that may be given to other conference participants. A comfortable conference system can be realized.

  Also, by controlling the stop of transmission / reception of captured images or collected sound that may cause discomfort or discomfort to other conference participants, the communication load increases due to images or sounds, and the images or sounds in the conference system. It is possible to suppress the processing load of the server device (second information processing device) that relays the transmission / reception of data.

1 is a configuration diagram illustrating a configuration of a conference system according to a first embodiment. 3 is a block diagram showing an internal configuration of a terminal device that constitutes the conference system of Embodiment 1. FIG. 2 is a block diagram showing an internal configuration of a conference server apparatus that constitutes the conference system of Embodiment 1. FIG. 3 is a schematic diagram schematically showing transmission and reception of images and sounds realized by the conference system of Embodiment 1. FIG. It is explanatory drawing which shows the example of the content of the table for image recognition memorize | stored in the memory | storage part of a conference server apparatus. It is explanatory drawing which shows the example of the content of an operation | movement detection table. 4 is a flowchart illustrating an example of image and audio transmission / reception processing and inappropriate operation detection processing procedure in the conference server device according to the first embodiment. It is explanatory drawing which shows the example of the transmission control performed when an inappropriate operation | movement is detected in the conference system in this Embodiment 1. FIG. 6 is a block diagram illustrating an internal configuration of a terminal device that constitutes the conference system according to Embodiment 2. 10 is a flowchart illustrating an example of image and audio transmission / reception processing and inappropriate operation detection processing procedure in the conference server device according to the second embodiment.

  Hereinafter, the present invention will be specifically described with reference to the drawings showing embodiments thereof.

(Embodiment 1)
FIG. 1 is a configuration diagram illustrating a configuration of the conference system according to the first embodiment. The conference system uses the terminal devices 1, 1,... Used by the conference participants, the network 2 to which the terminal devices 1, 1,... Are connected, and the images (videos) and audio between the terminal devices 1, 1,. And the conference server apparatus 3 that realizes transmission / reception and sharing of the system.

  The network 2 may be an in-house LAN of an organization in which a conference is held, or a public communication network such as the Internet. The network 2 includes a plurality of access points 21, and the terminal devices 1, 1,... Can communicate with the server device 1 by wireless communication.

  In the conference system configured as described above, the terminal devices 1, 1,... Receive connection authentication with the conference server device 3, and the authenticated terminal devices 1, 1,. (Video) and audio information are transmitted and received, and the received image (video) and audio are output to share the image (video) and audio with other terminal devices 1, 1,. Realize.

  Note that the conference server device 3 can realize a plurality of different conferences 1 and 2 in parallel. The conference server device 3 recognizes the terminal devices 1, 1,... In association with the group conference 1 and the conference 2, respectively, and relays images (video) and audio between the terminal devices 1, 1,. Each can be done independently.

  FIG. 2 is a block diagram showing an internal configuration of the terminal device 1 constituting the conference system of the first embodiment.

  The terminal device 1 includes a control unit 100, a temporary storage unit 101, a storage unit 102, an input processing unit 103, a display processing unit 104, a video processing unit 105, an input audio processing unit 106, and an output audio processing unit. 107, a communication processing unit 108, and an encoding / decoding processing unit 109. The terminal device 1 further includes a tablet 13, a display 14, a camera 15, a microphone (in the drawing and hereinafter referred to as a microphone) 16, a speaker 17, and a wireless communication unit 18 by built-in or external connection.

  The control unit 100 uses an arithmetic processing unit such as a CPU or MPU, reads the conference terminal program stored in the storage unit 102 into the temporary storage unit 101, and executes the conference terminal dedicated terminal in the present invention. The information processing apparatus is operated.

  The temporary storage unit 101 uses a RAM such as SRAM or DRAM. The temporary storage unit 101 stores the program read as described above, and stores information generated by the processing of the control unit 100.

  The storage unit 102 uses a nonvolatile memory such as an EEPROM (Electrically Erasable Programmable ROM) or a flash memory. In the storage unit 102, a program and data for realizing the function of the terminal device 1, such as a conference terminal program and conditions that the control unit 100 refers to at the time of control, for example, conditions such as a threshold for a detected value described later, are stored in advance. It is remembered. In addition, other application software programs in the terminal device 1 may be stored. The storage unit 102 may be an external device such as a hard disk or SSD.

  Connected to the input processing unit 103 is a tablet 13 that is built in the display 14 and receives an operation for character input or graphic input by the terminal pen 4. The input processing unit 103 receives information such as pressing information of a button (click button) input by the operation of the conference participant of the terminal device 1 and coordinate information indicating a position in the screen being displayed on the display. The presence / absence and contents of the input operation are determined and notified to the control unit 100. Note that a pointing device (input device) such as a mouse or a keyboard (not shown) may be connected to the input processing unit 103, and a signal corresponding to an operation accepted by the pointing device may be input.

  A touch panel type display 14 using a liquid crystal panel, an organic EL, or the like is connected to the display processing unit 104. The control unit 100 outputs an application screen for the conference terminal to the display 14 via the display processing unit 104, and displays an image (video) to be shared in the application screen. The images to be shared include images transmitted from the other terminal devices 1, 1,... Received from the conference server device 3 as described later. An image transmitted from the conference server apparatus 3 is H.264. 261, H.H. 263, H.M. In the case of encoding according to a standard such as H.264 or MPEG, the control unit 100 decodes the image and then outputs it to the display processing unit 104.

  A video card is used for the video processing unit 105. The video processing unit 105 is connected to the camera 15 included in the terminal device 1, controls the operation of the camera 15, and acquires video data captured by the camera 115. The camera 15 is mounted above the display 14 provided in the casing of the terminal device 1 in a direction in which the user's face or upper body is imaged. The camera 15 captures images at a frequency of several tens or hundreds of times per second, and outputs the image signals to the video processing unit 105 as video data continuously. The video processing unit 105 converts the video data acquired from the camera 15 into H.264. 261, H.H. 263, H.M. Processing to convert (encode) data into video standards such as H.264 and MPEG may be performed.

  The input voice processing unit 106 is connected to the microphone 16 included in the terminal device 1, and has an A / D conversion function that samples the voice collected by the microphone 16, converts it into digital voice data, and outputs the digital voice data to the control unit 100. . The input sound processing unit 106 may incorporate a mixer that performs processing such as signal level adjustment and band limitation of collected sound, and an echo canceller that removes an echo portion. Note that the input voice processing unit 106 converts the collected voice to G.G. 711, G.G. 722, G.G. 728, G.G. 720 or MPEGAudio or other standard audio data may be encoded.

  The output audio processing unit 107 is connected to the speaker 17 included in the terminal device 1. The output sound processing unit 107 has a D / A conversion function so that when sound data is given from the control unit 100, the sound is output from the speaker 17 as sound. Note that the audio data transmitted from the conference server apparatus 3 is G.P. 711, G.G. 722, G.G. 728, G.G. When the data is encoded according to a standard such as 729 or MPEGAudio, the control unit 100 decodes the audio data and then outputs the audio data to the speaker 17.

  The communication processing unit 108 realizes communication via the network 2 of the terminal device 1. The communication processing unit 108 is connected to the wireless communication unit 18 and realizes wireless communication with the conference server device 3 or another terminal device 1 via the access point 21. Specifically, the communication processing unit 108 packetizes information to be transmitted and received, reads information from the packet, and the like. The control unit 100 can transmit and receive image (video) and audio data by the communication processing unit 108. The communication protocol corresponds to the communication protocol in the communication processing unit 37 of the conference server device 3 to be described later.

  The encoding / decoding processing unit 109 uses an encoder / decoder chip. 261, H.H. 263, H.M. H.264 or MPEG based video / image encoding / decoding processing; 711, G.G. 722, G.G. 728, G.G. Audio encoding / decoding processing based on a standard such as H.729 or MPEGAudio is realized. When the encoded video, audio, or multiplexed video data is received from the conference server device 3, the control unit 100 provides the encoded / decoded processing unit 109 for decoding. Note that video (image) and audio standards may be other than the above examples.

  In the first embodiment, the terminal device 1 uses a dedicated terminal equipped with a touch panel display 14. However, the present invention is not limited to this, and a camera and speakers may be connected to a personal computer to which a desktop display is separately connected. Furthermore, it can be realized by connecting a camera, a speaker, and a network card to a general-purpose display, and connecting a device that realizes the function of the control unit 100 as described later. The conference system may include terminal devices 1 having different configurations.

  FIG. 3 is a block diagram showing an internal configuration of the conference server apparatus 3 constituting the conference system of the first embodiment.

  The conference server device 3 uses a server computer, and includes a control unit 30, a temporary storage unit 31, a storage unit 32, an encoding / decoding processing unit 33, an image processing unit 34, an audio processing unit 35, and image recognition. Unit 36, communication processing unit 37, and network I / F unit 38.

  The control unit 30 uses an arithmetic processing unit such as a CPU (Central Processing Unit) or MPU (Micro Processing Unit), and reads the conference server program 3P stored in the storage unit 32 into the temporary storage unit 31 and executes it. Thus, the server computer is operated as the conference server device 3 in the first embodiment.

  The temporary storage unit 31 uses a RAM such as an SRAM (Static Random Access Memory) or a DRAM (Dynamic Random Access Memory) to temporarily read the conference server program 3P read as described above, and at the same time the control unit 30 Information generated by this processing is temporarily stored.

  The storage unit 32 uses an external storage device such as a hard disk or an SSD (Solid State Drive). The storage unit 32 stores the conference server program 3P described above. Further, the storage unit 32 stores authentication data for authenticating the terminal devices 1, 1,... Used by the conference participants. The storage unit 32 of the conference server device 3 stores a conference information DB 321 including document data to be shared by the terminal devices 1, 1,. The document data is text data, photo data, figure data, etc., and the format is not limited. Further, the storage unit 32 stores an image recognition table 322 and an operation detection table 323, which will be described in detail later, and can be referenced from the control unit 30 and other components.

  The encoding / decoding processing unit 33 uses an encoder / decoder chip. 261, H.H. 263, H.M. An image encoding unit that encodes an image (video) based on a standard such as H.264 or MPEG (Moving Picture Experts Group) and a code processing unit that decodes the encoded image are included. When the images transmitted from the terminal devices 1, 1,... Are encoded, the control unit 30 decodes them at the encoding / decoding processing unit 33 and gives them to the image processing unit 34. The image to be synthesized is applied to the encoding / decoding processing unit 33 and encoded, and then transmitted to the terminal devices 1, 1,. In addition, the encoding / decoding processing unit 33 includes G. 711, G.G. 722, G.G. 728, G.G. 729, an audio encoding unit that encodes audio based on a standard such as MPEG Audio, and an audio decoding processing unit that decodes the encoded audio. When audio transmitted from the terminal devices 1, 1,... Is encoded, the control unit 30 decodes the encoded / decoded processing unit 33 to the audio processing unit 35. .. Is supplied to the encoding / decoding processing unit 33 and encoded, and then transmitted to the terminal devices 1, 1,... By the communication processing unit 37. Note that the encoding / decoding processing unit 33 may not only encode the image and the sound, but also perform a process of time-synchronizing and multiplexing the image and the sound to output them.

  The image processing unit 34 realizes a process of combining images based on a plurality of image data respectively transmitted from the plurality of terminal devices 1, 1,. In addition, the image processing unit 34 receives document data to be displayed on each terminal device 1, 1,... Among the shared documents included in the conference information DB of the storage unit 32, and converts the document data into an image. Output function. The image processing unit 34 can perform various types of image processing such as image enlargement / reduction, edge enhancement, or color adjustment.

  The voice processing unit 35 realizes a process of synthesizing voice based on a plurality of voice data respectively transmitted from the plurality of terminal devices 1, 1,. In addition, the audio processing unit 35 can perform various audio processes such as noise removal or volume adjustment.

  The image recognition unit 36 performs processing for recognizing a person's face (contour), hand / arm from the image given by the control unit 30, and processing for recognizing a mouth, eyes, eyebrows and the like in the face. The image recognition unit 36 recognizes a specific object (for example, tableware, cigarette, telephone) and outputs a recognition result. Specifically, information such as the presence or absence of recognized contours, mouths and eyes, and coordinates is output. The image recognizing unit 36 recognizes a change in position of the face, hand / arm, mouth, eyes, eyebrows, and other specific objects recognized in a plurality of images given in time series from the control unit 30. . The image recognition unit 36 may have a function of classifying a person's facial expression according to the position of the mouth, eyes, and eyebrows of the person's face in the image and outputting the classification result as a smile, an angry face, a crying face, or the like.

  The communication processing unit 37 realizes communication via the network 2 of the conference server device 3. The communication processing unit 37 is connected to a network I / F unit 38 using a network card connected to the network 2, and transmits / receives images or sound to / from the terminal devices 1, 1,. Packetization when reading, and reading information from the packet. The control unit 30 can transmit and receive images (video) and audio by the communication processing unit 37. In order to realize the conference system of the first embodiment, the communication protocol for transmitting and receiving images and sounds by the communication processing unit 37 is H.264. A protocol such as H.323, SIP (Session Initiation Protocol), or HTTP (Hypertext Transfer Protocol) may be used. The communication protocol is not limited to these. The network I / F unit 38 may include an antenna, and the communication processing unit 37 may be configured to perform wireless communication.

  FIG. 4 is a schematic diagram schematically showing transmission and reception of images and sounds realized in the conference system of the first embodiment. In the conference system configured as described above, basically, as shown in FIG. 4, the terminal devices 1, 1,... The video (or still image) data of the face or upper body is continuously transmitted to the conference server device 3 via the communication processing unit 108 and the wireless communication unit 18.

  When the conference server device 3 receives image data and sound data transmitted after being picked up and collected by each of the terminal devices 1, 1,..., The image data and the sound data are separated and encoded / decoded. The data is given to the processing unit 33 and decoded. As shown in FIG. 4, the control unit 30 performs image processing so that the image processing unit 34 synthesizes images (videos) from the plurality of terminal devices 1, 1,. To the unit 34. The control unit 30 instructs the audio processing unit 35 to superimpose audio from the plurality of terminal devices 1, 1,... After decoding. The control unit 30 gives the synthesized image and sound to the encoding / decoding processing unit 33, encodes or multiplexes them, and transmits them to the terminal devices 1, 1,... Via the communication processing unit 37.

  In each of the terminal devices 1, 1,..., The synthesized image and audio transmitted from the conference server device 3 are decoded by the encoding / decoding processing unit 109 and given to the display processing unit 104 and the output audio processing unit 107, respectively. , Images showing the face or upper body of the conference participants who use the terminal devices 1, 1,... Are displayed side by side. Thereby, the conference participant using each terminal device 1, 1,... Can listen to the speech of the conference participant including himself / herself while confirming the facial expressions and gestures of other conference participants.

  However, in the conference system in the first embodiment, when the operation of the conference participant shown in the image transmitted from any one of the terminal devices 1 is inappropriate, the face of the conference participant from the terminal device 1 or Transmission of the image obtained by capturing the upper body image and the sound emitted from the conference participant to the terminal device 1 is prohibited.

  Detection of an inappropriate operation by a conference participant is performed by the conference server device 3 that has received an image from the terminal device 1 used by the conference participant. Hereinafter, details of the inappropriate operation detection process in the conference server apparatus 3 will be described.

  In order for the conference server device 3 to detect an inappropriate operation of the conference participant, the storage unit 32 stores an image recognition table 322 and an operation detection table 323.

  FIG. 5 is an explanatory diagram showing an example of the contents of the image recognition table 322 stored in the storage unit 32 of the conference server device 3. The image recognition table 322 includes a number assigned to each inappropriate operation content, each operation content, and a recognition pattern.

In the example shown in FIG. 5, as an inappropriate operation, operations such as “telephone, snooze, chat, quarrel, cry, laugh, leave, look away, smoke, eat, put out tongue” are listed. For example, it is considered that “telephone” as an inappropriate operation during a meeting can be determined by an operation in which the hand / arm is brought close to the ear and the mouth is moving. Accordingly, as a recognition pattern in the image recognition of “1: telephone”, the position of the arm with respect to the body (arm up / down), opening / closing of the mouth over a plurality of images, or an image recognized as a telephone set. Presence is registered in advance.
“Dozing” may be a situation in which both eyes are closed and tilted. Therefore, with regard to “2: dozing”, the direction of the face (downward, inclination of the face center line more than a predetermined angle), whether both eyes are closed, or there is a periodic movement of the head (face). Registered in advance.
The inappropriate action “crying” can be discriminated from the image based on whether the face is facing down, the hand is holding the eyes, or the expression of the crying face. Therefore, “3: cry” is registered in advance such that the face is facing downward, the position of the hand / arm with respect to the body is in a position covering the face, or the crying face is recognized.
Inappropriate behavior such as “Laughter” can be a situation where the mouth is wide open and laughing. Therefore, “4: Laughter” is registered in advance as to whether the size of the mouth to be recognized is a predetermined size or more, whether the ratio to the face is a predetermined value or more, and whether a smile is recognized.
Since “seating” is a situation where no person is recognized in the first place, a pattern indicating that a face (person) cannot be recognized for “5: leaving” is registered in advance.
The operation of “look away” is a state in which the user is facing another direction without looking at the screen. Therefore, “6: Looking Away” is registered in advance that the face is in a direction other than the predetermined direction, or is largely inclined from the predetermined direction.
The operation of “smoking” can be discriminated based on whether the hand is holding the cigarette, holding the cigarette in the mouth, or repeating the operation of holding the cigarette in the mouth. Therefore, for “7: smoking”, the recognition of the cigarette image, the presence of the hand in the mouth, the repeated movement of the hand to the mouth, etc. are registered in advance as a pattern.
The operation of “meal” can be discriminated according to the operation of bringing tableware such as chopsticks or forks to the mouth by hand, or moving while the mouth is closed. Also, if chewing gum is included in an inappropriate operation of eating, in this case as well, it can be discriminated by the situation where the mouth is moving and the like. Therefore, for “8: Meals”, patterns such as recognition of tableware images, movement of hands / arms to the mouth, or movement of the mouth with the mouth closed are registered in advance. is there.
Since the action of “sticking out the tongue” is a state in which the tongue is sticking out from the mouth, a pattern such as whether the tongue image is recognized in the vicinity of the mouth is registered in advance for “9: sticking out the tongue”. is there.
Since “fight” is a situation where multiple people are engaged, for “10: fight”, it can be determined that a plurality of people are recognized and that a plurality of people are intertwined, or a facial expression of a person. A pattern such as recognition of an angry face is registered in advance.
The “chat” is a situation in which a plurality of people are talking differently from the agenda of the meeting. For example, a situation where two adjacent people face each other and their mouths move is considered. Therefore, for “11: chat”, a pattern in which a plurality of persons are recognized and the faces of the plurality of persons face each other is registered in advance.

  The control unit 30 of the conference server device 3 gives an image transmitted from one terminal device 1 to the image recognition unit 36, and obtains recognition results of a person's face, hands / arms, face, and the like. Inappropriate operations include those that cannot be recognized by only one image, and the control unit 30 provides a plurality of images for a predetermined time such as 0.3 seconds to obtain a recognition result. The control unit 30 compares the obtained recognition result with the recognition patterns of each inappropriate operation stored in the image recognition table 322, and determines the presence or absence of each inappropriate operation. In this way, the control unit 30 can detect an inappropriate operation based on the determination of the presence or absence of each inappropriate operation. When a new inappropriate operation is added or deleted, the operation content in the image recognition table 322 is newly added or deleted, or the determination for each operation is valid (permitted) / invalid (not valid). (Permission) should be set. By modifying the contents of the image recognition table 322, it becomes easy to appropriately change details of detection of inappropriate operations. Of course, even in the configuration without the image recognition table 322, the details of the contents of the inappropriate operation to be detected can be changed as appropriate by changing the conference server program 3P.

  FIG. 6 is an explanatory diagram showing an example of the contents of the motion detection table 323. The motion detection table 323 is information that holds the result of the control unit 30 determining for each inappropriate motion included in the image recognition table 322 for each terminal 1. The operation detection table 323 may be stored in the temporary storage unit 31 instead of the storage unit 32. The motion detection table 323 includes an identification number for identifying a terminal device that is a transmission source of an image, an inappropriate motion number, a determination result, and a duration. The information for identifying the terminal device is not limited to the number. The inappropriate operation number corresponds to the number assigned to the operation content registered in advance in the image recognition table 322. The duration in FIG. 6 is the number of times the determination result “1: Present” continues. It may be the actually measured number of seconds.

  FIG. 6 shows the determination result of whether or not the control unit 30 of the conference server apparatus 3 detects an inappropriate operation based on the image from the terminal apparatus 1 with the identification number “001”. Yes. Based on the image obtained from the terminal device 1 to which the identification number “001” is assigned, the control unit 30 determines whether or not the inappropriate motions having the numbers 1 to 11 are detected based on the motion detection table 323. I remember it. In the example of FIG. 6, an inappropriate operation “Laughter” of number 4 is detected based on the image obtained from the terminal device 1 assigned the identification number “001”, and the duration is “0005”. Yes.

  The control unit 30 of the conference server device 3 performs motion detection on the images from the terminal devices 1, 1,... That are periodically transmitted every time an image is received or every time it is received a plurality of times. The control unit 30 performs motion detection for each inappropriate motion, and when the previous determination result is “0: None” and the current determination result is “1: Yes”, the determination result “1” is operated. The data is stored in the detection table 323 and the timer value is stored as “0001”. When the previous determination result is “1: Yes” and the current determination result is also “1: Yes”, the control unit 30 stores the determination result “1” and sets the timer value to 1 or the number of seconds. to add. When the previous determination result is “1: Yes” and the current determination result is “0: None”, the control unit 30 stores the determination result “0” and clears the timer value to “0”. .

  In this way, in the motion detection table 323, detection results of inappropriate motion based on images from the terminal devices 1, 1,. The operation detection table 323 is stored and updated only when an inappropriate operation is detected from one terminal device 1.

  In the conference system configured as described above, images and sound are transmitted from the terminal devices 1, 1,..., Received by the conference server device 3, and shared images and A process of detecting an inappropriate operation performed in the process of transmitting as voice will be described below with reference to a flowchart.

  FIG. 7 is a flowchart illustrating an example of image and audio transmission / reception processing and inappropriate operation detection processing procedure in the conference server device 3 according to the first embodiment. In addition, the process shown below is a process with respect to the one terminal device 1. FIG. The following processing may be performed independently and in parallel for a plurality of terminal devices constituting the conference, or processing may be performed for all the terminal devices in the following steps.

  When the control unit 30 of the conference server device 3 authenticates the conference application from the terminal device 1 and starts the conference, the control unit 30 first starts a timer to measure the duration of inappropriate operation (step S1). ). The control unit 30 receives image and audio data from the terminal device 1 (step S2). At this time, when the image and the sound are encoded, the control unit 30 supplies the image to the encoding / decoding processing unit 33 for decoding, and then performs the following processing.

  The control unit 30 gives the image received in step S2 to the image recognition unit 36 and performs recognition processing (step S3). The control unit 30 compares the recognition result obtained from the image recognition unit 36 with the image recognition table 322 to determine whether or not there is an inappropriate operation (step S4).

  When the control unit 30 detects an inappropriate operation, that is, an inappropriate operation (S4: YES), the control unit 30 determines the number of the inappropriate operation determined as having an operation and the previous determination result based on the operation detection table 323. It is determined whether or not one or more inappropriate motion numbers for which “1: is present” match (step S5).

  When it is determined that one or more matches (S5: YES), the control unit 30 adds one or the elapsed time of the matching inappropriate operation, and newly adds an inappropriate operation that does not match. The motion detection table 323 is updated so that the determination result is “1: present” (step S6). Then, the control unit 30 determines whether or not the duration of the inappropriate operation with the matching number has exceeded a predetermined time (step S7). Specifically, it is determined whether or not the timer value for an inappropriate operation with a matching number is one or more larger than a predetermined number. When the control unit 30 determines that the predetermined time has elapsed (S7: YES), the control unit 30 performs control to restrict transmission of the image from the terminal device 1 that is the transmission source of the image to the other terminal device 1 (step S1). S8). At this time, the time measurement by the timer is continued.

  Specifically, step S8 performs the following processing. As one method, the control unit 30 does not provide the image processing unit 34 with the image received from the terminal device 1 on which an inappropriate operation has been performed, and stops transmission to the other terminal devices 1, 1,. As another method, the control unit 30 superimposes a white or black image on a corresponding position in the image where an inappropriate operation is detected, or superimposes a mosaic image and then supplies the image to the image processing unit 34. Then, it is combined with an image from another terminal device 1. As another method, the control unit 30 may perform an inappropriate operation and stop receiving an image from the terminal device 1. As another method, the control unit 30 may also stop receiving voice from the terminal device 1 that has performed an inappropriate operation. In addition, the control unit 30 may transmit frames from the terminal device 1 in which an inappropriate operation has been performed to other terminal devices 1, 1,..., Dropping frames, that is, reducing the rate (quality). Good.

  Further, the detailed processing of the control unit 30 in step S8 may be changed according to the content of the detected inappropriate operation. For example, the control unit 30 detects inappropriate operations such as “telephone”, “sleeping”, “chatting”, “quarreling”, “crying”, “laughing”, “seating”, “looking away”, “smoking”, “meal”, and “sticking out tongue”. In such a case, transmission of the “image” from the terminal device 1 in which an inappropriate operation has been detected to the other terminal devices 1, 1,... Superimpose or reduce the image transmission rate. When an inappropriate movement related to the movement of the mouth such as “smoking”, “meal”, and “sticking out the tongue” is detected, the control unit 30 superimposes a white-painted image (or another image) on the mouth. Also good. When an operation with an inappropriate voice or utterance such as “telephone”, “chat”, “fight”, “cry”, “laughter” is detected, the control unit 30 detects the inappropriate operation. Stops transmitting “speech” to other terminal devices 1, 1,.

  Note that the specific content of the transmission control that combines the stop of image transmission and the stop of audio transmission may be stored in advance in the storage unit 32 for each “inappropriate operation”. The content of the transmission control is set and stored in association with the number of each inappropriate operation in the image recognition table 322. For example, “1: telephone / call” is associated with “image: stop / voice: stop”, “2: doze” is associated with “image: stop / voice: transmission continued”, or “10: fight”. “Image: white overlay / sound: stop” or the like is stored. Further, when a plurality of persons are shown, it is stored that a white-painted image is superimposed on a place where an inappropriate operation is detected. By storing specific settings for transmission control in the storage unit 32 as described above, it is possible to change the settings appropriately in such a manner that the setting contents in the storage unit 32 are changed.

  Next, the control unit 30 determines whether or not the conference in the conference system has ended (step S9). When it is determined that the control unit 30 has not ended (S9: NO), the process returns to step S2 and continues. When it is determined that the control unit 30 has ended (S9: YES), the process ends.

  If no inappropriate operation is detected in step S4, that is, no inappropriate operation is detected (S4: NO), the control unit 30 initializes all determination results in the operation detection table 323 as “0: no”. (Step S10), the process proceeds to Step S9.

  The control unit 30 determines that the number of the inappropriate motion determined to have motion does not match the number of the inappropriate motion whose previous determination result is “1: yes” in the motion detection table 323. In the case (S5: NO), the operation detection table 323 is updated (step S11), and the process is terminated. At this time, for an inappropriate operation in which the previous determination result is “1: yes”, the duration is cleared to “0” and the determination result is stored as “0: none”. An inappropriate operation in which the current determination result is “1: yes” is saved as “1: yes” and the duration is saved.

  When it is determined that the predetermined time has not elapsed in step S7 (S7: NO), control unit 30 advances the process to step S9.

  In step S7, whether or not the predetermined time has elapsed is determined uniformly for all inappropriate operations. However, for a predetermined time that differs depending on the contents of the detected inappropriate operations. You may make it judge. As described above, the predetermined time that differs depending on the content of the inappropriate operation may be set in association with the number of each inappropriate operation in the image recognition table 322 and stored in the storage unit 32.

  FIG. 8 is an explanatory diagram illustrating an example of transmission control performed when an inappropriate operation is detected in the conference system according to the first embodiment. FIG. 8 shows an image that can be confirmed on the display 14 of the terminal device 1 different from the terminal device 1 in which an inappropriate operation is detected.

  In the example shown in FIG. 8, the upper part shows the contents of the image before the transmission restriction control is performed. Three persons are recognized from the upper image. The control unit 30 gives the image to the image processing unit 34 to obtain information such as the face (contour), mouth, eyes, hand / arm positions, etc. for the three persons, and compares the information with the recognition patterns in the image recognition table 322. To do. In the upper image of FIG. 8, the orientation of the face of the person on the left side of the three persons is different from the imaging direction of the camera 15 with the imaging direction from the front to the back in the figure. It can be judged. The control unit 30 determines that the direction of the face should be “front” and within a predetermined inclination, and because the direction of the face of the person shown on the leftmost side is greater than or equal to the predetermined inclination, “look away” and the recognized hand / arm Since the position of is at the height of the ear and the recognized mouth is opened and closed, an inappropriate operation such as “telephone” is detected. In this case, the control unit 30 superimposes a white-painted image on a part based on the position in the image where an inappropriate operation is detected so that these operations are not reflected. As a result, as shown in the lower part of FIG. 8, an image in which a person who is looking away is concealed can be confirmed. When an image showing a plurality of persons is transmitted from the terminal device 1, processing such as superimposing a part of the image with another image is performed in this way. This does not hinder the participation of other conference participants appearing in the same image.

  When the control unit 30 of the conference server device 3 stops the transmission of the image or the sound from the terminal device 1 in which an inappropriate operation is detected in step S8, the control unit 30 sends a message indicating that the transmission is stopped to the terminal device 1 May be sent to. At this time, the control unit 30 gives an instruction to output a warning display or an alarm sound to the terminal device 1 in which the inappropriate operation is detected before the predetermined time elapses, and the conference is performing an inappropriate operation. The participant may be alerted. Thereby, progress of a meeting can be made smooth. Moreover, the control part 30 may transmit the message to the effect of stopping the transmission of the image from the terminal device in which inappropriate operation | movement was detected to other terminal device 1,1, .... Thereby, the conference participant who uses the other terminal device 1 can grasp the situation and continue the progress of the conference system.

  With the above configuration, the operation state of a conference participant who uses a specific terminal device 1 is, for example, inappropriate (telephone, snooze, chat, quarrel, cry, laugh, leave, look away, smoking, meal, tongue out Etc.), it is avoided that an image showing the motion or a sound accompanying the motion is transmitted to other conference participants. An image showing an inappropriate operation is prohibited from being transmitted to other terminal devices 1, 1,..., Or even if transmitted to other terminal devices 1, 1,. It will fall or a part of the screen will be processed. As a result, it is not necessary to view an image showing an inappropriate operation on the other terminal devices 1, 1,..., Or it becomes impossible or it is difficult to confirm clearly, Or part of it is hidden and cannot be seen. As a result, it is possible to suppress the discomfort and discomfort of other conference participants and realize a comfortable conference system. Further, by restricting transmission of an image obtained by capturing an inappropriate operation and collected voice data, unnecessary data can be prevented from being transmitted and received, and the communication load on the network 2 can be reduced. be able to.

  In the first embodiment, an inappropriate operation is detected by “image recognition”. That is, an inappropriate operation involving voice or utterance is detected by image recognition based on the opening / closing operation of the mouth, etc., regardless of voice recognition. Inappropriate actions involving voice or speech are, for example, actions such as telephone calls, chats, fights, crying, or laughing. In these cases, movements of the mouth before actually making inappropriate comments, or Or there is a possibility that it can be detected by image recognition combined with arm movements. Therefore, improper motion can be detected before improper voice or speech is actually emitted by image recognition.

  In the first embodiment, the control unit 30 is configured to perform control so as not to transmit to other terminal devices 1, 1,... When an inappropriate operation is detected continuously for a predetermined time or longer. As a result, when an inappropriate operation is detected only once, transmission is immediately stopped, and the progress of the conference is made smoother than when the image is suddenly cut off. In addition, since it is possible to give a warning to a person who has performed an inappropriate operation before the predetermined time elapses, it is possible to reduce the possibility of disturbing a smooth conference by unnecessarily restricting participation in the conference. However, the present invention is not limited to this, and it is possible to omit the determination (S7) as to whether or not the continuation time is equal to or longer than a predetermined time, and to perform control to immediately limit transmission. The configuration can be changed as appropriate as long as the conference manager can set whether or not the predetermined time is zero.

(Embodiment 2)
In the second embodiment, transmission of an image or sound that causes other conference participants to feel uncomfortable by controlling to change the imaging direction of the camera or the sound collection direction of the microphone in the terminal device in which an inappropriate operation is detected. To avoid.

  The conference system in the second embodiment is an image between the terminal devices 5, 5,... Used by the conference participants and the network 2 to which the terminal devices 5, 5,. (Conference server device 3) that realizes transmission / reception and sharing of (video) and audio. That is, the configuration including the terminal device 5 is different from that of the first embodiment, and other configurations are the same as the configuration of the first embodiment. Therefore, in the following description, the same reference numerals are assigned to the devices and internal configurations that are the same as those in the configuration of the first embodiment, and detailed description thereof is omitted.

  As with the terminal device 1 in the first embodiment, the terminal device 5 uses a conference system dedicated terminal equipped with a tablet built-in display and has the same appearance.

  FIG. 9 is a block diagram showing an internal configuration of the terminal device 5 constituting the conference system of the second embodiment.

  The terminal device 5 includes a control unit 500, a temporary storage unit 501, a storage unit 502, an input processing unit 503, a display processing unit 504, a video processing unit 505, an input audio processing unit 506, and an output audio processing unit. 507, a communication processing unit 508, and an encoding / decoding processing unit 509. The terminal device 5 further includes a drive unit 59 in addition to the tablet 53, the display 54, the camera 55, the microphone 56, the speaker 57, and the wireless communication unit 58 by built-in or external connection.

  Of the components included in the terminal device 5, the configuration other than the configuration of the drive unit 59 is the same as the components in the first embodiment. Therefore, detailed description thereof will be omitted.

  The drive unit 59 can change the imaging direction of the camera 55 by a control signal from the control unit 500 based on an instruction from the conference server device 3. The camera 55 in the second embodiment is supported so that it can be moved inside the casing of the terminal device 5. The drive unit 59 is disposed so as to be in contact with the support unit of the camera 55 inside the housing, and includes a mechanism such as a stepping motor to change the direction of the support unit of the camera 55 in accordance with a control signal from the control unit 500 and take an image. Change direction.

  The control unit 500 reads out and executes the conference terminal program stored in the storage unit 502, thereby imaging the conference participants who use the terminal device 5 at the start of the conference. And the control part 500 changes the imaging direction of the camera 55 so that a conference participant may not be reflected, when the instruction | indication which changes an imaging direction from the conference server apparatus 3 is received.

  FIG. 10 is a flowchart illustrating an example of an image and audio transmission / reception process and an inappropriate operation detection process procedure in the conference server apparatus according to the second embodiment. Of the processing procedures shown below, the same steps as those shown in FIG. 7 of the first embodiment are denoted by the same step numbers, and detailed description thereof is omitted.

  When it is determined that the predetermined time has elapsed (S7: YES), the control unit 30 transmits an instruction to change the imaging direction of the camera 55 to the terminal device 1 that is the transmission source of the image (step S20). The process proceeds to S9. That is, step S20 is performed as an alternative to step S8 in the first embodiment.

  Thereby, the control unit 500 of the terminal device 5 moves the imaging direction of the camera 55 to the driving unit 59 so as not to display an image that is not suitable for the meeting place by imaging the conference participants who use the terminal device 5. A control signal is given and adjusted to change the sound collection direction.

  With such a configuration, the terminal device 5 in which an inappropriate operation has been detected by the conference participant is prevented from reaching the conference server device 3 with an image and sound that may make other conference participants uncomfortable. And a comfortable conference system can be realized.

  In the second embodiment, the control unit 500 of the terminal device 5 is configured to perform control to change the imaging direction of the camera 55 when an inappropriate operation state of the conference participant is detected. However, the present invention is not limited to this, and the microphone 56 having high directivity is used, and the sound collection direction of the microphone 56 is changed by the input sound processing unit 506 or sound from a specific direction (right, left, etc.) is used. You may perform the process of removing afterwards. Thereby, it is possible to avoid transmission of sounds that make other conference participants uncomfortable to the other terminal devices 1, 1,... Via the conference server device 3.

  Further, the control unit 30 of the conference server device 3 is configured to transmit an instruction to change the imaging direction and the sound collection direction. However, the present invention is not limited to this, and the control unit 30 transmits an instruction to stop imaging and sound collection and prohibit transmission of image and sound data to the terminal device 5 in which an inappropriate operation is detected. Also good. In addition to suppressing discomfort and discomfort, it is possible to prevent image and audio data not transmitted to the other terminal devices 5, 5,. Thereby, it is also possible to suppress an increase in communication load due to images or sounds and a processing load on the conference server device 3 that relays transmission / reception of images or sounds in the conference system. Furthermore, the transmission from the terminal device 1 is controlled, that is, not only the reception from the terminal device 1 at the conference server device 3 is stopped, but the conference server device 3 also starts from the terminal device 5 from which an inappropriate operation is detected. The transmission data amount of the image may be reduced. Specifically, the control unit 30 instructs the terminal device 5 to transmit a part of the image acquired by the video processing unit 505. As a result, the control unit 500 drops the image data to be transmitted, that is, reduces the rate. Alternatively, the control unit 30 may cause the terminal device 1 to reduce the imaging rate by the camera 55 from the video processing unit 505, or reduce the sampling rate in the input audio processing unit 506 to reduce the amount of audio data. You may make it reduce. Thereby, it is possible to prevent unnecessary data from being transmitted / received by restricting transmission of the image of improper operation and the collected voice data to the conference server device 3, and the network. The increase in communication load of 2 can be reduced.

  In the first and second embodiments, the conference server device 3 includes an image recognition unit 36 that performs image recognition on images received from the terminal devices 1 (5), 1 (5),. . Inappropriate operation patterns registered in advance in the image recognition table 322 can be unified. However, the subject that performs image recognition processing and inappropriate motion detection is not limited to the conference server device 3. In order to reduce the load on the control unit 30 of the conference server device 3, it may be performed by another device, or may be configured by each terminal device 1 (5), 1 (5),. When the processing is performed by the terminal devices 1 (5), 1 (5),..., Each device image recognition process is required, so that the processing load on each terminal device 1 (5), 1 (5),. On the other hand, the terminal device 1 (5) in which an inappropriate operation is detected stops transmission of images and sounds to the conference server device 3, that is, the conference server device 3 receives images and sounds from the terminal device 1. Control such as prohibition of reception becomes possible, and there is an effect that the communication load in the network 2 can be reduced.

  The disclosed embodiments should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

1 Terminal device (first information processing device)
100 control unit 105 video processing unit (acquiring means)
106 Speech processing unit (means for acquiring)
108 Communication processing unit (transmission means, transmission / reception means)
15 Camera (imaging device)
16 Microphone (sound collector)
2 Network 3 Conference server device (second information processing device, information processing device)
30 control unit (recognition means, detection means, transmission control means)
31 Temporary storage unit 32 Storage unit 322 Image recognition table 36 Image recognition unit (recognition means)
37 Communication processing unit

Claims (10)

A plurality of first information processing apparatuses each including an imaging device or a sound collecting device, a means for acquiring an image from the imaging device or a sound from the sound collecting device, and a transmission / reception means for transmitting / receiving the acquired image or sound; Common to a plurality of first information processing devices, including a second information processing device connected to the first information processing device via a communication medium and relaying an image or sound transmitted / received by each first information processing device. In a conference system that displays or outputs the image or sound and shares the information to realize the conference,
Recognizing means for recognizing the action of a person shown in the image acquired by the first information processing apparatus;
Detection means for detecting the presence or absence of inappropriate motion based on the recognition result by the recognition means;
Depending on the detection result of the detection means, reception of an image or sound from the first information processing apparatus or transmission to another first information processing apparatus, increase or decrease in transmission rate, or part of the image A transmission control means for controlling the processing of the conference system. The transmission control means includes
Whether the detection means detects an inappropriate operation, prohibits reception of an image or sound from the first information processing apparatus in which the inappropriate operation is detected, or transmission to another first information processing apparatus The conference system according to claim 1, wherein a transmission rate is reduced or processing of a part of the image is performed. Means for measuring a duration of the operation after the detection unit detects an inappropriate operation;
Determining means for determining whether or not the duration is a predetermined time or more,
The transmission control means receives the image or the sound from the first information processing apparatus in which the inappropriate operation is detected, or receives the image or the sound other than the image or the sound when the determining means determines that the predetermined time is exceeded. The conference system according to claim 1, wherein transmission to the first information processing apparatus is prohibited, a transmission rate is reduced, or a part of the image is processed. It has a table that is a list of image recognition results registered in advance as inappropriate operations,
2. The detection unit according to claim 1, wherein the detection unit is configured to detect presence / absence of an inappropriate operation based on whether or not a recognition result by the recognition unit corresponds to a recognition result included in the table. Conference system. When the detection means detects an inappropriate voice or an action involving utterance, the image or voice is received from the first information processing apparatus in which the inappropriate action is detected, or another image or voice is received. The conference system according to claim 1, wherein transmission to one information processing apparatus is prohibited. The transmission control means performs processing for superimposing another image on a part of the image corresponding to the inappropriate operation in the image from the first information processing apparatus in which the inappropriate operation is detected. The conference system according to claim 1. The conference system according to claim 1, wherein the recognition unit and the detection unit are provided in a second information processing apparatus. And a means for instructing a change in the imaging direction of the imaging device of the first information processing apparatus or the sound collection direction of the sound collector when the inappropriate means detects the inappropriate operation. The conference system according to claim 1, wherein: In an information processing apparatus comprising means for connecting with other devices via a communication medium and transmitting / receiving an image or sound to / from each device,
Means for recognizing the movement of a person in the received image;
A means for detecting the presence or absence of inappropriate movement based on the recognized result;
Depending on the detection result, reception of an image or sound from the other device, transmission or non-transmission of the image or sound to another device, increase or decrease of a transmission rate, or processing of a part of the image An information processing apparatus comprising: means for controlling. A plurality of first information processing apparatuses each including an imaging device or a sound collecting device, a means for acquiring an image from the imaging device or a sound from the sound collecting device, and a transmission / reception means for transmitting / receiving the acquired image or sound; A first information processing apparatus connected to the first information processing apparatus via a communication medium, and a second information processing apparatus that relays an image or sound transmitted / received by each first information processing apparatus. In an information processing method for controlling transmission / reception of an image or sound,
Recognizing the movement of a person in the image acquired by the first information processing apparatus;
Based on the recognized result, detect the presence or absence of inappropriate movement,
Depending on the detected result, reception of an image or sound from the first information processing apparatus or transmission or non-transmission of the image or sound to the first information processing apparatus, increase or decrease in transmission rate, or one of the images An information processing method characterized by controlling machining to a part.

Images