JP2005175627A - System for taking proceedings - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a simple system for taking proceedings available at any place. <P>SOLUTION: The system 2 for taking proceedings comprises a digital camera 10 for picking up the image of a white board 21 on which the content of a meeting is written, outputting digital image data, recording the speech of participants at the meeting and delivering digital voice data, a portable telephone 11 for transmitting the image data and the voice data to the outside, and a server 12 comprising a unit 91 for recognizing characters in the image data and converting the characters into first text data, a unit 92 for recognizing voice in the voice data and converting it into second text data, a unit 93 for editing the first and second text data automatically to make a file of proceedings, and a unit 95 for downloading the file of proceedings to the personal computer 15 of a client through the Internet 14. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a minutes creation system that automatically creates a minutes file.

  Conventionally, minutes of meetings have been created by participants taking notes, and after the meeting, the notes are checked against the contents of the meeting. For this reason, the participants who create the minutes have been inconvenient because they cannot concentrate on taking notes because they are devoted to taking notes. In addition, it involved a complicated task of organizing notes while remembering the contents of the meeting.

  In order to solve the above problems, a camera that captures a video of a participant in a conference, audio processing means for identifying the participant's location information from the participant's remarks, and the participant based on this location information A proposal proceeding support system comprising a control means for creating a node composed of an icon of a video indicating an icon and an icon indicating the content of a speech in units of speech content and controlling the node to be displayed on a monitor of a personal computer has been proposed. (See Patent Document 1).

  According to the agenda progress support system described in Patent Document 1, it is possible to input what and who has spoken to whom with a simple icon-based interface, thus reducing the trouble of creating minutes. can do. It is also possible to grasp the progress of proceedings at a glance. Furthermore, the created minutes can be used as multimedia minutes that can be shared by many participants at remote locations.

International Publication No. 96/27988 Pamphlet

  However, the proceeding support system described in Patent Document 1 includes a large number of devices such as a camera, a camera control device, a video switching device, a microphone, a sound source estimation device, a voice switching device, a voice recognition device, and a main control device. Since it is necessary and the system configuration becomes large, there is a problem that the available places are limited.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a minutes creation system that can be used without choosing a place with a simple system configuration.

  In order to achieve the above object, the minutes creation system of the present invention images the display area of a display device installed in a conference room, outputs digital image data, and records the comments of participants in the conference A digital camera that outputs digital audio data; a communication device that is connected to the digital camera and transmits the image data and audio data to the outside; and characters in the image data transmitted from the communication device A character recognition device that recognizes the character and converts it into first text data, a speech recognition device that recognizes the speech data transmitted from the communication device and converts it into second text data, and the first An editing apparatus that automatically edits the first and second text data to create a minutes file, and distributes the minutes file to customers via a communication network Characterized by being composed of a communication apparatus.

  The character recognition device, the speech recognition device, the editing device, and the distribution device are preferably provided in a server connected to the communication device via a communication network. The digital camera preferably includes voice recognition means for recognizing an operation command inputted by voice from the customer, and control means for controlling each unit to execute processing according to the operation command recognized by voice. . Furthermore, it is preferable to provide an automatic settlement apparatus that automatically withdraws the system usage fee from the bank account of the customer registered in advance after the delivery of the minutes file.

  According to the minutes creation system of the present invention, the display area of the display device installed in the conference room is imaged, digital image data is output, and the speech of the conference participant is recorded, and the digital audio is recorded. A digital camera that outputs data, a communication device that is connected to the digital camera and transmits image data and audio data to the outside, a character in the image data transmitted from the communication device is recognized, and the first text A character recognition device that converts data, a speech recognition device that recognizes speech data transmitted from the communication device and converts the speech data into second text data, and automatically edits the first and second text data, Since it consists of an editing device that creates the minutes file and a distribution device that distributes the minutes file to the customers via the communication network, It can be utilized to not selected. In addition, the minutes covering the contents of the meeting can be obtained immediately after the meeting in the form of a versatile data file.

  In FIG. 1, the minutes creation system 2 of the present invention includes a digital camera 10, a mobile phone 11, and a server 12. The cellular phone 11 is connected to the server 12 via the telephone line network 13. The server 12 is connected to a personal computer (PC) 15 of the customer 20 (see FIG. 2) via the Internet 14.

  The minutes creation system 2 transmits the image data and sound data acquired by the digital camera 10 to the server 12 via the mobile phone 11, and the minutes file is sent by the server 12 based on the transmitted image data and sound data. And the minutes file is distributed to the PC 15 of the customer 20 via the Internet 14.

  FIG. 2 shows a state in which a meeting is progressing using the minutes creation system 2. The customer 20 installs the digital camera 10 in a place where the entire whiteboard 21 can be photographed, connects the mobile phone 11 to the digital camera 10, and connects the PC 15 to the Internet 14. Then, the content of the conference written by the presenter 22 on the whiteboard 21 is photographed by the digital camera 10, and the statements of the customer 20, the presenter 22, and other participants 23 are recorded.

  3 and 4, an imaging lens 30, a viewfinder objective window 31, and a microphone 32 are provided on the front surface of the digital camera 10. A release button 33 is provided on the upper surface, and a memory card slot 34 into which a memory card 56 (see FIG. 5) is detachably loaded and a connector 35 (for example, a USB connector) are provided on the side surface. . Further, a finder eyepiece window 36, a liquid crystal display (LCD) 37, and an operation unit 38 are provided on the back surface.

  The release button 33 is a two-stage push switch. When the release button 33 is lightly pressed (half-pressed) after framing by the viewfinder or the LCD 37, the CPU 43 (see FIG. 5) performs various photographing preparation processes such as automatic exposure adjustment and automatic focus adjustment. In this state, when the release button 33 is pressed again (fully pressed) once, the imaging signal for one screen subjected to the imaging preparation process is converted into image data, and then image processing and compression processing described later are performed. Recorded in the memory card 56.

  In the digital camera 10, the microphone 32 is always used together with a still image shooting mode for shooting still images, a moving image shooting mode for shooting movies, a playback mode for displaying captured images on the LCD 37, a setting mode for setting various settings, and still image shooting. The minutes creation mode for recording audio can be selected.

  The digital camera 10 also has a voice operation function for receiving camera operation commands by voice via the microphone 32. In addition to basic commands such as shooting instructions and mode selection, the operation commands handled by this voice operation function include conference start / end, audio recording start / end, and data transmission used in the minutes creation mode. .

  In FIG. 5 showing the electrical configuration of the digital camera 10, a lens motor 41 and an iris motor 42 are connected to the imaging lens 30 and the diaphragm 40. These motors 41 and 42 are stepping motors, which are controlled in operation by drive pulses transmitted from motor drivers 44 and 45 connected to the CPU 43, and perform photographing preparation processing in response to half-pressing of the release button 33.

  The lens motor 41 performs zooming of the imaging lens 30 by moving the zoom lens of the imaging lens 30 to the wide side or the tele side in conjunction with the operation of the zoom operation button provided on the operation unit 38. Further, the focus lens of the imaging lens 30 is moved in accordance with the subject distance and zoom lens magnification, and the focus of the imaging lens 30 is adjusted so that the shooting conditions are optimized. The iris motor 42 operates the aperture 40 to adjust the exposure of the imaging lens 30.

  Behind the imaging lens 30 is a CCD 46 that photoelectrically converts subject light transmitted through the imaging lens 30 and outputs an imaging signal. A timing generator (not shown) controlled by the CPU 43 is connected to the CCD 46, and the shutter speed of the electronic shutter is determined by a timing signal (clock pulse) input from the timing generator.

  The imaging signal output from the CCD 46 is input to a correlated double sampling circuit (CDS) 47 and output as R, G, and B image data that accurately corresponds to the accumulated charge amount of each cell of the CCD 46. Image data output from the CDS 47 is amplified by an amplifier (AMP) 48 and converted to digital image data by an A / D converter (A / D) 49.

  The image input controller 50 is connected to the CPU 43 via the bus 51, and controls the CCD 46, the CDS 47, the AMP 48, and the A / D 49 in accordance with a control command from the CPU 43. The image data output from the A / D 49 is temporarily stored in the SDRAM 52. This image data is displayed on the LCD 37 via the LCD driver 53.

  The image signal processing circuit 54 reads the image data from the SDRAM 52, performs various image processing such as gradation conversion, white balance correction, and γ correction processing, and stores the image data in the SDRAM 52 again.

  The image data that has been subjected to various processes by the image signal processing circuit 54 is read from the SDRAM 52 to a YC conversion processing circuit (not shown) and converted into a luminance signal Y and color difference signals Cr and Cb. The converted image data is subjected to image compression in a predetermined compression format (for example, JPEG format) by a compression / decompression processing circuit (not shown). The compressed image data is recorded on the memory card 56 via the media controller 55. Alternatively, it is transmitted to the outside via the external I / F 57 and the connector 35.

  In addition to the release button 33 and the operation unit 38 described above, an EEPROM 58 is connected to the CPU 43. In the EEPROM 58, various control programs and setting information are recorded. In addition, voice data (vocabulary data and voiceprint data) of the operation command of the customer 20 using the voice operation function is registered in advance. The CPU 43 reads these pieces of information from the EEPROM 58 to the SDRAM 52 which is a working memory, and executes various processes.

  An amplifier (AMP) 59 is connected to the microphone 32. The sound input from the microphone 32 is amplified by an amplifier (AMP) 59 and converted into digital sound data by an A / D converter (A / D) 60. The audio data output from the A / D 60 is recorded on the memory card 56 after various signal processing such as noise removal is performed by the CPU 43. In the minutes creation mode, the data is transmitted to the vocabulary conversion circuit 61 and the voiceprint conversion circuit 62.

  The vocabulary conversion circuit 61 and the voiceprint conversion circuit 62 perform vocabulary conversion and voiceprint conversion on the voice data output from the A / D 60, respectively, and generate vocabulary data and voiceprint data. The vocabulary conversion circuit 61 performs vocabulary conversion in the same manner as a voice recognition device 92 of the server 12 described later.

  The vocabulary data and voiceprint data of the customer 20 registered in advance in the EEPROM 58 are preset in the first and second comparison circuits 63 and 64 through the CPU 43. The first and second comparison circuits 63 and 64 compare the preset data with the vocabulary data and the voiceprint data generated by the vocabulary conversion circuit 61 and the voiceprint conversion circuit 62. Then, a trigger signal is output when the generated data matches the registered data. Here, the first comparison circuit 63 transmits the matched vocabulary data to the CPU 43 together with the trigger signal.

  The logic circuit 65 calculates the logical product of the outputs of the first and second comparison circuits 63 and 64, and when both the first and second comparison circuits 63 and 64 generate a trigger signal, that is, both vocabulary data and voiceprint data. An interrupt signal is transmitted to the CPU 43 only when the generated data matches the registered data. The CPU 43 receives the interrupt signal from the logic circuit 65 and causes each part of the digital camera 10 to execute processing corresponding to the operation command represented by the vocabulary data transmitted from the first comparison circuit 63.

  The CPU 43 opens a line with the server 12 via the mobile phone 11 and the telephone line network 13 when a conference start command is inputted by voice from the customer 20. When a voice recording start / end command is input, voice recording by the microphone 32 is started / finished. When a data transmission command is input, the image data and audio data recorded on the memory card 56 are transmitted to the server 12 via the mobile phone 11. When a conference end command is input, the image data and audio data are transmitted to the server 12 and then the line with the server 12 is closed.

  In the digital camera 10, image data and audio data recorded on the memory card 56 in the minutes creation mode are transmitted to the server 12 at regular intervals through the mobile phone 11 and the telephone line network 13. Further, when the recording capacity of the memory card 56 is insufficient when the minutes creation mode is used, a warning is issued to the customer 20 in advance by turning on a beep sound or a lamp, and data transmission to the server 12 is performed. Or, it is urged to replace the memory card 56.

  As shown in FIG. 6, the mobile phone 11 is controlled centrally by the CPU 70. The mobile phone 11 includes a reception speaker 71 that outputs a communication partner's voice and a ringing melody, a transmission microphone 72 that collects a speaker's voice, and an operation unit 73 including various selection keys and dial keys. . The mobile phone 11 is provided with a liquid crystal display (LCD) 74, an antenna 75, a connector 76 (for example, a USB connector), and a memory 77.

  The LCD 74 displays a menu screen including various setting menus, a telephone number or mail address of the called party, an image downloaded from the Internet server, and the like via the LCD driver 78. The antenna 75 receives a radio signal from another mobile phone or the like via the communication I / F 79 and transmits a radio signal transmitted from the mobile phone 11 to the outside. The connector 76 is connected to the connector 35 of the digital camera 10 via a USB cable or the like, and transmits / receives data via the external I / F 80. The memory 77 temporarily stores image data and audio data transmitted from the digital camera 10.

  As shown in FIG. 7, the server 12 includes a buffer memory 90, a character recognition device 91, a voice recognition device 92, an editing device 93, a distribution device 94, and an automatic settlement device 95. The buffer memory 90 temporarily stores image data and audio data acquired by the digital camera 10 and transmitted from the mobile phone 11 via the telephone line network 13, and sorts these data by arranging them in time series. Are transmitted to the character recognition device 91 and the voice recognition device 92 at intervals of.

  The character recognition device 91 includes character recognition software, recognizes characters in the transmitted image data (image data obtained by photographing the entire whiteboard 21), and converts the characters into first text data. The voice recognition device 92 is provided with voice recognition software, recognizes the transmitted voice data (voice data containing the speech of the conference participants), and converts it into second text data.

  In character recognition, first, layout analysis is performed to extract a portion in which characters are written from image data, and one character is extracted from the extracted portion. A quantity such as an edge, contour, direction contribution, etc. is extracted and compared with a standard pattern registered in advance in a recognition dictionary (not shown). Then, several types of matched standard patterns are output as candidates, and misrecognition is corrected while taking into account the context before and after the candidates, and finally the remaining candidates are converted into first text data.

  On the other hand, in speech recognition, speech data of about 15 to 30 ms is set as one frame, and spectrum analysis is performed by fast Fourier transform or linear prediction while shifting by 5 to 20 ms, and the feature amount is calculated. Next, using a non-linear expansion / contraction pattern matching or a hidden Markov model, the calculated feature value is compared with a standard pattern registered in advance in a recognition dictionary (not shown). Then, several types of matched standard patterns are output as candidates, and misrecognition is corrected while taking into consideration the context before and after the candidates, and finally the remaining candidates are converted into second text data. In order to perform voice recognition with higher accuracy, the voices of the participants of the conference are registered in advance as sampling data in the voice recognition device 92 via the digital camera 10 before the conference starts. Voice recognition may be performed.

  The editing device 93 automatically edits the first and second text data converted by the character recognition device 91 and the speech recognition device 92 to create a minutes file. In the distribution device 94, the e-mail address of the customer 20 is registered in advance. The distribution device 94 adds the creation date and time and the name of the customer 20 to the header of the minutes file, and distributes them to the PC 15 of the customer 20 via the Internet 14 in the form of an e-mail.

  In the automatic settlement apparatus 95, the bank account of the customer 20 is registered in advance. The automatic settlement apparatus 95 automatically withdraws the system usage fee from the bank account of the customer 20 by electronic settlement after distribution of the minutes file by the distribution apparatus 94.

  Next, the effect | action by the said embodiment is demonstrated with reference to the flowchart of FIGS. First, as shown in FIG. 8, the customer 20 installs the digital camera 10 in a place where the entire whiteboard 21 can be photographed, connects the connectors 35 and 76 with a USB cable or the like, and connects the mobile phone 11 to the digital camera 10. To do. Further, the PC 15 is connected to the Internet 14. Then, the digital camera 10 is turned on, and the minutes creation mode is selected.

  At the start of the conference, the customer 20 inputs a conference start command to the microphone 32 by voice. The voice of the customer 20 input from the microphone 32 is amplified by the AMP 59 and converted into digital voice data by the A / D 60.

  As shown in FIG. 9, in the digital camera 10, the vocabulary data and voiceprint data of the customer 20 registered in advance in the EEPROM 58 in accordance with the selection of the minutes creation mode are sent through the CPU 43 to the first and second comparison circuits 63, Preset to 64. The voice data output from the A / D 60 is subjected to vocabulary conversion and voiceprint conversion by the vocabulary conversion circuit 61 and the voiceprint conversion circuit 62, respectively, and vocabulary data and voiceprint data are generated. Next, the first and second comparison circuits 63 and 64 compare the preset data with the generated vocabulary data and voiceprint data. When the generated data matches the registered data, a trigger signal is output from the first and second comparison circuits 63 and 64. Here, from the first comparison circuit 63, the matched vocabulary data is transmitted to the CPU 43 together with the trigger signal.

  In the logic circuit 65, the logical product of the outputs of the first and second comparison circuits 63 and 64 is calculated, and an interrupt signal is sent to the CPU 43 only when a trigger signal is generated in both the first and second comparison circuits 63 and 64. Sent. The CPU 43 receives an interrupt signal from the logic circuit 65 and opens a line with the server 12 via the mobile phone 11 and the telephone line network 13.

  8, when a voice recording start command is input to the microphone 32 by the customer 20 after the line is opened, voice recognition processing is performed in the same procedure as the processing shown in FIG. Recording of remarks begins.

  The sound recorded by the microphone 32 is amplified by the AMP 59 as described above, and converted to digital sound data by the A / D 60. The audio data output from the A / D 60 is recorded on the memory card 56 after various signal processing such as noise removal is performed by the CPU 43.

  The subject light incident through the imaging lens 30 and the aperture 40 of the digital camera 10 is photoelectrically converted by the CCD 46 and sampled by the CDS 47. The image data output from the CDS 47 is amplified by the AMP 48 and converted to digital image data by the A / D 49. The digitally converted image data is sequentially stored in the SDRAM 52 via the image input controller 50 and displayed on the LCD 37 as a through image.

  In the state described above, when the content of the conference written by the presenter 22 on the whiteboard 21 reaches one paragraph and the customer 20 inputs a shooting command into the microphone 32, the procedure similar to the process shown in FIG. Voice recognition processing is performed. At that time, image data (image data obtained by photographing the entire whiteboard 21) stored in the SDRAM 52 is read out to the image signal processing circuit 54, and various image processing is performed.

  The image data that has been subjected to various processes by the image signal processing circuit 54 is read from the SDRAM 52 to a YC conversion processing circuit (not shown) and converted into a luminance signal Y and color difference signals Cr and Cb. The converted image data is subjected to image compression in a predetermined compression format (for example, JPEG format) by a compression / decompression processing circuit (not shown). The compressed image data is recorded on the memory card 56 via the media controller 55.

  Image data and audio data recorded in the memory card 56 are transmitted from the mobile phone 11 to the server 12 via the telephone line network 13 at regular intervals. When the recording capacity of the memory card 56 is insufficient, a warning is issued to the customer 20 in advance, and data transmission to the server 12 or replacement of the memory card 56 is prompted.

  When the customer 20 inputs a data transmission command to the microphone 32 by voice, the voice recognition process is performed in the same procedure as the process shown in FIG. 9, and the image data and voice data recorded in the memory card 56 are transferred to the mobile phone. 11 is transmitted to the server 12 via 11. These series of processes are repeatedly performed until the conference is ended according to the operation command inputted by voice.

  When the customer 20 inputs a conference end command to the microphone 32, voice recognition processing is performed in the same procedure as the processing shown in FIG. 9, and after the image data and voice data are transmitted to the server 12, The line is closed.

  As shown in FIG. 10, the server 12 side first receives a conference start command from the customer 20 and opens a line with the digital camera 10 via the mobile phone 11. After the line is opened, the image data and audio data transmitted from the digital camera 10 via the mobile phone 11 are temporarily stored in the buffer memory 90. In the buffer memory 90, these data are sorted in time series and sorted, and transmitted to the character recognition device 91 and the speech recognition device 92 at regular intervals.

  In the image data transmitted from the digital camera 10, characters in the image data are recognized by the character recognition device 91, and converted into first text data. On the other hand, the speech data is speech-recognized by the speech recognition device 92 for speech of the participants in the conference and converted into second text data. When the conference end command is transmitted, the first text data and the second text data are automatically edited by the editing device 93, and a minutes file is created. If the transmitted data is none of the above, error processing is performed.

  The minutes file created by the editing device 93 has the date and time of creation and the name of the customer 20 added to the header by the distribution device 94 and is distributed via the Internet 14 to the PC 15 of the customer 20 in the form of an e-mail. After the minutes file is distributed by the distribution device 94, the automatic settlement device 95 automatically deducts the system usage fee from the bank account of the customer 20.

  With the configuration as described above, the customer 20 only needs to prepare the digital camera 10, the mobile phone 11, and the PC 15, and the minutes generation system 2 can be used anywhere as long as it can be connected to the telephone network 13. Can be used. Further, since the digital camera 10 having the voice operation function is used, the customer 20 can perform hands-free shooting, voice recording, data transmission, and the like, and can participate in the discussion. Furthermore, after the conference is over, the information can be easily and reliably shared by simply transferring the minutes file to each PC of the conference participants by e-mail.

  In the above embodiment, image data and audio data acquired by the digital camera 10 are transmitted to the server 12 via the mobile phone 11, but the digital camera 10 is directly connected to the telephone line network 13 via a modular jack. Alternatively, the digital camera 10 may be connected to the PC 15 of the customer 20 and data transmission may be performed via the Internet 14. Further, although the character recognition device 91, the voice recognition device 92, the editing device 93, and the distribution device 94 are provided in the server 12, these may be provided independently.

  Furthermore, the first and second text data converted by the character recognition device 91 and the speech recognition circuit 92 of the server 12 are converted into a more versatile file format such as RTF (RichText Format) or HTML (HyperText Markup Language). It may be output. In this way, the minutes file can be re-edited smoothly.

It is a figure which shows schematic structure of the minutes production system of this invention. It is explanatory drawing which shows the mode of the meeting using the minutes creation system. It is a front external perspective view of a digital camera. It is a back external appearance perspective view of a digital camera. It is a block diagram which shows the electric constitution of a digital camera. It is a block diagram which shows the internal structure of a mobile telephone. It is a block diagram which shows the internal structure of a server. It is a flowchart which shows the process sequence of a digital camera. It is a flowchart which shows the process sequence of the voice operation function of a digital camera. It is a flowchart which shows the process sequence of a server.

Explanation of symbols

2 Minutes creation system 10 Digital camera 11 Mobile phone 12 Server 13 Telephone network 14 Internet 15 Personal computer (PC)
20 Customer 21 Whiteboard 30 Imaging Lens 32 Microphone 33 Release Button 35 Connector 37 Liquid Crystal Display (LCD)
43 CPU
46 CCD
52 SDRAM
54 Image signal processing circuit 56 Memory card 58 EEPROM
61 vocabulary conversion circuit 62 voice print conversion circuit 63, 64 first and second comparison circuit 65 logic circuit 75 antenna 76 connector 77 memory 91 character recognition device 92 voice recognition device 93 editing device 94 distribution device 95 automatic settlement device

Claims (4)

A digital camera that captures the display area of the display device installed in the conference room, outputs digital image data, records the remarks of participants in the conference, and outputs digital audio data;
A communication device connected to the digital camera and transmitting the image data and audio data to the outside;
A character recognition device that recognizes characters in the image data transmitted from the communication device and converts the characters into first text data;
A voice recognition device that recognizes the voice data transmitted from the communication device and converts the voice data into second text data;
An editing device that automatically edits the first and second text data to create a minutes file;
A minutes creation system comprising: a delivery device for delivering the minutes file to a customer via a communication network.   2. The minutes creation according to claim 1, wherein the character recognition device, the speech recognition device, the editing device, and the distribution device are provided in a server connected to the communication device via a communication network. system.   The digital camera includes voice recognition means for voice recognition of an operation command inputted by the customer, and control means for controlling each unit to execute processing according to the voice-recognized operation command. The minutes creation system according to claim 1 or 2.   The automatic settlement apparatus according to any one of claims 1 to 3, further comprising: an automatic settlement device that automatically deducts a system usage fee from the bank account of the customer registered in advance after distribution of the minutes file. Minutes creation system.

Images