JP6332369B2 - Information processing apparatus and program - Google Patents

Description

The present invention relates to an information processing apparatus and a program .

  In the information-oriented society, it is important to consider the privacy of images taken with cameras. Recently, for privacy reasons, when a face image is detected from an image captured by a camera and the detected face image matches a face image of a specific person registered in advance, the face image is displayed. There has been proposed an apparatus in which mask processing (abstraction processing) is not performed and mask processing is performed on the face image when there is a mismatch (see Patent Document 1).

JP 2004-62560 A

  However, the above-mentioned Patent Document 1 lacks careful consideration according to the security level, and is not necessarily easy to use.

Therefore, the present invention has been made in view of the above-described problems, and an object thereof is to provide an information processing apparatus and a program that are easy to use in consideration of security.

  The information processing apparatus according to the present invention includes a first input device (52) that inputs an image, a second input device (52) that inputs at least one of position information and time information, and an output instruction for the image. A determination device (70) for determining whether or not to limit the output of the image based on information input by the second input device.

  In this case, the second input device inputs information on a position where the image is captured or information on a position where the image is output, and outputs information on a time when the image is captured or the image. It is possible to input time information.

  Further, the determination device may determine whether output restriction of the image is necessary based on whether security or privacy is ensured based on the information.

  The image processing apparatus may further include a first processing device (70) that restricts at least a part of the image when the determination device determines that the output of the image is restricted. In this case, the first processing device may limit at least a part of the image when the location information changes from a place where security is secured to a place where security is not secured. . In addition, the first processing device may identify a location where output restriction is applied in the image based on at least one of the position information and the time information. The image processing apparatus further includes a third input device (53) for inputting information on a subject in the image, and the first processing device applies output restriction in the image based on information input by the third input device. It is good also as specifying a location. In this case, the first processing device may output the subject information input by the third input device together with the image. Furthermore, the first processing device may convert the subject information input by the third input device based on a predetermined rule and output the information together with the image.

  In the information processing apparatus of the present invention, the determination device can determine whether or not to limit the output of the image based on information on the facial expression of the person in the image.

  The information processing apparatus of the present invention further includes a fourth input device (52) for inputting at least one of voice data and text data, and the determination device is based on at least one of the position information and the time information. It can be determined whether or not to limit the output of at least one of the voice data and the text data.

  In this case, the determination device may include a second processing device (70) for outputting text data converted / generated from the sound data when it is determined that the output of the sound data is limited. . A second processing device that converts and outputs at least one specific noun of the voice data and the text data to another noun when the judgment device judges that output restriction is required for the voice data; You may have.

  In the information processing apparatus of the present invention, the first housing (11) having the display unit (14) for displaying the image processed by the first processing device, and the second housing (51) having the determination unit. And the first housing and the second housing may be separated. In this case, a position detection unit (22) provided in the first housing for detecting position information of the first housing is provided, and the second input device inputs a detection result by the position detection unit. It is good to do.

  In addition, in order to explain the present invention in an easy-to-understand manner, the above description has been made in association with the reference numerals of the drawings representing one embodiment. However, the present invention is not limited to this, and the configuration of an embodiment described later is provided. May be modified as appropriate, or at least a part thereof may be replaced with another component. Further, the configuration requirements that are not particularly limited with respect to the arrangement are not limited to the arrangement disclosed in the embodiment, and can be arranged at a position where the function can be achieved.

INDUSTRIAL APPLICABILITY The present invention has an effect that it is possible to provide an information processing apparatus and a program that are improved in usability in consideration of security.

It is a functional block diagram of personal assistant system 100 concerning one embodiment. FIG. 2A to FIG. 2D are flowcharts showing a recording process of voice input from the voice input unit. It is a figure which shows voiceprint DB. It is a flowchart regarding the process of audio | voice data. It is a figure which shows storage data DB. It is a flowchart which shows the specific process of step S76 of FIG. It is a figure which shows keyword DB. It is a flowchart which shows the specific process of step S84 of FIG. It is a flowchart which shows the specific process of step S122 of FIG. It is a flowchart which shows the specific process of step S142 of FIG. 9, and S148. It is a figure which shows specific conversion word DB. It is a flowchart which shows the specific process of step S164 of FIG. 10, S166, S178, and S180. It is a figure which shows place name DB. It is a figure which shows keyword storage DB. FIG. 15A to FIG. 15D are diagrams illustrating examples of weight tables. It is a figure which shows keyword recording DB. It is a figure which shows command DB. FIG. 18A is a diagram showing a display example of a task list, and FIG. 18B is a diagram showing a display example of a recorded voice list. It is a flowchart which shows the process (the 1) performed simultaneously in step S96. It is a flowchart which shows the process (the 2) performed simultaneously in step S96. It is a flowchart which shows the process (the 3) performed simultaneously in step S96. It is a figure which shows security possible range DB. It is a figure which shows ambiguous word DB. It is a flowchart regarding input / storage processing of image data. It is a flowchart regarding the display processing of image data. It is a flowchart which shows the deletion process of the data by the portable terminal side. It is a flowchart which shows the deletion process of the audio | voice data by the server side.

  Hereinafter, an embodiment of the personal assistant system 100 will be described in detail with reference to FIGS. A block diagram of a personal assistant system 100 is shown in FIG. As shown in FIG. 1, the personal assistant system 100 includes a portable terminal 10 and a server 50.

  The portable terminal 10 is a terminal that can be carried by a user, such as a mobile phone, a smartphone, a personal handy-phone system (PHS), or a personal digital assistant (PDA). The size of the portable terminal 10 is, for example, enough to fit into a breast pocket. As shown in FIG. 1, the portable terminal 10 includes an input unit 12, a display unit 14, a playback unit 16, a warning unit 18, a biological information input unit 20, a position detection unit 22, and a time detection unit 24. A calendar unit 26, a flash memory 28, a communication unit 32, and a terminal-side control unit 30. In addition, the portable terminal 10 has a portable first housing 11 that stores at least a part of these components.

  The input unit 12 includes a voice input unit 42, an image input unit 43, and a text data input unit 44. The voice input unit 42 includes a microphone, acquires the user's voice and voices emitted around the user, and outputs the acquired voice to the voice encoding unit 45.

  The voice encoding unit 45 encodes the voice input from the voice input unit 42 by the encoding method determined by the initial setting of the terminal-side control unit 30 and outputs the encoded voice to the communication unit 32.

  The image input unit 43 includes an optical system, an image sensor, and the like, generates a moving image, and outputs it to the image encoding unit 46. The image encoding unit 46 encodes the image input from the image input unit 43 by the encoding method determined by the initial setting of the terminal-side control unit 30 and outputs the encoded image to the communication unit 32. In addition, when the input to the image input unit 43 is a moving image and there is an input to the audio input unit 42 simultaneously with the input of the moving image, the input data is associated (similar to the multiplexing process) Process). Note that the image input unit 43 is not limited to moving image data, and may output still image data to the image encoding unit 46.

  The text data input unit 44 includes an input interface such as a keyboard and a touch panel, and acquires text data corresponding to a user input operation. The input unit 12 also has a function of accepting user operation instructions from a touch panel or the like. Various data input from the input unit 12 may be stored in the flash memory 28 as necessary. In this case, various data stored in the flash memory 28 are deleted under the control of the server-side control unit 70 described later.

  The display unit 14 includes a display such as a liquid crystal display or an organic EL display. The display unit 14 displays data such as image data and character data on the display under the control of the terminal-side control unit 30 or the server-side control unit 70, or displays a menu for a user to perform an operation. To do.

  The reproduction unit 16 includes a speaker and outputs voice and sound. For example, the terminal-side control unit 30 or the server-side control unit 70 can cause the reproduction unit 16 to reproduce the sound of the moving image input by the audio input unit 42 when the display unit 14 reproduces a moving image. It should be noted that text data converted from voice to text data on the server 50 side may be displayed on the display unit 14 as a telop instead of or along with voice playback. In particular, it is desirable to reproduce text data instead of voice data in a place where confidentiality and privacy are not maintained.

  The warning unit 18 gives a warning to the user when an error occurs in the portable terminal 10. For example, the warning unit 18 outputs a warning sound via the playback unit 16 or a warning via the display unit 14. Display and so on.

  The biometric information input unit 20 acquires, for example, at least one of biometric information such as the state of the user's muscles (tension and relaxation) or blood pressure, heart rate, pulse, body temperature, etc., to the terminal-side control unit 30. This is a device that inputs data. In addition, as a method for detecting biological information, for example, a wristwatch type as described in JP-A-2005-270543 (US Pat. No. 7,538,890) can be adopted. The blood pressure and pulse may be detected by a pulse wave detection sensor using infrared rays, and the heart rate may be detected by a vibration sensor. When the heart rate is higher than normal, it is in a tension state, and when it is reduced, it is in a relaxed state. Further, since the pupil is enlarged in the tension state and the pupil is reduced in the relaxation state, a configuration may be applied in which the pupil is detected to determine whether the pupil is in the tension state or the relaxation state.

  The position detection unit 22 detects a user's position (absolute position). Here, for example, a GPS (Global Positioning System) is adopted. As the position detection unit 22, an absolute position measurement system using RFID (Radio Frequency IDentification) or the like may be employed.

  The time detection unit 24 has a time measuring function for detecting the current time. The calendar unit 26 stores a date and a day of the week in association with each other. The flash memory 28 is a memory that temporarily stores data. The communication unit 32 includes a wireless LAN unit for accessing an access point by WiFi communication, a wired connection unit using an Ethernet (registered trademark) cable, or a USB connection unit for communicating with an external device such as a computer. ing. In the present embodiment, the communication unit 32 can communicate with the communication unit 52 of the server 50.

  The terminal-side control unit 30 performs overall control of the components of the portable terminal 10 and executes processing on the portable terminal 10 side. For example, the terminal-side control unit 30 acquires the time when voice data, image data, text data, and the like are input to the voice input unit 42, the image input unit 43, and the text data input unit 44 via the time detection unit 24. In addition, the position of the portable terminal 10 when voice data, image data, and text data are input is acquired via the position detection unit 22. Then, when transmitting the voice data, image data, and text data to the server 50 side, the terminal-side control unit 30 transmits time and position information as metadata together with the data.

  For example, the server 50 is installed in a company where a user who uses the portable terminal 10 works. However, the present invention is not limited to this, and the server 50 may be installed in a system management company. As shown in FIG. 1, the server 50 includes a communication unit 52, a face recognition unit 53, a text data generation unit 54, a voice print analysis unit 55, a weighting unit 56, an extraction unit 58, and a classification unit 60. A conversion unit 62, a flash memory 64, a hard disk 66, and a server-side control unit 70. Further, the server 50 has a second housing 51 that accommodates at least a part of these parts. The second casing 51 and the first casing of the portable terminal 10 described above are mechanically (physically) separated.

  The communication unit 52 is the same as the communication unit 32 on the portable terminal 10 side, and in this embodiment, communication with the communication unit 32 on the portable terminal 10 side is possible. Data (sound data, image data and text data) received by the communication unit 52 is stored in the flash memory 64 via the server side control unit 70. That is, the communication unit 52 functions as an input unit that inputs voice data, text data, and image data to the server-side control unit 70 in the server 50.

  The face recognition unit 53 detects the face area of the subject (person) from the image transmitted from the portable terminal 10 side and stored in the flash memory 64, and the face of the detected face area is registered in the hard disk 66. And the face image included in the image transmitted from the portable terminal 10 side is identified. Various methods can be used as a face recognition method. In this embodiment, face recognition is performed by a face detection method by edge detection or shape pattern detection and / or a face detection method by hue extraction or skin color extraction. Shall. The face recognition unit 53 cooperates with the server-side control unit 70 to perform noise reduction processing and edge enhancement processing on the image data transmitted from the portable terminal 10 side to facilitate face recognition. Can also be applied. The face recognition unit 53 notifies the server-side control unit 70 of information (name, title, etc.) on the subject (person) recognized from the face.

  When a face is registered in the hard disk 66, the confidentiality (high, medium, low) of the face may be set. For example, if the face is registered as a highly confidential face, the server-side control unit 70 prohibits the transfer of the face image to the portable terminal 10 or sets the resolution of the image when displayed on the display unit 14. Processing such as dropping or applying a mosaic can be performed. Further, the server-side control unit 70 can display the information of the person notified from the face recognition unit 53 in the vicinity of the face image that has undergone processing such as resolution reduction or mosaicing. By doing this, even if it becomes impossible to recognize who the face is from the image alone due to mosaicing etc., the person in the mosaiced image can be identified by displaying the person information Will be able to. In this case, the server-side control unit 70 may display the person information (for example, name and title) after initial conversion.

  Further, when the face-side image not registered in the hard disk 66 is input, the server-side control unit 70 prohibits the transfer of the unregistered face image to the portable terminal 10 or displays it on the display unit 14. When you do this, you can reduce the resolution of the image or apply a mosaic. In this case, the privacy of unregistered persons can be protected.

  Note that a character recognition unit may be provided instead of or in addition to the face recognition unit 53. In this case, the server side control unit 70 is notified of the result of the character recognition unit recognizing characters existing in the image (for example, characters written on a whiteboard or blackboard, or characters displayed on the screen). And If the server-side recognition unit 70 determines that the content of the character is highly confidential, the server-side recognition unit 70 can reduce the resolution of the character portion of the image or apply a mosaic. The server-side control unit 70 may convert the recognition result by the character recognition unit according to a predetermined rule, and display the converted content near the character portion of the image.

  The text data generation unit 54 acquires voice data stored in the flash memory 64, converts the voice data, and generates text data. The generated text data is stored in the flash memory 64 via the server side control unit 70.

  The voiceprint analysis unit 55 performs voiceprint analysis by pattern matching with registered voiceprint data using the loudness (strength), frequency, and length of the voice, and identifies the person who produced the voice. is there. In the voiceprint analysis, it is not necessary to use all of the size (intensity), frequency, and length of the voice, and it is also possible to specify the person who has produced the voice by using at least the frequency of the voice. Here, when a plurality of human voices are input to the voice input unit 42, the accuracy of voiceprint analysis by the voiceprint analysis unit 55 may be reduced. In this case, if the image when the voice is input is input to the image input unit 43, the voice print analysis unit 55 may perform the voice print analysis in consideration of the identification result by the face recognition unit 53. For example, the voiceprint analysis unit 55 may analyze the input voice in consideration of the movement of a person's mouth that can be recognized by the face recognition unit 53.

  The weighting unit 56 acquires voice data stored in the flash memory 64 and text data generated from the voice data or text data input from the text data input unit 44, and weights each text data. The weighting unit 56 stores the numerical value (task priority) obtained by weighting in the flash memory 64 together with the text data.

  The weighting by the weighting unit 56 is performed based on, for example, the volume and frequency of speech and the meaning of text data. Specifically, the weighting unit 56 performs weighting based on the result (information indicating who is the person who made the sound) analyzed by the voiceprint analysis unit 55 based on the volume and frequency of the sound, and confidentiality from the meaning of the text data. We perform weighting according to sex. In the present embodiment, confidentiality means a degree of preference that is not seen by others (unspecified third parties).

  A change unit 72 and a setting unit 74 are connected to the weighting unit 56. The changing unit 72 changes the setting related to the weighting of the weighting unit 56, and the setting unit 74 changes the setting related to the weighting of the weighting unit 56 based on a user instruction. The setting unit 74 may change the setting based on a user instruction input from an input unit (keyboard or the like) of the server, or may be input from the portable terminal 10 via the communication units 52 and 32. The setting may be changed in response to a user instruction input from the unit 12.

  The extraction unit 58 extracts a predetermined word from the text data stored in the flash memory 64. That is, a predetermined word is extracted from information input to the input unit 12 of the portable terminal 10. The predetermined word means a word that is preferably not seen by others, that is, a word having a relatively high confidentiality. This word is determined in advance in a keyword DB (see FIG. 7) stored in the hard disk 66. ing.

  Note that the extraction unit 58 may extract confidentiality by setting a keyword for each user in cooperation with the face recognition unit 53 and the server-side control unit 70.

  The classification unit 60 classifies the words extracted by the extraction unit 58 into words with a high confidentiality level (first word) and words with a slightly high confidentiality level (secondary word) (second word). This classification is performed based on a keyword DB (see FIG. 7) stored in the hard disk 66.

  The conversion unit 62 converts the word having the confidentiality level “high” and the word having the confidentiality level “medium” based on a predetermined rule, and stores the converted word in the flash memory 64.

  The flash memory 64 temporarily stores data processed in the server 50. An erase unit 76 is connected to the flash memory 64. The erasure unit 76 erases voice data and text data stored in the flash memory 64 at a predetermined timing based on an instruction from the server-side control unit 70. The specific timing for erasing data will be described later. Note that other volatile memories may be used instead of the flash memory 64.

  The hard disk 66 stores data such as a database used in various processes. In place of the hard disk 66, other nonvolatile memory may be used.

  The server-side control unit 70 comprehensively controls each unit in the server 50 and executes processing on the server 50 side. The server 50 actually has a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), etc., and the CPU stores a program stored (installed) in the ROM or the like. By executing the functions, the functions of the respective units such as the face recognition unit 53, the text data generation unit 54, the weighting unit 56, the extraction unit 58, the classification unit 60, the conversion unit 62, and the voiceprint analysis unit 55 described above are realized.

  Further, the server-side control unit 70 performs masking processing (decreasing the resolution or applying a mosaic) on at least a part of the image based on the face recognition result of the subject (person) in the image performed by the face recognition unit 53. . For example, when the person in the image is a person (a highly confidential person) registered in advance as an important person inside or outside the company, masking processing is performed on the face of the person. Alternatively, if the person in the image is a person who is registered in advance as a telecommuter, a masking process is performed on the background image of the person (the part in which the situation in the home of the telecommuter is photographed). To do. Further, when the face recognition unit 53 can detect an expression such as whether the person who is the subject is smiling (the expression is soft) or the eyebrows are placed between the eyebrows (the expression is hard), the server-side control unit For example, if the face is a smile, the image is not subjected to masking processing, and if it is a facial expression with a wrinkle between the eyebrows, the subject or the background of the subject can be masked. As described above, since the time and position information are attached to the various data from the input unit 12 of the portable terminal 10 as metadata, the server-side control unit 70 uses this metadata to generate an image. It is also possible to determine whether or not to perform masking processing. For example, even if it is an image of the same person, if it can be estimated from the time and position as an image acquired (captured) in the office or at the home of a telecommuter, a masking process is performed and an image acquired outside the company is estimated. If possible, the masking process may not be performed. In addition, for example, even if it is an image of the same person, when it can be estimated that the image is viewed outside the company from the time and position of browsing the image, it is possible to estimate that the image is viewed inside the company by performing a masking process. In some cases, the masking process may not be performed. The server-side control unit 70 stores the masked image (converted image data) in the flash memory 64 together with the original image (original image data). In addition, what is necessary is just to detect the eyelid between eyebrows mentioned above and a smile detection by the facial expression detection part not shown. When the eyelid between eyebrows is detected by a facial expression detection unit (not shown), an image with eyelids between the eyebrows may be stored in the flash memory 64 as a reference image and detected by pattern matching, and the left eye and right eye may be detected. You may detect from the shadow distribution of the part of between. The detection of eyebrows between eyebrows is also disclosed in US Patent Publication No. 2008-292148. When a smile is detected by a facial expression detection unit (not shown), the evaluation points of the shape of each element of the eyebrows, pupils, and lips are calculated by referring to smile data stored in the flash memory 64. Good. Smile detection is also disclosed in US Patent Publication No. 2005-201594.

  Next, the process in the personal assistant system 100 of this embodiment is demonstrated in detail based on FIGS.

  First, a process (recording process) for storing the voice input from the voice input unit 42 in the flash memory 64 on the server 50 side will be described with reference to FIGS. 2 (a) to 2 (d). Of course, the recording process may be performed all the time. However, in order to save power and perform efficient recording, in the present embodiment, the recording process shown in FIGS. 2A to 2D is performed. Among the processes, at least two processes are performed in parallel or only one process is performed.

  In the image input process, if the portable terminal 10 is a type that is put in a pocket, the start timing may be set when the user performs an image input operation using a switch (not shown). For example, when an image input operation is performed simultaneously with the start of the conference, the situation of the conference is photographed (input), or when a telecommuter makes a meeting with a person in the company, the image input operation is performed, Shoot (input) faces and materials. In this case, the terminal-side control unit 30 controls the operation of the image input unit 43 based on the image input operation.

  Recently, a life log system for recording one's actions has also been proposed. When the portable terminal 10 is a type suspended from the user's neck or a type such as a head-mounted display, or when the image input unit 43 is provided separately from the portable terminal 10 (for example, glasses The image input may be performed at the same timing as the recording process.

(Recording timing example 1)
FIG. 2A is a flowchart showing a process of recording only while a certain person is uttering sound. Note that the voice input to the voice input unit 42 is input to the server-side control unit 70 via the communication units 32 and 52.

  In the process of FIG. 2A, in step S10, the server-side control unit 70 determines whether or not a voice is input from the voice input unit 42. If the determination here is affirmed, in step S12, under the instruction of the server-side control unit 70, the voiceprint analysis unit 55 performs voiceprint analysis of the input voice. In this voice print analysis, the voice data included in the voice print DB (see FIG. 3) stored in the hard disk 66 and the input voice data are collated (pattern matching) to thereby generate a person who has emitted the input voice. Is identified. Here, in the voiceprint DB of FIG. 3, a person name and voiceprint data of the person are associated with each other. When this system is used for business, for example, voiceprint data of all employees of the company can be registered in the voiceprint DB. When this system is used for private use, each user can individually register voiceprint data of family members, relatives, friends, etc. in the voiceprint DB. This registration is performed from the voice input unit 42 of the portable terminal 10.

  Next, in step S14, the server-side control unit 70 determines whether or not a person has been specified in step S12, that is, whether or not the voice of the person registered in the voiceprint DB. If the determination here is affirmed, the server-side control unit 70 starts recording (storage in the flash memory 64) in step S16. Since the recorded voice data is converted into text data by the text data generation unit 54, it can be said that the recording start timing is the text data generation timing. On the other hand, if the determination in step S14 is negative, the process returns to step S10.

  After the determination in step S14 is affirmed and the process proceeds to step S16, the server-side control unit 70 continues recording until voice input is interrupted for a predetermined time in step S18. Then, when there is no voice input for a predetermined time, that is, when it is considered that the voice input has ended, the determination in step S18 is affirmed. If the determination in step S18 is positive, the terminal-side control unit 30 ends the recording in step S20 and returns to step S10.

  After that, by repeating the above process, recording is executed at each timing when a person registered in the voiceprint DB utters a sound. The person who determines the recording timing may be managed in a DB different from the voiceprint DB. Thereby, for example, the person who determines the recording timing can be limited to the meeting organizer.

  In FIG. 2A, the case where recording is started at a timing when a certain person utters based on a voiceprint has been described. However, the present invention is not limited to this, and for example, a frequency related to a telephone (for example, a frequency of a ringtone) ) May be started from the point in time when it is input from the voice input unit 42. As a result, it is possible to record a telephone conversation without leaking it.

(Recording timing example 2)
FIG. 2B is a flowchart showing a process for performing recording at a pre-registered time. 2B is different from the case of FIG. 2A in that the voice data is transmitted from the communication unit 32 on the portable terminal 10 side to the server 50 side or not. The recording timing is switched.

  In FIG. 2B, in step S <b> 24, the terminal side control unit 30 detects the current time via the time detection unit 24. Next, in step S26, the terminal-side control unit 30 determines whether it is a predetermined recording start time. Here, the recording start time may be determined in advance when the portable terminal 10 is shipped, or may be input in advance by the user or the like from the input unit 12. As the recording start time, for example, there are many conversations with people and the amount of information is considered to be large (for example, one hour immediately after entering the company), or the time when concentration is easily interrupted (for example, before and after lunch break) 30 minutes or overtime (after 8 pm, etc.) at which fatigue reaches its peak.

  When the determination in step S26 is affirmed, the process proceeds to step S28, and the communication unit 32 transmits the voice data input to the voice input unit 42 to the server 50 side under the instruction of the terminal-side control unit 30. start. In this case, the audio data is stored (recorded) in the flash memory 64 via the communication unit 52 and the server-side control unit 70.

  Next, in step S <b> 30, the terminal-side control unit 30 detects the current time via the time detection unit 24. In the next step S32, the terminal-side control unit 30 determines whether or not a predetermined recording end time has arrived. If the determination here is affirmed, the process proceeds to step S34. If the determination is denied, the process returns to step S30. When the process proceeds to step S34, the communication unit 32 stops the audio data transmission to the server 50 side under the instruction of the terminal-side control unit 30. Thereby, recording ends. After that, it returns to step S24 and repeats the said process. Thereby, recording can be performed every time the recording start time arrives.

(Recording timing example 3)
FIG. 2C is a flowchart showing a process for performing recording at the end of a conference registered in advance. In the process of FIG. 2C, the recording timing is switched by switching whether or not the voice data is transmitted from the communication unit 32 to the server 50 side, as in FIG. 2B.

  In FIG. 2C, in step S <b> 36, the terminal side control unit 30 detects the current time via the time detection unit 24. Next, in step S38, the terminal-side control unit 30 extracts a meeting schedule from a task list (to be described later) stored in the flash memory 28, and a predetermined time before the meeting end time (for example, 10 minutes ago). If the determination here is affirmed, in step S40, recording is started in the same manner as in step S28 in FIG.

  In next step S <b> 42, the terminal side control unit 30 detects the current time via the time detection unit 24. In the next step S44, the terminal-side control unit 30 determines whether or not the conference end time used for the determination in step S38 has come. If the determination here is affirmed, the process proceeds to step S46. If the determination is denied, the process returns to step S42. When the process proceeds to step S <b> 46, the communication unit 32 stops the audio data transmission to the server 50 side under the instruction of the terminal side control unit 30. After that, it returns to step S36 and repeats the said process. This enables recording at a predetermined time at the end of the conference. The reason for recording at the end of the conference is that the conclusion of the conference is more likely to be announced and the schedule of the next conference is more likely to be announced at the end of the conference.

  In the process of FIG. 2C, recording may be continuously performed during the time when the conference is being performed. In addition, when the presenter or presenter of the conference is registered in the task list, only the voice of the registered presenter or presenter is recorded in combination with the process of FIG. Anyway.

(Recording timing example 4)
FIG. 2D is a flowchart showing a process of performing recording based on information input from the biological information input unit 20 (here, the user's muscle state (tension and relaxation)). is there. Note that the processing in FIG. 2D is also performed in the same manner as in FIGS. 2B and 2C, by switching whether or not the voice data is transmitted from the communication unit 32 to the server 50 side, thereby recording timing. Is to switch.

  In FIG.2 (d), the terminal side control part 30 acquires the state of a user's muscle via the biometric information input part 20 in step S50. Next, in step S52, the terminal-side control unit 30 compares the muscle state with a predetermined threshold value, and determines whether or not a predetermined relaxed state. If the determination here is affirmative, in step S54, recording is started in the same manner as in step S28 in FIG.

  In the next step S56, when the terminal side control unit 30 acquires the muscle state again, in the next step S58, the terminal side control unit 30 compares the muscle state with a predetermined threshold value. It is determined whether or not the predetermined tension state. If the determination here is affirmed, the process proceeds to step S60. If the determination is denied, the process returns to step S56. When the process proceeds to step S60, the communication unit 32 stops the audio data transmission to the server 50 side under the instruction of the terminal-side control unit 30. After that, it returns to step S50 and repeats the said process. Through the above processing, the degree of tension of the user is determined from the state of the muscles, and the voice of the user in a situation where the user is too relaxed and is not listening to other people's story (for example, a dozing situation). Automatic recording can be performed.

  In FIG. 2D, the voice is recorded only when the person is too relaxed. However, the voice may be recorded when the person is moderately strained together with or instead of the voice. good. This is because there is a high possibility that an important story is given when the person is moderately nervous.

  In addition, it is possible to detect whether the user is in a tense state or a relaxed state based on the amount of sweating of the hand holding the handset and the holding power of the handset by providing at least one of a sweating sensor and a pressure sensor on the handset (portable terminal housing). May be.

  The outputs of the perspiration sensor and the pressure sensor are transmitted to the terminal-side control unit 30, and the terminal-side control unit 30 starts recording by the voice input unit 42 when it is determined that the user is in a tension state or a relaxation state. May be.

  A sweat sensor may be provided with a plurality of electrodes to measure hand impedance. Mental sweating such as emotion, excitement, and tension has a small amount of sweating and a short sweating time, so a sweating sensor may be provided on the handset corresponding to the palm side of the middle hand where the amount of sweating is greater than that of a finger.

  As the pressure sensor, any of a capacitance type, a strain gauge, and an electrostrictive element may be used. When the user grips the handset with a pressure that is 10% or more higher than the pressure at which the user normally grips the handset, it is determined that the tension state is present. You just have to do it.

  Further, at least one of the perspiration sensor and the pressure sensor may be provided in the portable terminal 10 or may be provided in a mobile phone or the like.

  As described above, even when the recording start timing is reached in the processes of FIGS. 2A to 2D, for example, when the portable terminal 10 is present at the recording prohibited position, Recording may not be started. As the recording prohibition position, for example, a company other than the company where the user works can be employed.

(Audio data processing)
Next, audio data processing performed after audio data is recorded will be described with reference to FIGS. FIG. 4 is a flowchart relating to processing of audio data.

  In step S <b> 70 of FIG. 4, the server-side control unit 70 determines whether audio data has been recorded in the flash memory 64. If the determination here is affirmed, the process proceeds to step S72, and the text data generation unit 54 converts the voice data into text under the instruction of the server-side control unit 70. In this case, the audio data is converted into text every time a predetermined time is interrupted. The server-side control unit 70 also converts the voice data into text (text data), the time when the voice data is input to the voice input unit 42, the position where the voice data is input, and the voice level of the voice data. Is registered in the storage data DB (FIG. 5) in the flash memory 64. Note that the time and position information registered here is transmitted from the communication unit 32 together with the audio data as described above. Next, in step S74, the voiceprint analysis unit 55 specifies a person who has made voice analysis by voiceprint analysis under the instruction of the server-side control unit 70, and registers it in the storage data DB. In the case where the process of step S12 of FIG. 2A has been performed, the content of step S12 may be registered in the storage data DB without performing step S74.

  FIG. 5 shows the data structure of the storage data DB. The storage data DB stores the time, position, text data, voiced person, voice level, task flag, and task priority described above. The items of task flag and task priority will be described later.

  Returning to FIG. 4, in the next step S76, a subroutine for task determination is executed. In the task determination subroutine, for example, the processing of FIG. 6 is executed. In the process of FIG. 6, in step S <b> 110, the server-side control unit 70 determines whether the date and time is included in the text data. The date and time here includes not only a specific date and time of what year, month, day, but also date and time such as tomorrow, the day after tomorrow, morning, and afternoon. If the determination here is affirmed, the process proceeds to step S78 in FIG. 4 after it is determined as a task in step S114. If the determination is negative, the process proceeds to step S112.

  In step S112, the server-side control unit 70 determines whether or not a specific word is included in the text data. Here, a specific word is a word related to a task, for example, “to do”, “please do”, “do” (or “shiro”, “do”), “do” (or “do”) ) "," I will "," I am planning ". This specific word may be stored as a table in the hard disk 66 in advance when the apparatus is shipped, or may be added by the user as needed. If the determination in step S112 is affirmative, the process proceeds to step S78 in FIG. 4 after it is determined as a task in step S114. On the other hand, if the determination in step S112 is negative, after determining that the task is not a task in step S116, the process proceeds to step S78 in FIG.

  Returning to FIG. 4, in step S <b> 78, the server-side control unit 70 determines whether it is determined as a task as a result of the processing in FIG. 6. Hereinafter, processing when the determination in step S78 is affirmed and when it is denied will be described.

(If the determination in step S78 is affirmative (if it is a task))
If the determination in step S78 is affirmative, the process proceeds to step S80, and the server-side control unit 70 sets the task flag of the storage data DB (FIG. 5) to ON. Next, in step S82, the extraction unit 58 extracts keywords based on the keyword DB (FIG. 7) stored in the hard disk 66 under the instruction of the server-side control unit 70. As shown in FIG. 7, in the keyword DB, keywords are associated with detailed information, attributes, and confidentiality levels of the keywords. Therefore, the extraction unit 58 pays attention to the keyword item in the keyword DB, and extracts the keyword registered in the keyword DB from the text data.

For example, text data is
“At 13:00 on November 20th, we will have a meeting with Ichiro Aoyama of Dai Tokyo Co., Ltd. regarding the software specifications of Cool Blue Speaker 2.”
Suppose that

  In this case, the extraction unit 58 extracts “Cool Blue Speaker 2”, “Software”, “Specification”, “Dai Tokyo Co.”, and “Ichiro Aoyama” registered in the keyword DB of FIG. 7 as keywords. .

  The keyword DB needs to be created in advance. Also, it is assumed that the registered contents of the keyword DB can be added / changed as appropriate (for example, during maintenance). In FIG. 7, keywords may be registered with attributes such as patent information, budget information, and business negotiation information in addition to attributes such as personal names, company names, and technical terms.

  Returning to FIG. 4, in the next step S84, an analysis subroutine for each keyword is executed. FIG. 8 is a flowchart showing specific processing of the analysis subroutine in step S84.

  In FIG. 8, in step S120, the classification unit 60 acquires the confidentiality level of the keyword from the keyword DB under the instruction of the server-side control unit 70. Specifically, the classification unit 60 acquires “medium” as the confidentiality level of “cool blue speaker 2” from the keyword DB, acquires “medium” as the confidentiality level of “software”, and sets “specification”. Obtain “Medium” as the confidentiality level, “High” as the confidentiality level of “Dai Tokyo Co., Ltd.”, and “High” as the level of “Ichiro Aoyama”.

  Next, in step S122, the conversion unit 62 converts the keyword based on the confidentiality acquired in step S120 under the instruction of the server-side control unit 70, and stores the converted keyword in the flash memory 64. Execute.

  FIG. 9 is a flowchart showing specific processing of the subroutine of step S122. As shown in FIG. 9, the conversion unit 62 first selects one keyword from the keywords extracted by the extraction unit 58 in step S138. Here, as an example, it is assumed that “Daito Tokyo Co., Ltd.” is selected.

  Next, in step S140, the conversion unit 62 determines whether or not the confidentiality level of the selected keyword is “high”. As described above, since “Great Tokyo Co., Ltd.” has the confidentiality level “High”, the determination here is affirmed, and the process proceeds to Step S142. In step S142, the conversion unit 62 executes a subroutine for converting keywords according to confidentiality. Specifically, the process is executed according to the flowchart shown in FIG.

  In step S160 of FIG. 10, the conversion unit 62 determines whether or not the specific conversion word is included in the selected keyword. Here, the specific conversion word is, for example, a word that is frequently used for a company name as defined in the specific conversion word DB of FIG. 11 (corporation, limited company, (stock), (existing), etc.) Also, it means words that are frequently used in institutions such as countries (mechanisms, ministries, agencies, etc.) and words that are frequently used in educational institutions (universities, high schools, etc.).

  In the case of the selected keyword “Dai Tokyo Co., Ltd.”, since the specific conversion word “corporation” is included, the determination in step S160 is affirmed, and the process proceeds to step S162. In step S162, the conversion unit 62 converts the specific conversion word based on the specific conversion word DB. In this case, the “stock” portion of “Dai Tokyo Co., Ltd.” is converted to “company”. Next, in step S164, a conversion subroutine other than the specific conversion word is executed.

  FIG. 12 is a flowchart showing specific processing of the conversion subroutine of step S164. As shown in FIG. 12, in step S190, the conversion unit 62 determines whether the conversion target portion (portion other than the specific conversion word) is a place name. Here, although the conversion target portion “Daito Tokyo” includes the place name, it is not the place name itself, so the determination is denied and the process proceeds to step S194.

  In step S194, the conversion unit 62 determines whether the conversion target part is a name. Here, since it is not a name, judgment is denied and it transfers to step S198. In step S198, the conversion unit 62 initially converts the conversion target portion “DaiTokyo” to “D”. When the process of step S198 ends, the process proceeds to step S165 of FIG.

  In step S165, the conversion unit 62 combines the words after the conversion in steps S162 and S164. Specifically, “D” and “Company” are combined into “D Company”.

  Next, in step S168, the conversion unit 62 determines whether information is attached to the conversion target keyword “Dai Tokyo Co., Ltd.”. Here, the fact that the information is attached means a case where information is input in the information column of the keyword DB in FIG. Here, since “Denki Tokyo Co., Ltd.” is accompanied by “Shinagawa-ku, Tokyo,” the determination in step S168 is affirmed and the process proceeds to step S170.

  In step S170, the conversion unit 62 selects one piece of information that has not been selected from the accompanying information. Next, in step S172, the conversion unit 62 determines whether or not the confidentiality level of the selected information (for example, “electric machine”) is “high” or “medium”. Here, if the “electricity” has a confidentiality level of “low”, the determination in step S172 is negative, and the process proceeds to step S182. In step S182, the conversion unit 62 determines whether all information has been selected. Here, since “Shinagawa-ku, Tokyo” is not yet selected, the determination is denied and the process returns to step S170.

  Next, the conversion unit 62 selects unselected information “Shinagawa-ku, Tokyo” in step S170, and whether or not the confidentiality level of “Shinagawa-ku, Tokyo” is “high” or “medium” in step S172. Determine whether. Here, as shown in the keyword DB of FIG. 7, the place name is defined as “low” or according to the confidentiality level of the accompanying keyword, so “Shinagawa Ward, Tokyo” ”, The confidentiality level is“ high ”. Therefore, the determination in step S172 is affirmed, and the process proceeds to step S174. In step S174, the conversion unit 62 determines whether or not a specific conversion word is included in “Shinagawa-ku, Tokyo”. If the determination is negative, the process proceeds to step S180, and a conversion subroutine for converting information is executed. The conversion subroutine in step S180 is basically the same processing as in step S164 described above (FIG. 12).

  That is, in FIG. 12, the conversion unit 62 determines whether or not “Shinagawa-ku, Tokyo” is a place name in step S190. If the determination here is affirmed, in step S192, the conversion unit 62 executes conversion processing based on the place name DB shown in FIG. Specifically, the conversion unit 62 converts “Shinagawa-ku, Tokyo” to “Southern Kanto” by converting it with a conversion method having a confidentiality level of “high”. In the place name DB of FIG. 13, when the confidentiality level is “high”, the place name is expressed as a position in a relatively wide area, and when the confidentiality level is “medium”, the place name is displayed. Is expressed as a position in a narrower area than when the confidentiality level is “high”.

  After the process of step S192 is complete | finished, it transfers to step S182 of FIG. In step S182, since all information (electrical equipment, Shinagawa-ku, Tokyo) has already been selected, the determination in step S182 is affirmed and the process proceeds to step S184. In step S184, the conversion unit 62 associates the converted information with the converted keyword (step S165 or S166). Here, “Company D (Electric machinery, southern Kanto)”. Thereafter, the process proceeds to step S144 in FIG.

  In step S144 of FIG. 9, the converted keyword is stored in the area A of the keyword storage DB (see FIG. 14) stored in the flash memory 64. As shown in FIG. 14, the keyword storage DB includes storage areas for areas O, B, and C in addition to area A. In the region O, raw keyword data (keyword before conversion) is stored. When the storage process is completed, the process proceeds to step S154, and it is determined whether or not all the keywords extracted by the extraction unit 58 have been selected. If the determination here is negative, the process returns to step S138.

  Next, a case where the conversion unit 62 selects “Cool Blue Speaker 2” as a keyword in step S138 will be described. In this case, since the keyword is “cool blue speaker 2” and the confidentiality level is “medium”, the determination in step S140 is denied, and the determination in step S146 is affirmed, and the process proceeds to step S148. .

  In step S148, a subroutine for converting keywords according to confidentiality is executed. Specifically, as in step S142, the processing in FIG. 10 is executed. In the process of FIG. 10, in step S160, the conversion unit 62 determines whether or not the “cool blue speaker 2” includes the specific conversion word. However, since the determination here is negative, the process proceeds to step S166. Migrate and execute the conversion subroutine. In the conversion subroutine of step S166, the processing of FIG. 12 is executed in the same manner as steps S164 and S180 described above. In FIG. 12, since “Cool Blue Speaker 2” is neither a place name nor a person name, the determinations in steps S190 and S194 are denied, and the conversion unit 62 performs initial conversion in step S198. In this case, in the keyword DB, the English notation “Cool Blue Speaker 2” written together with “Cool Blue Speaker 2 (Japanese notation)” is converted into “CBS2” by initial conversion (upper case part conversion).

  When the processing in FIG. 12 is completed as described above, the process proceeds to step S168 in FIG. 10, but in the keyword DB in FIG. 7, no information is attached to “cool blue speaker 2”. The determination is denied, and the process proceeds to step S150 in FIG. In step S150, the converted keyword is stored in area B of flash memory 64 shown in FIG. That is, the conversion unit 62 stores the keyword itself in the region O and stores “CBS2” in the region B corresponding to the keyword. When the storage process is completed, the process proceeds to step S154, and it is determined whether or not all the keywords extracted by the extraction unit 58 have been selected. If the determination here is negative, the process returns to step S138 again.

  Next, a case where the conversion unit 62 selects the keyword “Ichiro Aoyama” in step S138 will be described. In this case, since the confidentiality level of “Ichiro Aoyama” is “high”, the determination in step S140 is affirmed and the process proceeds to step S142.

  In step S142, the process of FIG. 10 is executed as described above. In the process of FIG. 10, step S160 is denied and the process proceeds to step S166 (process of FIG. 12). In step S190 of FIG. 12, the determination is denied and the process proceeds to step S194. In step S194, the conversion unit 62 determines whether “Ichiro Aoyama” is the name. And if judgment here is affirmed, it will transfer to step S196. In step S194, it is determined that “Ichiro Aoyama” is the name because the attribute of “Ichiro Aoyama” in the keyword DB of FIG. 7 is the name of the supplier.

  In step S196, the conversion unit 62 initially converts “Ichiro Aoyama”. In step S196, if the confidentiality level of the keyword is “high”, both the name and the name are initially converted. That is, “Aoyama Ichiro” is converted to “AI”. On the other hand, for example, when the confidentiality level of the keyword is “medium” as in “Ueda Saburo” registered in the keyword DB of FIG. 7, only the name is initially converted. That is, “Ueda Saburo” is initially converted to “Ueda S”. In addition, only Mr. may be converted into "U Saburo" by initial conversion.

  When the process of step S196 is completed, the process proceeds to step S168 in FIG. Here, as shown in FIG. 7, the keyword “Ichiro Aoyama” is accompanied by “Dai Tokyo Co., Ltd., Camera AF Motor, October 15, 2009 Patent Workshop (Tokyo)” as information, so step S168. This determination is affirmed, and the routine goes to Step S170. In step S170, for example, the information “Dai Tokyo Co., Ltd.” is selected. Since “Great Tokyo Co., Ltd.” has the confidentiality level “High” as described above, the determination in Step S172 is affirmed and the process proceeds to Step S174. Since “Dai Tokyo Co., Ltd.” includes the specific conversion word “Co., Ltd.”, the determination in step S174 is affirmed, conversion of the specific conversion word (step S176), and conversion other than the specific conversion word (Step S178) is executed. Steps S176 and S178 are the same as steps S162 and S164 described above. If the determination in step S182 is negative, the process returns to step S170.

  Thereafter, Steps S170 to S182 are repeated until all the information has been selected. After all the information has been selected, in step S184, the converted information is associated with the converted keyword. Here, for example, it is associated with “AI (camera, AFM, October 15, 2009 T meeting (Tokyo))”. Then, when the storage in the area A is completed in step S144 in FIG. 9, the process proceeds to step S154, and it is determined whether or not all the keywords extracted by the extraction unit 58 have been selected. If the determination here is negative, the process returns to step S138 again.

  In the above process, if the determination in step S146 of FIG. 9 is negative, that is, if the confidentiality level of the keyword is “low”, the keyword is left as it is in region C (and region C) in step S152. O). When information is attached to the keyword, the information is also stored in the area C. For example, as shown in FIG. 14, in the case of the keyword “SBS company”, the area C is stored as “SBS machine Germany Munich”.

  In the above processing, for example, if the keyword selected in step S138 is “software”, the software is initially converted to “SW” and the information <sponge> shown in FIG. 7 is converted. Without associating with SW. In this case, the notation <xx> means that the word is handled on an equal basis with the keyword. That is, it means that either “software” or “sponge” is used. Therefore, when the above process is performed on the keyword “software”, “SW” and “sponge” are stored in the area B of the flash memory 64 on an equal basis. The use of “SW” and “sponge” will be described later.

  The above processing is also performed for other keywords (here, “specifications”), and if the determination in step S154 in FIG. 9 is affirmed, the process proceeds to step S124 in FIG.

  In step S124, the server-side control unit 70 acquires a weight related to the attribute of the speaker. In this case, the weight (Tw) is acquired from the position of the speaker based on the weight table regarding the attribute shown in FIG. For example, when the speaker is Saburo Ueda in FIG. 7, “2” of the manager (M) is acquired as the weight (Tw).

  Next, in step S126, the server-side control unit 70 acquires a weight related to the sound level. In this case, the server-side control unit 70 obtains the weight (Vw) based on the weight table relating to the sound level shown in FIG. 15B and the sound level stored in the storage data DB (see FIG. 5). To do. As shown in FIG. 5, when the audio level is 70 db, the weight (Vw) is 3. The reason why the weight (Vw) is larger as the voice level is larger is that the greater the voice level, the stronger the request and the higher the importance.

  Next, in step S128 of FIG. 8, the server-side control unit 70 acquires a weight related to the keyword. In this case, the server-side control unit 70 acquires the weight (Kw) based on the weight table related to the keyword shown in FIG. 15C and the keyword included in the text data of the storage data DB. In FIG. 15C, since “important”, “important”, “very important”, and “very important” are registered, if these keywords are included in the text data, the weight (Kw) is 2 Or 3 is acquired. In step S128, the server-side control unit 70 determines how many keywords with a confidentiality level “high” and keywords with a confidentiality level “medium” are included in the text data, and the determination result. And the weight (Cw) regarding the confidentiality of text data is acquired based on the weight table regarding the confidentiality of the keyword shown in FIG. For example, if the text data includes two keywords with a confidentiality level of “high” and one keyword with a confidentiality level of “medium”, the server-side control unit 70 sets Cw = 8 ( = 3 × 2 + 2 × 1).

When the process of step S128 in FIG. 8 is completed, the process proceeds to step S86 in FIG. In step S86, the server-side control unit 70 calculates the task priority (Tp) and registers it in the storage data DB (FIG. 5). Specifically, the server-side control unit 70 calculates the task priority (Tp) using the following equation (1).
Tp = Uvw × Vw + Utw × Tw
+ Ufw × Fw + Ukw × Kw + Ucw × Cw (1)

  Note that Uvw, Utw, Ufw, Ukw, and Ucw in the above equation (1) are weighting factors that take into account the importance of each weight (Vw, Tw, Fw, Kw, Cw). Can be set via the setting unit 74.

  Next, the process proceeds to step S88 in FIG. 4, and the server-side control unit 70 registers the keywords included in the text data in the keyword recording DB shown in FIG. Note that the keyword record DB of FIG. 16 is created, for example, in units of one week, one month, or one year. In the keyword record DB of FIG. 16, the keywords used at the same time as the keywords (called registered keywords) included in the text data, the speakers of the registered keywords, the date and time when the registered keywords were spoken, the location, etc. Record relevant information step by step. In addition, the number of times the registered keyword is associated with the related information is recorded as the degree of association. Furthermore, the number of times the registered keyword is issued is recorded as the appearance frequency. Note that the search frequency item in the keyword record DB in FIG. 16 will be described later.

  In addition, after the process of step S88 is completed, it returns to step S70.

(When the determination in step S78 is negative (when it is not a task))
Next, a case where the determination in step S78 is negative will be described. If step S78 is negative, the process proceeds to step S90, and the server-side control unit 70 turns off the task flag. Next, in step S92, the server-side control unit 70 determines whether or not the speaker is the user himself / herself. If the determination here is affirmed, the process proceeds to step S94, and it is determined whether or not the word issued by the user is a command. Here, for example, as shown in the command DB of FIG. 17, the word “task list” is a command for displaying a task list, and the word “voice recording text” is a command for displaying a voice recording list. It is assumed that the word “conversion” is a command for conversion processing. It is assumed that this command DB is stored in the flash memory 28 on the portable terminal 10 side or the hard disk 66 on the server 50 side. In this command DB, for example, when a user's voice is a “task list”, it is defined that a task list as shown in FIG. Details of the task list will be described later. Further, the command DB defines that when the user's voice is a “voice recording list”, a voice recording list as shown in FIG. 18B is displayed. Details of the voice recording list will be described later.

  Returning to FIG. 4, if the determination in step S94 is negative, the process returns to step S70, but if the determination in step S94 is affirmative, the process proceeds to step S96, and the server-side control unit 70 determines that the command A subroutine that executes processing according to the above is executed. Specifically, the processes of FIGS. 19, 20, and 21 are executed concurrently.

  First, processing on the server 50 side will be described with reference to the flowchart of FIG. On the server 50 side, in step S202, the server-side control unit 70 determines whether or not the command is a display request. In this case, as described above, commands such as “task list” and “voice recording list” correspond to the display request.

  Next, in step S <b> 204, the server-side control unit 70 extracts data necessary for performing display according to the command from the flash memory 64. For example, if the command is “task list”, text data to be displayed in the task list (text data with the task flag in FIG. 5 turned on) is extracted from the flash memory 64. In this case, the text data in which the task flag is turned on includes not only text data converted from voice data but also text data directly input from the text data input unit 44. Note that the task flag of the text data that is directly input is turned on / off by the same process as in FIG.

  Next, in step S206, the server-side control unit 70 acquires the current position of the user. In this case, the position information detected by the position detection unit 22 included in the portable terminal 10 is acquired via the terminal-side control unit 30, the communication units 32 and 52, and the like.

  Next, in step S208, the server-side control unit 70 determines whether it is a place where security can be ensured based on the acquired position information (current position). Here, as a place where security can be ensured, for example, the inside of a company can be mentioned. Registration of the company location is performed by the following method.

  For example, the user connects the portable terminal 10 to a PC (Personal Computer) by USB or the like, and activates a dedicated application using map information on the PC. Then, the location of the company is registered with the application by designating the location of the company. The location is specified by a drawing operation using a mouse or the like. The position of this company is represented as an area of a predetermined area. Accordingly, the company position can be represented by two diagonal points (latitude and longitude) of a rectangular area as shown in the security assured range DB of FIG. 22 is stored in the hard disk 66 of the server-side control unit 70.

  That is, in step S208, the server-side control unit 70 refers to the security assured range DB in FIG. 22 and is located in a place where security can be ensured when the user is within the range. To be judged.

  If the determination in step S208 is affirmed, the process proceeds to step S210. In step S210, the server-side control unit 70 acquires a conversion word associated with the keyword included in the extracted data from the regions O, A, B, and C, and proceeds to step S214. On the other hand, if the determination in step S208 is negative, the process proceeds to step S212. In step S212, the server-side control unit 70 acquires a conversion word associated with the keyword included in the extracted data from the areas A and B, and proceeds to step S214.

  In step S <b> 214, the server-side control unit 70 transmits the converted word associated with the extracted data and keyword to the portable terminal 10 via the communication unit 52.

  If the determination in step S202 is negative, that is, if the command is a command other than a display request, the server-side control unit 70 performs processing according to the command in step S216.

  Next, processing in the portable terminal 10 will be described based on FIG. In step S220 of FIG. 20, the terminal-side control unit 30 determines whether data has been transmitted from the server side. In this step, the determination is affirmed after step S214 of FIG. 19 is executed.

  Next, in step S221, the terminal-side control unit 30 determines whether or not the conversion words for the areas A, B, and C have been transmitted. Here, the determination is affirmed when step S210 in FIG. 19 is passed, and the determination is denied when step S212 is passed.

If the determination in step S221 is affirmed, in step S222, the terminal-side control unit 30 converts the keywords included in the extracted data with the conversion words of the regions A, B, and C. That is, for example, the extracted data is
“At 13:00 on November 20th, we will have a meeting with Ichiro Aoyama of Dai Tokyo Co., Ltd. regarding the software specifications of Cool Blue Speaker 2.”
Suppose that In this case, using the conversion words of regions A, B, and C,
“At 13:00 on November 20th, we will have a meeting with AI (Camera, AFM, T-Meeting, Tokyo, October 15, 2009) from Company D (Electricity, Southern Kanto) about SWSP of CBS2.”
Is converted.

On the other hand, if the determination in step S221 is negative, in step S223, the terminal-side control unit 30 converts the extracted data with the conversion word of region B and deletes the word of region A. In this case, the extracted data is
“Scheduled meeting with Nosan about CSP2 SWSP at 13:00 on November 20”
Is converted. Thus, in this embodiment, the data display mode is changed depending on whether security is ensured.

  After the process of step S222 or step S223 is performed as described above, the process proceeds to step S224, and the terminal-side control unit 30 displays the converted text data at a predetermined position on the display unit 14. Execute the process. In this display, the task time (date and time) may be simply displayed in order from the current time (date and time), but in this embodiment, the task priority is displayed in descending order instead. . As a result, the user can reduce oversight of important tasks, and even when a plurality of schedules are double-booked, the user can schedule a task with higher priority. When double-booking is performed, the terminal-side control unit 30 may issue a warning via the warning unit 18, or a person related to a schedule with a lower priority is included in the task. In this case, the terminal-side control unit 30 may automatically request the person to change the task schedule request by e-mail. However, the display is not limited to the order in which the task priority is high as described above. Further, it may be displayed in order of date and time so that the font, color, size, etc. of the task with high task priority are changed and displayed prominently. In addition, the tasks having the same task priority may be displayed in order of date and time after being arranged in the order of higher task priority.

  As described above, the screens shown in FIGS. 18A and 18B are displayed by the processing of FIGS. Note that a task item is provided in the recorded voice list of FIG. The user can switch the task flag on and off by touching the item of the task on the touch panel. In this case, the server-side control unit 70 changes the task flag in FIG. 5 when recognizing the task flag switching operation by the user. Thereby, as a result of the processing of FIG. 6, even if the on / off state of the task flag is different from the user's recognition, the user can manually change the task flag. When the user turns on the task flag, the server-side control unit 70 may automatically turn on the task flag for text data similar to the text data of the task thereafter.

  In the process of FIG. 20, the terminal-side control unit 30 transmits the current position acquired by the position detection unit 22 to the server-side control unit 70 side, and uses the conversion word transmitted from the server-side control unit 70. The text data is converted and displayed. Therefore, in the present embodiment, it can be said that the terminal-side control unit 30 restricts display on the display unit 14 according to the current position acquired by the position detection unit 22.

  Next, processing performed in parallel with the processing of FIG. 20 will be described based on FIG. In FIG. 21, in step S232, the terminal-side control unit 30 determines whether or not the document conversion button has been pressed by the user. The document conversion button is a button displayed at the upper right end in FIGS. 18A and 18B. The user presses the document conversion button by a touch panel operation, a keyboard operation, or the like. If the determination in step S232 is affirmed, the process proceeds to step S234. If the determination is negative, the process proceeds to step S238.

  In step S234, the terminal-side control unit 30 determines whether or not a convertible keyword is displayed. Here, the convertible keyword means a keyword in which a plurality of conversion words are associated with one keyword, such as “SW” and “sponge” shown in FIG. Accordingly, when such keywords are included in the text data displayed on the display unit 14, the determination here is affirmed, and the process proceeds to step S236. On the other hand, if the determination in step S234 is negative, the process proceeds to step S238.

When the process proceeds to step S236, the terminal-side control unit 30 converts the keyword. Specifically, for example,
“At 13:00 on November 20th, we will have a meeting with AI (Camera, AFM, T-Meeting, Tokyo, October 15, 2009) from Company D (Electricity, Southern Kanto) about SWSP of CBS2.”
Can be converted into “sponge”, the terminal-side control unit 30
“At 13:00 on November 20, CBS2 sponge SP is scheduled to be discussed with AI (Camera, AFM, October 15, 2009 T meeting (Tokyo)) of Company D (Denki, South Kanto)”
And converted and displayed.

  Even if the software cannot be recalled with the display “SW”, the user can press the document conversion button and look at the notation “sponge”, so that the association such as sponge → soft → software (Software) Be able to recall software. When the word sponge is seen for the first time, such an association may not be possible, but if the method of the association is made known in-house, the software can be easily recalled.

  Next, in step S238, the terminal-side control unit 30 determines whether or not the pre-conversion display button (see FIGS. 18A and 18B) has been pressed. The case where the user presses the pre-conversion display button is when the user wants to see a sentence in which the keyword is not converted. If the determination is negative, the process returns to step S232. If the determination is positive, the process proceeds to step S240. In step S240, the terminal-side control unit 30 acquires the current position of the user. In step S242, the terminal-side control unit 30 determines whether the current position is a place where security can be ensured. If the determination here is negative, that is, if the user is in a place where security cannot be ensured, it is necessary to restrict the user from showing the text before conversion, so that the user cannot display in step S252. To return to step S232. In addition, as a notification method in step S252, display on the display unit 14, a warning via the warning unit 18, or the like can be employed.

  When the determination in step S242 is affirmed, the process proceeds to step S244, and the terminal-side control unit 30 displays a question item (a question that can be easily answered by a user) on the display unit 14. The question items are assumed to be stored in the hard disk 66 on the server 50 side, and the terminal-side control unit 30 reads the question items from the hard disk 66 and displays them on the display unit 14. For example, the user may register the questions and answer examples in advance.

Next, in step S246, the terminal-side control unit 30 determines whether or not the user has input an answer to the input unit 12 by voice. If the determination here is affirmed, in step S248, the terminal-side control unit 30 determines whether or not it is the voice of the user and the answer is correct. Whether or not the voice is a user's voice is determined by using the result of analyzing the voice in the voice print analyzing unit 55 on the server 50 side. If the determination is negative, in step S252, the user is notified that display is impossible. On the other hand, if the determination in step S248 is affirmed, the process proceeds to step S250, and the keyword is converted into a pre-conversion state with the conversion word in region O and displayed. Specifically, the sentence as it is input by voice, that is, in the above example,
“At 13:00 on November 20th, we will have a meeting with Ichiro Aoyama of Dai Tokyo Co., Ltd. regarding the software specifications of Cool Blue Speaker 2.”
Is displayed. Thereafter, the process proceeds to step S232, and the above process is repeated. In the above description, the case where the user answers the question with a voice has been described. However, the present invention is not limited thereto, and the answer may be input from a keyboard or the like. In this case, the terminal-side control unit 30 may determine whether to display the state before conversion based on the result of biometric authentication such as fingerprint authentication in addition to the answer to the question.

  As described above, when the process of step S96 in FIG. 4 is completed, the process returns to step S70.

  On the other hand, if the determination in step S92 in FIG. 4 is negative, that is, if the speaker is not a user, the process proceeds to step S100, and the terminal-side control unit 30 displays the speaker information. Here, the terminal-side control unit 30 performs display based on the information received from the server-side control unit 70. Specifically, if the speaker is Ichiro Aoyama, the terminal side control unit 30 receives the information from the server side control unit 70 and displays “Ichiro Aoyama”. In addition, when the information incidental to Ichiro Aoyama is received, the information may be displayed. Further, when a task related to Ichiro Aoyama is received from the server-side control unit 70, the task may be displayed together.

  By doing so, for example, when Mr. Ichiro Aoyama calls the user “Good morning”, the name, related information, tasks, and the like can be displayed on the display unit 14. Thereby, it is possible to assist the user in remembering the name and information of the person or what to do in relation to the person.

  Next, in step S102, the server-side control unit 70 determines whether or not a word registered in the ambiguous word DB shown in FIG. If the determination is negative, the process returns to step S70, but if the determination is affirmative, the process proceeds to step S104.

  In step S104, the server-side control unit 70 and the terminal-side control unit 30 execute processing corresponding to the spoken word based on the ambiguous word DB of FIG. Specifically, when “that case” or “example case” is stated, the server-side control unit 70 refers to the keyword record DB, and the speaker appears among the keywords included in the related information. A keyword having a frequency higher than a predetermined threshold is extracted and transmitted to the terminal-side control unit 30. Then, the terminal side control unit 30 displays the received keyword on the display unit 14. For example, when the speaker is the manager Yamaguchi and the appearance frequency threshold is 10, the keyword “project A” in the keyword record DB of FIG. 16 is displayed on the display unit 14. Further, as shown in FIG. 23, when “the case of (place name)”, for example, “the case of Hokkaido” is stated, the speaker is included in the related information, and the voice data is It seems that the keyword or speaker whose input position (latitude, longitude) is within a predetermined range (for example, within Hokkaido) is included in the related information, and the word “Hokkaido” is included in the related information. Key words are extracted and displayed on the display unit 14. Furthermore, for example, when “May ○ Day (MM / DD)” is stated, the date and time when the speaker is included in the related information and the voice data is entered is ○ The keyword or speaker that matches the day (MM / DD) is included in the related information, and the word “○ month ○ day (MM / DD)” is included in the related information The keywords are extracted and displayed on the display unit 14. Furthermore, there is a case where it can be easily estimated from a keyword record DB in FIG. 16 that a person speaks at a certain time (date and time). In such a case, related keywords may be displayed from the speaker and the current time.

  In step S104, even if the speaker has asked an ambiguous question by executing the processing as described above, the question is automatically judged and displayed to the user. It becomes possible. In step S104, each time a keyword is displayed, the server-side control unit 70 updates the keyword record DB search frequency. This search frequency can be used, for example, when a keyword with a higher search frequency is preferentially displayed.

(Image data processing)
Next, processing of image data by the server-side control unit 70 when the image data is transmitted to the server-side control unit 70 will be described with reference to FIGS. 24 and 25. FIG. 24 is a flowchart regarding image data input / storage processing.

  In the process of FIG. 24, first, in step S302, the server-side control unit 70 determines whether image data has been input. If the determination here is affirmed, the process proceeds to step S304. In step S304, the server-side control unit 70 uses the position information detected by the position detection unit 22 or the time detection unit 24 when the image input unit 43 acquires an image metadata (image information metadata in the portable terminal 10). Detected time information) is acquired.

  Next, in step S306, the server-side control unit 70 determines whether the image data is data of a masking process target image based on the position information or time information. In this case, when the position where the image is acquired is the house of the company or teleworker, or when the time when the image is acquired can be estimated to be the shooting time at the house of the company or telecommuter It is determined that the data is data of an image to be masked (the determination in step S306 is affirmed).

  When the determination in step S306 is affirmed, the process proceeds to step S308, and the server-side control unit 70 specifies a portion to be masked in the image. For example, the server-side control unit 70 acquires the face recognition result of the subject (person) in the image from the face recognition unit 53, and based on the face recognition result, the subject is an important person (confidentiality) inside or outside the company. If it is a high person), the subject is specified as a portion to be masked. Further, for example, if the subject is a telecommuter based on the face recognition result by the face recognition unit 53, the server-side control unit 70 specifies the background portion of the subject as a portion to be masked. . In addition, for example, when a character recognition unit (not shown) recognizes characters written on a whiteboard or blackboard, or characters displayed on a screen, a highly confidential word is included. The server-side control unit 70 specifies the character part as a part to be masked.

  Next, in step S310, the server-side control unit 70 performs a masking process on the part specified in step S308 in the image data, generates an image (converted image) after the masking process, and stores the converted image in the flash memory. 64. In this case, the server-side control unit 70 stores the converted image in the storage area of the converted image provided in the flash memory 64.

  Next, in step S <b> 312, the server-side control unit 70 stores the original image data in the flash memory 64. In this case, the original image is stored in the storage area of the original image provided in the flash memory 64. The converted image and the original image of the converted image are stored in the flash memory 64 in an associated state.

  If the determination in step S306 is negative, that is, if the image input to the server-side control unit 70 is not a masking process target image, in step S312, the server-side control unit 70 Is stored in the flash memory 64.

  As described above, the image data input / storage processing in the server-side control unit 70 is completed.

  In the above description, whether or not to perform the masking process is determined based on the position and time when the image is acquired. However, the present invention is not limited to this. If the face recognition unit 53 can detect the facial expression of the subject, that is, whether the person who is the subject is smiling (the facial expression is soft) or the eyebrows are placed between the eyebrows (the facial expression is hard), the server For example, if the smile is a smile, the side control unit 70 does not perform masking processing on the image, and if the facial expression looks like a wrinkle between the eyebrows, the side control unit 70 determines to perform masking processing on the subject or the background of the subject. May be.

  Next, image data display processing by the server-side control unit 70 will be described with reference to FIG. FIG. 25 is a flowchart relating to image data display processing.

  In the processing of FIG. 25, first, in step S330, the server-side control unit 70 waits until a display instruction from the user via the portable terminal 10 is input. Next, in step S332, the server-side control unit 70 specifies the image data specified by the display instruction (included in the display instruction). Next, in step S334, the server-side control unit 70 determines whether a converted image corresponding to the specified image data exists in the flash memory 64. If the determination is negative, the process proceeds to step S342, and the server-side control unit 70 transmits the original image data (data input to the server-side control unit 70) to the terminal via the communication units 52 and 32. Transmit to the side control unit 30. The terminal-side control unit 30 displays the received image data via the display unit 14.

  On the other hand, if the determination in step S334 is affirmed, the process proceeds to step S336. In step S336, the server-side control unit 70 acquires position information or time information. In this case, the server-side control unit 70 acquires the position or time of the portable terminal 10 where the display instruction is input from the user.

  Next, in step S338, the server-side control unit 70 determines whether to display the converted image based on the acquired position or time information. For example, the server-side control unit 70 can estimate that the acquired position is a place where security is not maintained (for example, outside the company) or that the portable terminal 10 is located where security is not maintained from the time. Is determined to display the converted image (Yes at step S338). In this case, the server-side control unit 70 transmits the converted image data to the terminal-side control unit 30 via the communication units 52 and 32 in step S340. The terminal-side control unit 30 displays the received converted image data via the display unit 14. On the other hand, for example, when the acquired position is a place where security is maintained (for example, in-house) or when it can be estimated from the time that the portable terminal 10 is located in a place where security is maintained. It is determined that the converted image is not displayed, that is, the original image is displayed (No in step S338). In this case, the server-side control unit 70 transmits the original image data to the terminal-side control unit 30 via the communication units 52 and 32 in step S342. The terminal-side control unit 30 displays the received original image data via the display unit 14.

  As described above, when the process of step S340 or step S342 is completed, the entire process of FIG.

  In the above, a converted image is generated in advance based on the position or time at which the image was acquired in the process of FIG. 24, and generated in advance in the process of FIG. 25 based on the position or time of the portable terminal 10. The case where it is determined whether or not to display the converted image has been described, but the present invention is not limited to this. For example, the process of FIG. 24 may be omitted, and the converted image may be generated only when it is determined in the process of FIG. 25 that the converted image should be displayed based on the position or time of the portable terminal 10. . Even in this case, the converted image may be generated based on the image acquisition position and time and / or the image display position and time.

  Note that when a converted image is generated in the process of FIG. 24, the converted image may always be displayed (the original image is not converted) regardless of the display position and time. Also, the user may be able to set whether to always display the converted image or to perform the processing of FIG.

  Further, in the above, the server-side control unit 70 acquires information on the position or time at the time when the display instruction is input, and based on the information, whether to display the converted image data or the original image data Explained the case of judging. However, the present invention is not limited to this. For example, the server-side control unit 70 monitors the position of the portable terminal 10 that issued the display instruction, and displays the original image data while the portable terminal 10 exists in a place where security is ensured. Then, when the portable terminal 10 moves to a place where security is not ensured, the converted image data may be displayed. By doing in this way, it becomes possible to display appropriately considering the movement of the portable terminal 10.

  When the image data is moving image data, the playback unit 16 can output sound associated with the image from the speaker when displaying the image (moving image). Further, when the audio data of the moving image is converted into text data by the method described above, the playback unit 16 may display the text data as a telop when displaying the image. In this case, when the text data is changed to a superordinate concept or the like, the reproducing unit 16 may display the changed text data. Further, when outputting the sound from the speaker, the reproducing unit 16 may output the text data converted into the superordinate concept or the like as the sound data. In this case, there is no voice data in the text data converted into a superordinate concept or the like. Therefore, in such a case, a speech unit (for example, an associative word of an encoder) used for speech synthesis with reference to a phoneme symbol from among a large number of speech units of speech waveform data (speech database) recorded in advance. It is assumed that a speech synthesis technique for selecting “gu”, “ru” “gu”, “ru”) is used. The speech synthesis technique is described in, for example, Japanese Patent No. 3,727,885. If the image data displayed on the display unit 14 is converted image data, no sound may be output.

  In addition, when the masked portion of the converted image is a person's face, the server-side control unit 70 provides information on the person (name or title) recognized by the face recognition unit 53 in the vicinity of the portion. ) May be displayed. The person information in this case may be an initial or the like.

  In the above description, an example in which the server-side control unit 70 displays a converted image when it can be determined that security is not secured based on the position or time has been described. However, the present invention is not limited to this. For example, when security is not ensured, neither image data nor converted image data may be displayed. Also, either image data or converted image data may be displayed or neither image data may be displayed depending on the degree of security.

  The determination as to whether or not to generate / display the converted image data is not limited to the above. For example, the server-side control unit 70 acquires the position and time at which the image is acquired, the position and time at which the image is displayed, It is also possible to digitize elements such as a person and a facial expression of the person displayed in the image, and determine whether to generate / display converted image data based on the total value of each element. Further, the server-side control unit 70 may weight each element and determine whether to generate / display converted image data based on the total value of each element after weighting.

  Next, the erasing process of data acquired by the portable terminal 10 and the server 50 will be described with reference to FIGS.

(Data erasure process (Part 1: Elimination of conversion data (text data)))
FIG. 26 is a flowchart showing a process for deleting information acquired by the portable terminal 10 from the server 50 side. As shown in FIG. 26, the terminal-side control unit 30 determines whether or not a certain time (for example, 2 to 3 hours) has elapsed since data acquisition in step S260. If the determination here is affirmative, the process proceeds to step S262, and the terminal-side control unit 30 erases the text data (including the word before conversion and the word after conversion) stored in the flash memory 28. To do. On the other hand, even if the determination in step S260 is negative, the terminal-side control unit 30 determines whether or not the user has moved from the company to the outside in step S264. If the determination here is affirmed, the process proceeds to step S262, and data is erased in the same manner as described above. If the determination in step S264 is negative, the process returns to step S260. As described above, when a predetermined time elapses after data is acquired or when security cannot be ensured, leakage of important data can be prevented by deleting the data. In the above description, the case of deleting all of the text data has been described. However, the present invention is not limited to this, and only the most important data may be deleted in step S262. For example, only the data in the area A and the data in the area O may be erased.

  In the process of FIG. 26, when the user (portable terminal 10) has been outside the company from the beginning, the converted data may be erased from the flash memory 28 immediately after being displayed on the display unit 14. .

(Data erasure process (part 2: audio data erasure))
The server-side control unit 70 executes the erasure process of FIG. 27 for each audio data. The server-side control unit 70 determines whether or not the text data generation unit 54 has converted (successfully) the speech data into text data in step S270 of FIG. If the determination is negative, the process proceeds to step S280. If the determination is positive, the process proceeds to step S272, and the server-side control unit 70 acquires the name of the person who uttered the voice data. To do. Here, the server-side control unit 70 acquires the name of the person who uttered from the voiceprint analysis unit 55, and proceeds to step S274.

  In step S274, the server-side control unit 70 determines whether the person who has spoken is other than the user himself / herself. If the determination here is affirmed, the server-side control unit 70 deletes the voice data converted into the text data in step S276. On the other hand, if the determination in step S274 is negative, that is, if it is the user's own voice data, the process proceeds to step S278, where the voice data is erased after a predetermined time, and all the processes in FIG. To do.

  On the other hand, if the determination in step S270 is negative and the process proceeds to step S280, the server-side control unit 70 enables the audio data to be reproduced. Specifically, the server-side control unit 70 transmits audio data to the flash memory 28 of the portable terminal 10. In step S280, the user is warned via the warning unit 18 that the voice data could not be converted into text data. Based on this warning, when the user inputs an instruction to reproduce audio data from the input unit 12 of the portable terminal 10, the audio data stored in the flash memory 28 is reproduced via the reproduction unit 16.

  Next, in step S282, the server-side control unit 70 deletes the audio data transmitted to the flash memory 28 (that is, the audio data reproduced by the reproduction unit 16), and the entire process of FIG.

  By executing the audio data erasing process as described above, the storage amount of the audio data in the server 50 can be reduced, so that the storage capacity of the flash memory 64 of the server 50 can be reduced. In addition, it is possible to consider privacy by deleting voice data other than the user immediately after being converted into text data. In this embodiment, if a speech synthesizer is provided in the server 50, even if the speech data is deleted, the speech is reproduced from the text data stored in the flash memory 64 by the function of the speech synthesizer. can do.

(Data deletion process (3: Image data deletion))
The image data (including the original image data and the converted image data) is deleted from the flash memory 64 by the server-side control unit 70 after a predetermined time has elapsed since the data acquisition, for example. However, the present invention is not limited to this. For example, it may be erased with the same logic as that of text data.

(Data deletion process (part 4: task deletion))
The server-side control unit 70 deletes the task according to the following rules.
(1) When the task is related to a meeting outside the company In this case, the current position detected by the position detection unit 22 matches the meeting holding location specified by the task, and the current time detected by the time detection unit 24 When the meeting start time specified by the task has passed, the task is deleted. When the current time has passed the conference start time, but the current position does not match the conference location, the server-side control unit 70 receives the warning from the warning unit 18 via the terminal-side control unit 30. Alert the user. Thereby, it is possible to suppress forgetting to execute the task. Further, the present invention is not limited to this, and for example, a warning may be issued before a predetermined time (30 minutes before the task). Thereby, it is possible to prevent forgetting to execute the task.
(2) When the task is related to an in-house meeting In this case, a position detection unit that can detect that the user has entered the conference room, such as RFID, is adopted as the position detection unit 22. The task is deleted when the current position detected by is coincident with the conference room specified by the task, and the current time detected by the time detection unit 24 has passed the conference start time specified by the task. Also in this case, a warning can be used together as in (1) above.
(3) When the task is related to shopping and the place to shop is specified In this case, the current position detected by the position detector 22 matches the place specified by the task, and “Thank you. The task is erased when a voice such as “It was received” is input from the voice input unit 42 or purchase information is input from the POS cash register terminal to the input unit 12 by radio or the like. In addition to the input from the POS cash register terminal, for example, when the portable terminal has an electronic money function, the task may be deleted at the stage where payment is completed by the function. In the case of a life log camera in which the image input unit 43 is hung from the neck or attached to the ear, the task may be deleted based on the imaging result of the life log camera.
(4) In addition, when the time is specified in the task In this case, the task is deleted when the current time detected by the time detection unit 24 has passed the implementation time specified in the task.

  As described above in detail, according to the present embodiment, the communication unit 52 inputs an image, position information, and time information, and the server-side control unit 70 receives an image output instruction (display instruction) from the user. If there is, it is determined whether to limit the display of the image based on the position or time. This makes it possible to perform image display restrictions (masking processing: mosaic processing, resolution adjustment, etc.) according to position and time. In the present embodiment, for example, when the position information and time information input by the communication unit 52 are information on the position and time at which the image is captured, the server-side control unit 70 captures the image. When it can be determined from the position and time that there is a high possibility that at least a part of the image has confidentiality, it can be determined that the display of the image is restricted. For example, when the position information and time information input by the communication unit 52 are information on the position and time at which the image is viewed, the server-side control unit 70 estimates the location where the image is viewed from the information. Then, it can be determined that the display is limited to the image according to the place. In this way, the server-side control unit 70 determines whether to restrict the display of an image based on position information or time information, thereby preventing information leakage from the image, protecting privacy, etc. Usability can be improved.

  In the present embodiment, the server-side control unit 70 restricts at least a part of the image when determining that the image display is restricted. For example, if the image is in a meeting, Restrict the display of the human face part of the image, or restrict the display of the character part of the image (characters written on the whiteboard or blackboard, characters displayed on the screen, etc.) Can do. This allows appropriate display restrictions (display restrictions such as hiding a confidential part to the extent that the contents of the image can be understood) even when displaying an image in which the part that should be restricted depends on position information and time information. Is possible.

  Further, in this embodiment, the server-side control unit 70 restricts at least a part of the image when determining from the position information that the location has changed from a place where security is ensured to a place where security is not ensured. I will give it. Therefore, when the user moves from a place where security is secured to a place where security is not secured, the user changes the image viewing method accordingly (for example, from a state where the image was browsed majestically to the surroundings The image can be browsed without changing to a state in which the image is browsed while taking care of

  Further, in the present embodiment, the server-side control unit 70 identifies a location to which display restriction is applied in the image based on at least one of the position information and the time information. Display restriction (display restriction such as hiding a confidential part to the extent that the contents of the image can be understood) becomes possible.

  In the present embodiment, the face recognition unit 53 inputs information on the subject in the image to the server-side control unit 70, and the server-side control unit 70 restricts display in the image based on the information. Specify where to add. Thereby, display restrictions can be added so as to hide a confidential person among the subjects. In addition, display restrictions can be added so as to hide confidential characters in the subject.

  In the present embodiment, the server-side control unit 70 displays the subject information input to the server-side control unit 70 by the face recognition unit 53 together with the image, so that the face is hidden from the image. Even when the restriction is applied, by displaying the subject information (for example, the name of a person), the user can easily grasp the person to whom the display restriction is applied.

  In the present embodiment, the server-side control unit 70 converts the subject information based on a predetermined rule, and displays it together with the image. As a result, for example, information (initial etc.) after conversion of the person name is displayed together with the image, so that the person name can be concealed.

  In the present embodiment, the server-side control unit 70 can determine whether to limit the output of the image based on the facial expression information of the person in the image. In other words, if the facial expression of the person in the image is hard, there is a high possibility of having confidentiality. Therefore, it is determined that display restrictions will be applied to the image, and if the facial expression of the person in the image is soft, it may have confidentiality. Therefore, it can be determined that display restrictions are not applied to the image. Thereby, it is possible to appropriately limit image display.

  In this embodiment, when at least one of voice data and text data is input from the communication unit 52, the server-side control unit 70 performs voice data and text data based on at least one of position information and time information. Limit the output of at least one of As a result, also in audio data and text data, security protection and privacy protection can be achieved by limiting the output of a confidential part (word) as in the case of images.

  In this case, in the present embodiment, when the server-side control unit 70 determines that there is a need to limit the output of the voice data, the text data converted / generated from the voice data is output, so that the output restriction of the voice data is necessary. Even in such a case, the contents of the voice data can be confirmed by text data.

  Further, in the present embodiment, when the server-side control unit 70 determines that output restriction is necessary for the voice data, at least one specific noun of the voice data and the text data is converted into another noun and is output. By converting a specific noun into a noun that can be understood by a specific person, the specific person can recognize the contents of voice data and text data, while protecting the security and privacy. .

  Moreover, in this embodiment, it has the 1st housing | casing 11 which has the display part 14, and the 2nd housing | casing 51 which has the server side control part 70, and the 1st housing | casing 11 and the 2nd housing | casing 51 are included. Since the first casing 11 side is a mobile terminal and the second casing 51 side is a server, the mobile terminal is provided with functions such as the server-side control unit 70. Thus, the mobile terminal can be reduced in size and weight.

  In the present embodiment, the first casing 11 is provided with a position detection unit 22 that detects position information of the first casing 11, and the communication unit 52 inputs a detection result by the position detection unit 22. Thereby, it is possible to perform image display restriction according to the position of the first housing 11.

  According to the present embodiment, the confidentiality level of the communication unit 52 to which information is input, the extraction unit 58 that extracts a predetermined keyword from the data input to the communication unit 52, and the keyword extracted by the extraction unit 58 is A classification unit 60 that classifies the keywords having a high level and a keyword having a confidentiality level of “medium”, converts a keyword having a confidentiality level of “high” by a predetermined conversion method, and has a confidentiality level of “medium”. A conversion unit 62 that converts the above-described keyword by a conversion method different from that for the keyword having the confidentiality level “high”. In this way, by classifying keywords according to the confidentiality level and performing different conversions according to the respective levels, it is possible to display data in consideration of the confidentiality level. Thereby, usability can be improved.

  In addition, even if the voice data does not contain a confidential keyword, the server-side control unit 70 can display the information on the display unit 14 of the portable terminal 10 when there is a problem of confidentiality or privacy in the image data. The display of image data may be prohibited or masked, and the reproduction of text data or audio data may be prohibited or restricted.

  Moreover, in this embodiment, since the communication part 52 which communicates with the portable terminal 10 transmits the result converted by the conversion part 62 to the portable terminal 10, even if it does not process a data in the portable terminal 10, It is possible to display data that considers the level of confidentiality.

  In the present embodiment, the text data generation unit 54 that generates text data from voice data is provided, and the extraction unit 58 extracts keywords from the text data generated by the text data generation unit 54. Keywords can be easily extracted.

  Further, in the present embodiment, since the keywords are initially converted, each keyword can be easily converted without creating a special conversion table for each keyword. Also, if the keyword is a full name, if the confidentiality level is “High”, both the full name and the name are converted to initials, and if the confidentiality level is “Medium”, either the full name or the first name is converted to the initial. Therefore, display according to the confidentiality level can be performed. Further, when the keyword is a place name, if the confidentiality level is “high”, it is converted into information of a predetermined area (position information within a wide range), and if the confidentiality level is “medium”, Since the information is converted into information of an area narrower than the area (position information within a narrow range), display according to the confidentiality level can also be performed from this point.

  In the present embodiment, the position detection unit 22 that detects position information, the input unit 12 that performs input, the display unit 14 that displays information related to input, and the position detected by the position detection unit 22 are used. A terminal-side control unit 30 that restricts display on the display unit 14. In this way, by limiting the display according to the position, it is possible to perform display in consideration of security, and as a result, it is possible to improve usability.

  Moreover, in this embodiment, when the terminal side control part 30 judges that security is not maintained based on the output of the position detection part 22, since the display on the display part 14 is restrict | limited, security is appropriately set. Display restrictions can be made in consideration. Further, in this embodiment, when it is determined that security is maintained based on the output of the position detection unit 22, at least a part of the display restriction on the display unit 14 is released. It is possible to perform display restriction that appropriately considers the above.

  In addition, the personal assistant system 100 according to the present embodiment performs the display restriction in consideration of security as described above, and the server 50 that applies the display restriction to at least part of the data input from the portable terminal 10. Therefore, the mobile terminal 10 can display data with display restrictions on the display unit 14 of the portable terminal 10 without applying display restrictions to at least a part of the data. . As a result, the processing burden on the portable terminal 10 can be reduced, and as a result, the portable terminal 10 can be simplified, reduced in size and weight, and the like.

  Moreover, in this embodiment, the display part 14 which displays text data, the audio | voice input part 42 which inputs an audio | voice, and the terminal side control which displays the information relevant to the said audio | voice on the said display part according to the analysis result of an audio | voice. Therefore, when a person utters a voice such as “Good morning” as in step S100 of FIG. 4, information about the person (name and other information is registered). Information or tasks to be performed on the person) can be displayed on the display unit 14. Thereby, even if the user has forgotten the person who has spoken, the user can remember the person by looking at the display unit 14. Thus, according to the present embodiment, it is possible to provide a personal assistant system 100 and a portable terminal 10 that are easy to use. In this case, an appropriate display can be performed by using the analysis result of the voice print analysis unit 55 that analyzes the voice print.

  Further, in the present embodiment, the terminal-side control unit 30 and the server-side control unit 70 determine whether a predetermined word (for example, “that case” or “Hokkaido case” included in the voice data according to the analysis result of the voice print analysis unit 55. ) Etc.) is displayed on the display unit 14, so even if an ambiguous question such as “That case” or “Hokkaido case” is made, by confirming the display unit 14, Can be remembered. In this embodiment, it can be said that the easy-to-use personal assistant system 100 and the portable terminal 10 can be provided also from this point. In the present embodiment, information related to a predetermined word (such as “Hokkaido”) is selected according to the frequency of input to the input unit together with the predetermined word (for example, “Hokkaido”), Since it displays on the display part 14 (step S104 of FIG. 4), appropriate information display becomes possible.

  Further, in step S104 in FIG. 4, information corresponding to the position when the voice data is input is displayed on the display unit 14, so that appropriate information display can be performed from this point.

  Further, in step S104 in FIG. 4, information corresponding to the time when the voice data is input (such as information input within a predetermined time from the time when the voice data is input) is displayed on the display unit 14. Therefore, appropriate information can be displayed from this point.

  In the present embodiment, the voice input unit 42 that inputs voice, the text data generation unit 54 that generates text data based on the voice data input to the voice input unit 42, and the voice data input to the voice input unit 42 A voice print analysis unit 55 for analyzing the voice print data of the voice data, and a deletion unit 76 for deleting the voice data after the text data is generated by the text data generation unit 54 according to the analysis result by the voice print analysis unit 55. . Thereby, the storage capacity required for the flash memory 64 can be reduced by erasing the voice data after the text data is generated. In the present embodiment, since the voice data is erased according to the analysis result by the voiceprint analysis unit 55, it is possible to realize a good usability in consideration of privacy by erasing the voice data of a specific person. Become.

  In this embodiment, the confidentiality level of the communication unit 52 to which information is input, the extraction unit 58 that extracts a predetermined keyword from data input to the communication unit 52, and the keyword extracted by the extraction unit 58 is “ A classification unit 60 that classifies the keywords having a high level and a keyword having a confidentiality level of “medium”, converts a keyword having a confidentiality level of “high” by a predetermined conversion method, and also converts the keyword having a confidentiality level of “medium”. A conversion unit 62 that converts the keyword by a conversion method different from that of the keyword having a confidentiality level of “high”. In this way, by classifying keywords according to the confidentiality level and performing different conversions according to the respective levels, it is possible to display data in consideration of the confidentiality level.

  In this embodiment, the voiceprint analysis unit 55 analyzes whether or not the voiceprint data of the voice data is the registered user's voiceprint data, and the erasure unit 76 erases the voice other than the user. It is possible to effectively reduce the storage capacity of the flash memory 64 and to increase privacy considerations.

  Further, in the present embodiment, the erasure unit 76 uses different times for analysis and erasure between the user's voice and the non-user's voice (steps S276 and S278). Thereby, the user's voice is also erased after a predetermined time, so that the storage capacity can be further reduced.

  In the present embodiment, when the text data generation unit 54 cannot generate text data from the voice data, the warning unit 18 issues a warning, so that the user recognizes that the text data could not be generated from the voice data. be able to. In addition, when the text data generation unit 54 cannot generate the text data from the voice data (when step S270 is negative), the playback unit 16 plays back the voice data according to the user's instruction. The contents that could not be made into text data can be confirmed by reproducing the voice data.

  Further, according to the present embodiment, the display unit 14 that performs display, the audio input unit 42 that inputs sound, and the weighting unit 56 that performs weighting based on at least one of the magnitude, frequency, and meaning of the input sound. And control units 70 and 30 for changing the display mode of the task on the display unit based on the voice input by the voice input unit 42 and the weighting of the weighting unit 56. As a result, the display mode of the task in the display unit 14 is changed based on the weighting performed by the weighting unit 56 in accordance with the input method of the audio data, the content of the audio data, and the like. The display mode according to the can be realized. As a result, usability can be improved.

  Further, according to the present embodiment, the weighting unit 56 uses at least the frequency of the audio data to identify the person who has emitted the sound, and performs weighting according to the person (position in this embodiment). Thus, it is possible to perform appropriate weighting regarding the importance of the audio data.

  In addition, according to the present embodiment, the weighting unit 56 performs weighting according to confidentiality based on the meaning of the voice. From this point as well, appropriate weighting regarding the importance of the voice data can be performed. .

  In the present embodiment, when date information is included in the voice input from the voice input unit 42, the task can be displayed based on the date information. Can also be satisfied. In the present embodiment, the task is displayed in consideration of information related to the time detected by the time detection unit 24 or in consideration of date information in the calendar unit 26. Tasks to be performed can be displayed in an order such as in order of distance from the time.

  In this embodiment, since the text data generation unit 54 that converts the voice input from the voice input unit 42 into text data is provided, the weighting unit 56 can weight the text data. As a result, weighting can be performed more simply than when audio data is handled.

  In this embodiment, since the display order, color, display size, display font, and the like are changed based on the weighting result, the weighting result can be expressed by various methods.

  In the present embodiment, the display mode on the display unit is changed according to the output of the position detection unit 22 that detects the position, that is, it is determined that the task has been executed based on the current position. In addition, since the task is not displayed (deleted), the storage capacity can be reduced.

  Further, in the present embodiment, it is determined whether or not the task is a task based on whether or not the fixed word is included in the audio data, and whether or not to display on the display unit 14 is determined using the determination result. Since it is determined, it is possible to automatically determine whether or not it is a task, and it is also possible to automatically determine whether or not to display on the display unit.

  In the present embodiment, since the setting unit 74 is provided in the server 50 in order to allow the user to set the weighting, the user can set the weighting according to his / her preference. Is possible.

  Further, according to the present embodiment, the voice input unit 42 for inputting voice, the text data generation unit 54 for converting the input voice to text data, and the text data when the voice input unit 42 inputs a specific frequency. And a server-side control unit 70 that starts conversion by the generation unit 54, that is, starts recording and starts conversion into text data. Therefore, when a certain person utters and a sound of a specific frequency is input, recording and conversion to text data are started based on the input of the sound (see FIG. 2A). Now you can start recording and converting to text data. This simplifies the user's operation and improves usability.

  Further, in this embodiment, since the voice input unit 42 can start conversion to text data when a frequency related to the telephone is input, for example, the voice of the telephone from the time when the telephone ringtone is generated. Can be recorded and converted to text data. Thereby, recording and conversion into text data can be performed without leaking the telephone conversation.

  Further, in this embodiment, recording and conversion to text data can be started at an appropriate timing based on the task, for example, when the date and time of the meeting is reached. It is possible to simplify and improve usability. In addition, for example, recording or conversion to text data can be performed according to the end time of the conference (see FIG. 2C). Data recording and conversion to text data can be automatically started.

  Further, in the present embodiment, recording and conversion to text data can be started at an appropriate timing based on the user's biological information (see FIG. 2D). Can be simplified, and usability can be improved.

  Furthermore, in the present embodiment, recording and conversion to text data can be started when the current time reaches a predetermined time (see FIG. 2B). Thus, the user's operation can be simplified and the usability can be improved.

  Further, in the present embodiment, conversion by the text data generation unit 54 can be prohibited according to the detection result of the position detection unit 22, so that there is a problem in recording such as an outside meeting, for example. In addition, recording can be automatically prohibited. Thereby, usability can be further improved.

  In the above embodiment, the case where the level of confidentiality is determined for each word has been described. However, the present invention is not limited to this. For example, in the classification unit 60, a word used in a business has a high level of confidentiality, Private words may be classified as words with a low level of confidentiality.

  In the above embodiment, when the current position detected by the position detection unit 22 of the portable terminal 10 is a position where security is not maintained, the keyword is converted and displayed, that is, the display unit 14. Although the case where the display in is restricted has been described, the present invention is not limited to this. For example, when the time detected by the time detection unit 24 is a predetermined time (for example, within working hours), the display on the display unit 14 may be limited. Even in this case, it is possible to perform display in consideration of security as in the above embodiment. In the case of performing such control, instead of acquiring the current position of the user in step S206 of FIG. 19, instead of acquiring the current time and determining whether or not it is a place where security can be ensured in step S208. In addition, it may be determined whether it is a time when security can be ensured.

  In the above embodiment, whether or not the voice data is a task is determined based on the presence / absence of date / time information and the type of ending of the voice data. However, the present invention is not limited to this. Based on the above, task determination may be performed.

  In the above-described embodiment, the case where the word of the confidentiality level “high” and the word of “medium” are converted into the initial, which is a superordinate concept thereof, is not limited to this. For example, a converted word for each word may be defined in the keyword DB. In this case, for example, as a word after conversion for the keyword “camera”, it is possible to define “precise equipment” that is a superordinate concept of the camera, “imaging device” that is a subordinate concept. In this case, if “camera” is confidentiality level “high”, it is converted to “precision equipment”, and if “camera” is confidentiality level “medium”, it may be converted to “photographing device”. it can. In this way, display in consideration of the security level can be performed by converting the word into a higher concept word and a middle concept word according to the confidentiality level. Further, when money amount information such as a budget is registered in the keyword DB, a value expressed by the number of digits that is a superordinate concept of the money amount information may be defined.

  In the above embodiment, the case where the voice is Japanese has been described, but a foreign language such as English may be used. In a foreign language (for example, English), it may be determined whether the task is based on the presence or absence of a predetermined word or the presence or absence of a predetermined syntax.

  In the above embodiment, the case where the flash memory 28 is mounted has been described in order to reduce the size and weight of the portable terminal 10. However, in addition to or instead of this, a storage device such as a hard disk is used as the portable terminal 10. It is good also as mounting in.

  In the above-described embodiment, the case where the portable terminal 10 is connected to an external PC and the setting is performed on the external PC when setting the company position or the like has been described. However, the present invention is not limited to this. For example, the company position may be registered in advance in the hard disk 66 of the server 50 and the company position may be downloaded from the hard disk 66. In addition, for example, by installing an application for setting a company position or the like in the portable terminal 10, the company position or the like may be set on the portable terminal 10.

  In the above embodiment, the task priority is calculated based on the above formula (1). However, the present invention is not limited to this, and the task priority may be calculated using another formula. For example, each weight may be added or simply multiplied. In addition, the task priority may be determined using the above formula (1) or the like, and any of the weights may be selected and the task priority may be determined in descending order of the selected weights. In this case, the user may be allowed to set which weight determines the task priority.

  In the above-described embodiment, the initialized keyword out of the keyword initialized (for example, “SW” in software) and the image based (for example, “sponge” in software) is displayed first. Although described, the present invention is not limited to this, and an image-based one may be displayed first. In addition, the initial version and the image-based version may be displayed at the same time.

  In the above embodiment, the case where the name and information of the speaker is displayed when the voice of a person other than the user is input to the input unit 12 is described. However, the present invention is not limited to this. You may make it display the image regarding a speaker, such as a face photograph. In this case, for example, it is necessary to store the images in the hard disk 66 of the server 50 and register the images in the information item of the keyword DB.

  In the above embodiment, closeness with the user may be used as the weight. In this case, for example, a person who receives a relatively large amount of voice or a person who has a chance to approach among people who have a portable terminal can be set as a highly intimate person.

  The configuration described in the above embodiment is an example. That is, at least a part of the configuration of the server 50 described in the above embodiment may be provided on the portable terminal 10 side, or at least a part of the configuration of the portable terminal 10 described in the above embodiment may be provided on the server 50 side. It is good also as providing in. Specifically, for example, the voice terminal analysis unit 55 and the text data generation unit 54 of the server 50 may be provided in the portable terminal 10.

  In the above-described embodiment, the case where the present invention is used for business is mainly described. However, it may be used privately, or may be used both privately and for business.

  The above-described embodiment is an example of a preferred embodiment of the present invention. However, the present invention is not limited to this, and various modifications can be made without departing from the scope of the present invention.

  DESCRIPTION OF SYMBOLS 10 ... Portable terminal, 12 ... Input part, 14 ... Display part, 16 ... Reproduction part, 18 ... Warning part, 20 ... Biometric information input part, 22 ... Position detection part, 24 ... Time detection part, 26 ... Calendar part, 28 ... Flash memory, 30 ... Terminal side control unit, 32 ... Communication unit, 50 ... Server, 52 ... Communication unit, 53 ... Face recognition unit, 54 ... Text data generation unit, 55 ... Voiceprint analysis unit, 56 ... Weighting unit, 58 ... Extraction unit, 60 ... Classification unit, 62 ... Conversion unit, 64 ... Flash memory, 66 ... Hard disk, 70 ... Server side control unit, 72 ... Change unit, 74 ... Setting unit.

Claims (10)

An information processing apparatus for processing an image to be displayed on a display device,
A first input unit for inputting first image data;
A second input unit for inputting position information of the display device ;
Based on the position information of the display device input by the second input unit , the first image data and the second image data obtained by limiting the first image data are output. And an information processing apparatus.   The information processing apparatus according to claim 1,
  A third input unit for inputting information when the first image data input to the first input unit is generated;
  A second processing unit that generates the second image data in which the first image data is limited based on information input by the third input unit,
  The information processing apparatus, wherein the first processing unit outputs either the first image data or the second image data based on position information of the display device input by the second input unit. The information processing apparatus according to claim 2 ,
The information processing apparatus, wherein the first processing unit converts information input by the third input unit based on a predetermined rule. The information processing apparatus according to claim 2 or 3 ,
The third input unit inputs at least one of position information and time information,
An information processing apparatus comprising: a determination unit that determines whether to limit the output of the first image data based on information input by the third input unit when an output instruction related to the first image data is given . The information processing apparatus according to claim 4 ,
The third input unit includes:
Information on the position at which the first image data was photographed, information on the position at which the first image data was output, information on the time at which the first image data was photographed, and information on the time at which the first image data was output Information processing apparatus for inputting at least one of The information processing apparatus according to claim 4 or 5 ,
The first processing unit is an information processing apparatus that specifies a location to which output restriction is applied in the first image data based on at least one of the position information and the time information. In the information processing apparatus according to any one of claims 4 to 6 ,
A fourth input unit for inputting at least one of voice data and text data;
The information processing apparatus that determines whether or not to limit the output of at least one of the voice data and the text data based on at least one of the position information and the time information. The information processing apparatus according to claim 7 ,
An information processing apparatus comprising: a third processing unit that outputs text data converted and generated from the audio data when the determination unit determines that the output of the audio data needs to be restricted. The information processing apparatus according to claim 7 ,
A third processing unit for converting and outputting at least one specific noun of the voice data and the text data to another noun when the judgment unit judges that output restriction is required for the voice data; Information processing device. In a computer that processes images to be displayed on a display device,
A first input process for inputting first image data;
A second input process for inputting positional information of the display device ;
Output processing for outputting one of the first image data and the second image data in which the first image data is limited based on the position information of the display device input in the second input processing When,
A program that executes

Images