JP2020003892A - Writing conversion device, conversion display device, conversion display system, control method, and recording medium - Google Patents

Abstract

To provide a writing conversion device capable of achieving communication among users having affinity with different writing methods.SOLUTION: A translation display device comprises: a display control unit 110 determining whether or not a plurality of viewpoints are present in a first display area among the respective viewpoints on a display surface for a plurality of users who gaze at the display surface and determining a language having affinity with the plurality of users who gives the viewpoints into the first display area as a second language except for the first language only when the plurality of viewpoint of exists in the first display area; and a translation unit 111 translating text data written in the first display area into translated text data in the second language. The display control unit 110 generates position information indicating the position of the second display area for displaying the translated text data, and outputs the translated text data and the information to a display unit 102.SELECTED DRAWING: Figure 3

Description

  The present invention, Japanese, language notation such as English, dialect notation such as standard and dialect, kanji, katakana, relates to notation conversion technology that can be suitably used in notation methods including character display such as hiragana, The present invention relates to a technique for converting data represented by one notation into a notation represented by another notation.

  2. Description of the Related Art In an image forming apparatus such as a printer, a copying machine, or a multifunction peripheral, an operation panel that receives various settings is provided. The image forming apparatus disclosed in Patent Document 1 receives personal information data including a user's nationality from a mobile terminal by short-range wireless communication, determines an official language of the user based on the received nationality, and changes a display language. Set. Thus, the user can operate the image forming apparatus by using the operation panel displayed using a language that the user can understand.

  However, assuming that a plurality of users using different languages try to operate the image forming apparatus together, the display language of the operation panel is set by the first language used by the first user. If the first language is used, the first user can understand the display content of the operation panel and can operate the image forming apparatus, but the first language is used for the second user who uses the second language. In such a case, the contents displayed on the operation panel cannot be understood, and the operation of the image forming apparatus becomes difficult.

  In order to solve this problem, according to Patent Document 2, a head mounted type video display device of a transmission type is mounted, and a user is displayed on the video display device so as to overlap the scenery seen through the video display device. You can see the video that you have. When the user looks at the English character string through this video display device, the character string is specified in the attention area, and the video display device determines the translation destination language to Japanese based on the user information, The English character string in the attention area is automatically translated into Japanese, and the Japanese character string is displayed so as to overlap the English character string in the attention area in the view before translation.

  For example, when a first user who understands English views a Japanese character string through a head-mounted image display device, a character string obtained by translating the Japanese character string into English is displayed on the image display device. . Similarly, when a second user who understands Spanish reads the above Japanese character string through the same video display device, a character string obtained by translating the Japanese character string into Spanish is displayed on the video display device. You. For this reason, the first user and the second user can know the meaning of the Japanese character string even if they cannot understand Japanese.

  In this way, the problem of Patent Document 1 described above when a plurality of users using different languages try to operate the apparatus jointly is solved.

JP-A-2017-211853 JP 2014-93050 A

  However, according to the technology disclosed in Patent Document 2, the first user does not know where the second user is paying attention, and the second user is not aware where the first user is paying attention. I don't know. As described above, since the points of interest of the other party are not known, when a plurality of users work together, this is one factor that hinders communication between the plurality of users.

  In the above, problems in linguistic expressions such as Japanese, English and Spanish were pointed out, but similar problems occur in expressions such as standard and dialects, kanji, katakana and hiragana. Exists.

  The present invention has been made in view of the above points, and in the case where a plurality of users having affinity for different notations work jointly, a notation conversion device, a conversion display device, which can achieve communication between users, It is an object to provide a conversion display system, a control method, and a recording medium.

  In order to achieve the above object, one embodiment of the present invention is a notation conversion device that converts first data written in a first notation in a first display area on a display surface into a notation in a second notation. Viewpoint determination means for determining whether or not a plurality of viewpoints are present in the first display region among respective viewpoints on the display surface of a plurality of users gazing at the display surface; and the first display region. Only when there are a plurality of viewpoints within, except for the first notation, determining means that determines a notation compatible with a plurality of users who pour eyes at the first display area as a second notation. And a notation conversion unit for converting the first data written in the first display area into second data written in the second notation, and a position of the second display area on the display surface. Generating means for generating position information indicating For displaying the second data in the second display area, the second data and the location information, and wherein further comprising an output means for outputting to an external display device.

  Here, the first notation may be an expression in a first language, and the second notation may be an expression in a second language.

  Here, the first notation is an expression in kanji, the second notation is an expression in katakana, hiragana or romaji, or the first notation is an expression in katakana, hiragana or romaji. Yes, the second notation may be an expression using Chinese characters.

  Here, the first notation is an expression in a standard language used nationwide as a norm, and the second notation is an expression in a dialect that has developed differently depending on the region, or the first notation May be an expression in a dialect, and the second notation may be an expression in a standard language.

  Here, the first notation may be expressed by a symbol mark that symbolizes a predetermined idea, and the second notation may be expressed by a character that expresses the idea symbolized by the symbol mark. .

  Here, the first notation may be based on a simple expression, and the second notation may be based on an expression that explains the simple expression in detail.

  Here, the expression for explaining the expression in detail may be a character or a graphic including the character.

  Here, the affinity notation determined by the determination means may be a notation registered in advance as a notation familiar to each user.

  Here, when there are a plurality of notations that have an affinity for the plurality of users, the determination unit determines one notation that has an affinity for the most users among the plurality of users, It may be determined as the second notation.

  Here, when the plurality of notations having affinity with the plurality of users are determined as the second notation by the determining unit, the notation conversion unit determines the first data, respectively. May be converted by a plurality of notations.

  Here, the second display area is divided into a plurality of partial areas each displaying the converted first data by the plurality of notations, and the generation unit further includes the plurality of partial areas. The output unit may further generate the partial area information to be displayed, and the output unit may further output the partial area information.

  Here, the output unit, in the second display area, for each of the plurality of the converted first data, for switching to display, a plurality of the converted first data, sequentially output, It may be.

  Here, the generation unit generates the position information indicating the position of the first display area, and the output unit displays the second data in the first display area as the second display area. The position information may be output.

  Here, furthermore, there is provided a blank judging means for judging whether or not there is a blank area larger than the size of the second data and in which no data is displayed in an area around the first display area, When it is determined that the second display area exists, the generation unit may generate the position information indicating the position of the second display area in the blank area.

  Here, the viewpoint determination means may further include, among the respective viewpoints on the display surface of the plurality of users gazing at the display surface, the number of the viewpoints present in the third display area is equal to or greater than a predetermined number. And determining whether or not, if the number is greater than or equal to a predetermined number, the generating means generates expanded range information indicating a range of an expanded area obtained by expanding the third display area, and the output means further includes: The enlarged range information may be output to the display device in order to enlarge and display the third display area.

  Here, on the display surface, the second display area exists within a predetermined distance from the first display area, and the second display area exists in contact with the first display area, and the first display area The second display area may be present in an overlapping manner, or the second display area may be included in the first display area.

  Here, the generation unit further generates the same color pattern information indicating the display color or display pattern in the first display area and the display color or display pattern in the second display area, and The means may further output the color pattern information to the display device for displaying using the display color or the display pattern.

  Here, the apparatus may further include a detection unit that detects a viewpoint on the display surface of a user who gazes at the display surface.

  In this case, the detecting means extracts a standing position of the user and a viewing direction of the user from the image data, and a camera for acquiring image data of the user as a subject, and the extracted standing position And a calculation unit that calculates a viewpoint on the display surface of the user who gazes at the display surface based on the gaze direction.

  Here, the display device is provided with the rectangular display surface facing upward, the user stands around the display device, and the generation unit further configures the second data. The image data generated by rotating the second data is generated so that the character to be displayed looks upright from the user's standing position, and the output unit outputs the image data as the second data. And the display device may display the image data in the second display area.

  Here, the information processing apparatus further includes a storage unit that stores notation information indicating a notation system compatible with each of the plurality of users, and the determining unit stores the notation information from the storage unit. Information may be acquired, and a notation that is compatible with the plurality of users may be determined using a notation indicated by the acquired notation information.

  Here, the viewpoint determination unit is attached to the user's face, acquires the viewpoint from glasses outside the notation conversion device including a camera and a calculation unit, and the camera captures an eye direction of the user. And generating the image data, and the calculation unit may calculate a position of an object appearing in a central portion of the image data as a viewpoint.

  Here, the viewpoint determination unit is mounted on the user's face and acquires the viewpoint from the glasses-type display device equipped with a camera, and the display device captures the user's line of sight with the camera. Then, image data may be generated, and the position of an object appearing at the center of the generated image data may be calculated as a viewpoint.

  Further, another aspect of the present invention is a conversion display device that converts the first data written in the first notation into the first notation in the first display area on the display surface and converts the first data into the notation in the second notation, and displays the converted data. Display means for displaying the first data in the first display area, viewpoint acquisition means for acquiring respective viewpoints on the display surface of a plurality of users who gaze at the display surface, among the acquired viewpoints, Determining means for determining whether a plurality of viewpoints are present in the first display area, and only when the plurality of viewpoints are present in the first display area, excluding the first notation, the first display Determining means for determining a notation compatible with a plurality of users who pour eyes at the area as the second notation, and the first data written in the first display area, the second notation Notation converter that converts to the second data notation And a generating means for generating position information indicating a position of a second display area on the display surface, and an output means for outputting the second data and the position information to the display means, The means receives the second data and the position information, and displays the second data in the second display area.

  Further, another aspect of the present invention is a conversion display system that converts and displays the first data written in the first notation in the first display area on the display surface into the notation in the second notation, It has a notation conversion device, a display device, and a detection device, wherein the detection device is a pair of glasses that is mounted on the face of the user and includes a camera and a calculation unit, and the camera captures the gaze direction of the user. Acquisition of image data, the calculation unit calculates a position of an object appearing in a central portion of the image data as a viewpoint, and the notation conversion device acquires from the detection device, and uses a plurality of methods of gazing at the display surface. Viewpoint determination means for determining whether or not a plurality of viewpoints are present in the first display area among respective viewpoints on the display surface of the user, and only when a plurality of viewpoints exist in the first display area. , Except for the first notation A notation method that is compatible with a plurality of users who pour eyes at the first display area, a determination unit that determines the second notation method, and the first data written in the first display area, Notation conversion means for converting into second data written in the second notation, generating means for generating position information indicating a position of a second display area on the display surface, and the second data and the position Output means for outputting information to the display device, wherein the display device has the display surface, displays the first data in the first display area of the display surface, the second data And receiving the position information and displaying the second data in the second display area.

  Further, another aspect of the present invention is a conversion display system that converts and displays the first data written in the first notation in the first display area on the display surface into the notation in the second notation, It is provided with a notation conversion device, a bulletin board, and a detection display device, wherein the bulletin board has the display surface, and the detection display device is a glasses-type display device equipped with a camera, which is mounted on a user's face, The camera obtains image data by capturing the direction of the user's line of sight, calculates the position of an object in the center of the obtained image data as a viewpoint, displays the image data, and displays the image data. A viewpoint determination that determines whether a plurality of viewpoints are obtained from the detection display device and a plurality of viewpoints are present in the first display area among respective viewpoints on the display surface of a plurality of users who gaze at the display surface. Means Only when there are a plurality of viewpoints in the first display area, except for the first notation, a notation compatible with a plurality of users who pour eyes at the first display area, a second notation. Determining means to determine as, the first data written in the first display area, the notation conversion means to convert to the second data written by the second notation, on the display surface, Generating means for generating position information indicating the position of the second display area, and output means for outputting the second data and the position information to the detection display device, wherein the detection display device is the second data And receiving the position information, and displaying the second data in the second display area indicated by the position information.

  Another aspect of the present invention is a control method used in a notation conversion device that converts first data written in a first notation in a first display area on a display surface into a notation in a second notation. A viewpoint determining step of determining whether a plurality of viewpoints are present in the first display area among respective viewpoints on the display surface of a plurality of users who gaze at the display surface; and Only when there are a plurality of viewpoints in the area, except for the first notation, a notation that is compatible with a plurality of users who pour eyes at the first display area is determined as the second notation. And a notation conversion step of converting the first data written in the first display area into second data written in the second notation, and, on the display surface, Location information indicating the location A generation step of forming, for displaying the second data to the second display area, the second data and the location information, and wherein the and an output step of outputting to an external display device.

  Another aspect of the present invention is a control computer used in a notation conversion device that converts first data written in a first notation in a first display area on a display surface into a notation in a second notation. A computer-readable recording medium that records a program, wherein the notation conversion device that is a computer has a plurality of viewpoints among the viewpoints on the display surface of a plurality of users who gaze at the display surface. A viewpoint determining step of determining whether or not the first display area exists, and only when there are a plurality of viewpoints in the first display area, except for the first notation, a line of sight to the first display area. Determining a notation that has an affinity for a plurality of users who pours, as a second notation, and the first data described in the first display area, the second notation A notation conversion step of converting the data into second data, a generation step of generating position information indicating a position of a second display area on the display surface, and displaying the second data in the second display area. And outputting an output step of outputting the second data and the position information to an external display device.

  According to the above aspect, only when there are a plurality of viewpoints of the user in the first display area in which the first data is displayed by the first notation, the first display area is compatible with a plurality of users who look at these viewpoints. The first data is converted according to the second notation (excluding the first notation), and the second data is displayed in the second display area.

  By displaying the second data in the second display area in this manner, it can be understood that a plurality of users are paying attention to the first display area. In other words, one user who is paying attention to the first display area has a user who is paying attention to the first display area other than his / her own, and that user, like his / her own, is in the first display area. It is possible to know that the user is interested in the first data displayed in the area. As a result, communication can be achieved between a plurality of users. Also, since the second data is displayed in the second display area, even a user who does not have an affinity for the first notation can know the meaning of the first data displayed in the first display area. Can be.

1 shows an external perspective view of a translation display device 1 according to an embodiment. The horizontal view angles 321 to 324 of the cameras 12a to 12d included in the translation display device 1 are shown. FIG. 2 is a block diagram illustrating an internal configuration of the translation display device 1. 5 shows an example of the data structure of a display data table 171. 4 shows a relationship between text data displayed on the display unit 102 of the translation display device 1 and a user's standing position. 13 shows an example of a data structure of a user attribute table 191 and an example of face image data. 1 shows an example of image data. 4 shows an example of a data structure of an object information table 121 and an example of object data. 2 shows an example of a data structure of a parts information table 131 and an example of parts data. 4 shows an example of a data structure of a user position table 141. 4 shows an example of the data structure of a horizontal direction table 156. 4 shows an example of the data structure of a line-of-sight information table 151. 4 shows an example of the data structure of a viewpoint position table 161. 9 shows an example of the data structure of a display area table 181. 4 shows an example of the data structure of a translation text data table 201. (A) shows a schematic view of the user's head facing in three directions. (B) Indicates the horizontal orientation of the user's head. (C) A schematic diagram of a user's head having three inclinations with respect to the vertical direction is shown. (D) The inclination of the user's head from the vertical direction is shown. 2 shows an example of text data and translated text data displayed on the display unit 102 of the translation display device 1. (A) An example in which the first display area 351a and the second display area 352a are in contact with each other is shown. (B) An example in which the second display area 352b exists within a predetermined distance from the first display area 351b. (C) An example in which the second display area 352c exists so as to overlap the first display area 351c. (D) An example in which the second display area 352d is included in the first display area 351d is shown. (E) An example in which the hatching pattern applied to the background in the first display area 351e matches the hatching pattern applied to the background in the second display area 352e. (F) An example in which the dot pattern applied to the background in the first display area 351f matches the dot pattern applied to the background in the second display area 352f. (G) An example in which the first display area 351g and the second display area 352g are linked by a line image 353. (H) An example in which the display pattern of the frame of the first display area 351h matches the display pattern of the frame of the second display area 352h. 4 is a flowchart illustrating an operation of the translation display device 1. It continues to FIG. 4 is a flowchart illustrating an operation of the translation display device 1. Continued from FIG. (A) The relationship between the text data displayed on the display unit 102 of the translation display device 1 as a modification (1) and the user's standing position is shown. (B) shows an example of text data and translated text data which are displayed on the display unit 102 of the translation display device 1 in a large scale. The relationship between the text data displayed on the display unit 102 of the translation display device 1 and the user's standing position as a modified example (2), and an example of the text data and translated text data displayed on the display unit 102 are shown. 14 shows a schematic configuration of a translation display system 501 as a modification (3). (A) A schematic configuration of a translation display system 531 as a modification (4) is shown. (B) An example of an image displayed on the display unit 533b of the glasses 533 is shown.

1. Embodiment A translation display device 1 according to one embodiment of the present invention will be described.

1.1 Translation display device 1
FIG. 1 is an external perspective view of the translation display device 1.

  The translation display device 1 is a table-type display device as shown in FIG. A rectangular display unit 102 is incorporated in the top plate of the translation display device 1, and cameras 12a, 12b, 12c, and 12d are installed at four corners of the top plate, respectively. The main body of the translation display device 1 is supported by legs 11. Scales 15a, 15b, 15c, and 15d are provided at equal intervals, for example, at intervals of 5 cm, around the four sides of the top plate.

  The display unit 102 is provided with a rectangular display surface facing upward. The user stands around the translation display device 1 including the display unit 102.

  In this specification, the origin is set at one vertex 102a of the rectangular display unit 102, the X axis and the Y axis are set along the short side 102b and the long side 102c of the rectangular display unit 102, and Z is set in the vertical direction. Set the axis. In addition, the space in which the translation display device 1 exists is specified by XYZ coordinates, and the display surface of the display unit 102 is specified by XY coordinates.

  FIG. 2 shows horizontal angles of view 321 to 324 of the cameras 12a to 12d.

  The camera 12a is installed such that its optical axis is directed upward by about 22.5 degrees from a horizontal plane toward a diagonal corner, and its shooting direction is fixed. The horizontal angle of view 321 of the camera 12a is, for example, approximately 90 degrees, and the vertical angle of view is, for example, approximately 45 degrees. The cameras 12b, 12c and 12d are the same as the camera 12a.

  As shown in FIG. 2, the photographing range of the cameras 12 a to 12 d almost completely covers the periphery of the translation display device 1. Therefore, the users standing around the translation display device 1 by the cameras 12 a to 12 d, particularly, the user Face can be taken.

1.2 Internal Configuration of Translation Display Apparatus 1 FIG. 3 is a block diagram showing the internal configuration of translation display apparatus 1.

  As shown in this figure, the translation display device 1 includes cameras 12a to 12d, a main control unit 101, a display unit 102, an output unit 103, a storage unit 104, input units 105a to 105d, a face recognition processing unit 106, a position calculation unit. 107, a gaze viewpoint calculation unit 108, a display area detection unit 109, a display control unit 110, a translation unit 111, and a synthesis unit 229.

  Here, the main control unit 101, a part of the storage unit 104, the display control unit 110, and the translation unit 111 convert the data written in the first language (first notation) into one display area on the display surface. It constitutes a translation device (notation conversion device) that translates (converts) into notation in two languages (second notation). Here, the first notation is an expression in a first language, and the second notation is an expression in a second language. The cameras 12a to 12d, the main control unit 101, a part of the storage unit 104, the input units 105a to 105d, the face recognition processing unit 106, the position calculation unit 107, the line-of-sight viewpoint calculation unit 108, and the display area detection unit 109 A detection unit configured to detect a viewpoint on a display surface of a user who gazes at the surface is configured. Each of the cameras 12a to 12d acquires image data of a user as a subject. The face recognition processing unit 106, the position calculation unit 107, the line-of-sight viewpoint calculation unit 108, and the display area detection unit 109 extract the user's standing position and the user's line-of-sight direction from the image data, and based on the extracted standing position and line-of-sight direction. Thus, the calculation unit that calculates the viewpoint on the display surface of the user who gazes at the display surface is configured.

  The translation display device 1 includes a memory that stores a computer program for control and a processor that operates according to the computer program. The main control unit 101, the face recognition processing unit 106, the position calculation unit 107, the line-of-sight viewpoint calculation unit 108, the display area detection unit 109, the display control unit 110, and the translation unit 111 are each operated by a processor according to a computer program. Perform its function.

(1) Storage unit 104
The storage unit 104 includes, for example, a hard disk.

  The storage unit 104 stores the image data 111a, 111b, 111c, 111d, the object information table 121, the object data 128,..., The parts information table 131, the parts data 137a, 137b, 137c,. Eye information table 151, horizontal direction table 156, viewpoint position table 161, display data table 171, display area table 181, user attribute table 191, face image data 197a, 197b, 197c, ..., translation text data table 201, dictionary It has an area for storing a database 211, display image data (FIG. 5), and other data (not shown).

(Display data table 171)
FIG. 4 shows an example of the data structure of the display data table 171.

  The display data table 171 is a data table for storing text data (first data) displayed by the display unit 102. The display data table 171 includes a plurality of pieces of display data related information 172 in advance as shown in FIG.

  Each display data related information 172 includes a data ID 173, a display position 174, a width 175, a height 176, text data 177, and a language 178 to be used.

  The data ID 173 is an identifier for identifying the display data related information 172. The display position 174 indicates a coordinate value (X coordinate, Y coordinate) of the lower left point of the first display area for displaying the text data 177 on the display unit 102. The width 175 and the height 176 indicate the width and the height of the first display area for displaying the text data 177, respectively. Text data 177 indicates the content displayed in the first display area of display unit 102. Here, the data is text data in consideration of ease of translation. The used language 178 indicates the language of the text data 177.

  FIG. 5 shows display image data displayed on the display unit 102 of the translation display device 1 based on the display data table 171 as an example.

  The first display areas 301, 302, 303, 304,... Are identified by data IDs “K001”, “K002”, “K003”, “K004”,. The text data in the display data related information is displayed.

(User attribute table 191)
FIG. 6 shows an example of the data structure of the user attribute table 191.

  The user attribute table 191 is a data table that holds the attributes of each user.

  The user attribute table 191 contains a plurality of pieces of user attribute information 192 in advance as shown in FIG.

  Each user attribute information 192 is composed of a user ID 193, a user name 194, a language 195 used, and a face image ID 196.

  The user ID 193 is an identifier for identifying a user. The user name 194 is a name indicating a user. The language 195 (notation information) is a language that the user can understand. Here, “understandable” means that the user can read (see) and understand the characters in the language, but may also mean that he can listen, understand, speak, and write. As described above, the used language 195 indicates a language (notation) compatible with each user. The face image ID 196 is an identifier for identifying face image data representing a face image of the user.

(Face image data)
The storage unit 104 has an area for storing a plurality of face image data.

  FIG. 6 also shows an example of the face image data.

  The face image data 197a, 197b, 197c, 197d,... Shown in FIG. 6 are respectively user IDs “U001”, “U002”, “U003”, “U004”,.・ It supports.

  Since the face image data is used for recognizing the user, a face image taken from the front of the user's face is preferable. Furthermore, a face image taken from the left side of the user's face and a face image taken from the right side of the face may be included.

(image data)
The storage unit 104 has an area for storing the image data 111a, 111b, 111c, 111d (FIG. 3).

  The image data 111a, 111b, 111c, and 111d are multi-gradation image data generated by shooting with the cameras 12a, 12b, 12c, and 12d, respectively. In the image data, a plurality of pixels are arranged in a matrix.

  FIG. 7 shows, as an example of image data, image data 111a generated by shooting with the camera 12a. The image data 111a includes images 343a, 343b, 343c, and 343d of a plurality of users standing around the translation display device 1, and an image 343e of a part of the top plate of the translation display device 1. Since the shooting direction of the camera 12a is fixed, the same part of the top of the translation display device 1 is always reflected in the image data 111a. In the image 343e, scales 344a, 344b, and 344c provided on the periphery of the top board are reflected. Therefore, the user's standing position represented by the images 343a, 343b, 343c, 343d can be known from the positional relationship between the scales 344a, 344b, 344c in the image 343e and the images 343a, 343b, 343c, 343d.

  The image data 111a illustrated in FIG. 7 shows object data 342a, 342b, 342c, and 342d, which are the face portions of the user, of the images 343a, 343b, 343c, and 343d. The shape of each object data is a rectangle.

  Image data 111b, 111c, and 111d generated by photographing with the cameras 12b, 12c, and 12d are the same as the image data 111a.

  Each of the image data 111a, 111b, 111c, and 111d is a still image, but may be a moving image.

(Object information table 121)
FIG. 8 shows an example of the data structure of the object information table 121.

  The object information table 121 is a data table for storing object information indicating object data which is a user's face portion reflected in image data generated by photographing by the cameras 12a to 12d.

  The object information table 121 has an area for storing a plurality of pieces of object information 122 as shown in FIG. The object information 122 corresponds to one object data, and each object information 122 includes a camera ID 123, an image ID 124, an object ID 125, an object range 126, and a user ID 127.

  The camera ID 123 is an identifier for identifying a camera that has generated image data including the object data. The image ID 124 is an identifier for identifying image data including the object data. The object ID is an identifier for identifying the object data. The object range 126 indicates coordinates where the lower left point and the upper right point of the rectangular object data exist in the image data. The user ID is an identifier for identifying a user whose face is reflected in the object data.

  Since different cameras may shoot one user, the object information table 121 may include a plurality of pieces of object information for the same user.

(Object data)
The storage unit 104 has an area for storing a plurality of object data 128,.

  The object data 128,... Stored in the storage unit 104 is obtained by cutting out image data of a user's face from image data.

  FIG. 8 also shows an example of the object data.

  The object data 128 shown in FIG. 8 corresponds to the object ID “OB0001” in the object information table 121 in FIG.

  The object data is used to detect the user's standing position and the user's line of sight.

(Parts information table 131)
FIG. 9 shows an example of the data structure of the parts information table 131.

  The parts information table 131 has an area for storing a plurality of parts information 132 as shown in FIG.

  Here, the parts (the left eye, the right eye, the mouth, the nose, the left ear, the right ear, etc.) constituting the object (face) are referred to as parts, and the left eye, the right eye, Parts data as partial images of the mouth, nose, left ear, right ear, and the like are cut out. Here, the shape of the parts data is a rectangle.

  The part information 132 corresponds to one part data, and each part information 132 includes an object ID 133, a part ID 134, a part range 135, and a part type 136.

  The object ID 133 is an identifier for identifying object data including the part data. The part ID 134 is an identifier for identifying the part data. The part range 135 indicates coordinates where the lower left point and the upper right point of the rectangular part data exist in the object data including the part data. The part type 136 indicates the type of the part data. That is, the part type 136 indicates whether the part data is the left eye, the right eye, the mouth, the nose, the left ear, or the right ear.

(Part data)
The storage unit 104 has an area for storing a plurality of parts data 137a,.

  The part data 137a,... Stored in the storage unit 104 is obtained by cutting out image data of a part as the part from object data of the user's face.

  FIG. 9 also shows an example of the part data.

  The part data 137a to 137f illustrated in FIG. 9 correspond to the part IDs “PT0001” to “PT0006” of the object ID “OB0001” in the part information table 131 in FIG. 9, respectively.

  The part data is used to detect the user's line of sight.

(User position table 141)
FIG. 10 shows an example of the data structure of the user position table 141.

  The user position table 141 is a data table for storing user position information indicating a user's standing position around the translation display device 1.

  The user position table 141 includes an area for storing a plurality of pieces of user position information 142, as shown in FIG. The user location information 142 corresponds to one user.

  Each user location information 142 includes a user ID 143 and a user location 144.

  The user ID 143 is an identifier for identifying the user. The user position 144 indicates the position of the center of the user's face standing around the translation display device 1 (that is, the user's standing position) in XYZ coordinates.

(Horizontal table 156)
FIG. 11 shows an example of the data structure of the horizontal direction table 156.

  The horizontal direction table 156 is a data table used to calculate the horizontal direction of the user's face.

  The horizontal direction table 156 stores a plurality of pieces of horizontal direction information 157 in advance as shown in FIG. The horizontal direction information 157 corresponds to one horizontal direction of the user's face.

  Each horizontal information 157 includes a ratio 158 and a horizontal angle 159.

  The ratio 158 indicates the position of the center line between the left intersection position and the right intersection position by a ratio (%). Specifically, the ratio “left 0” indicates that the center line matches the left intersection position. The ratio “left 25” indicates that the center line exists between the left intersection position and the right intersection position at a position 25% from the left intersection position. The ratio “50” indicates that the center line exists at the center between the left intersection position and the right intersection position. The ratio “right 25” indicates that the center line exists between the left intersection position and the right intersection position at a position 25% from the right intersection position. The ratio “right 0” indicates that the center line coincides with the right intersection position.

  Here, the center line, the left intersection position, and the right intersection position will be described with reference to FIG.

  The center line 364a is a straight line that passes through the midpoint between the left eye and the right eye in the user's head 363a in the vertical direction of the head. The left intersection position 367 is a position where the line 369 connecting both eyes intersects with the left outline 366a, and the right intersection position 368 is a position where the line 369 connecting both eyes intersects with the right outline 365a. Note that the left contour line 366a indicates the contour line on the left side of the head 363a, and the right contour line 365a indicates the right contour line of the head 363a.

  Here, the distance between the center line and the left intersection is called a left distance, and the distance between the center line and the right intersection is called a right distance.

  The horizontal angle indicates the horizontal direction of the user's face in degrees (degrees).

  For example, in the horizontal direction table 156 shown in FIG. 11, the ratio “left 0” corresponds to the horizontal angle “left 90”. In the case of the ratio “left 0”, it indicates that the user's face is facing left by 90 degrees from the front.

  The ratio “left 25” corresponds to the horizontal angle “left 45”. Here, when the ratio is “left 25”, it indicates that the user's face is facing left by 45 degrees from the front.

  Further, the ratio “50” corresponds to the horizontal direction angle “0”. Here, when the ratio is “50”, it indicates that the user's face is facing the front.

  The same applies to other cases.

(Gaze information table 151)
FIG. 12 shows an example of the data structure of the eye-gaze information table 151.

  The line-of-sight information table 151 is a data table for storing line-of-sight information indicating the line of sight of the user.

  The gaze information table 151 has an area for storing a plurality of gaze information 152 as shown in FIG. The line of sight information 152 corresponds to the line of sight of one user.

  Each line of sight information 152 includes a user ID 153, a user position 154, and direction information 155.

  The user ID 153 is an identifier for identifying the user. The user position 144 indicates the position of the center portion of the user's face standing around the translation display device 1 by XYZ coordinates. The direction information 155 indicates the direction in which the user's line of sight is directed by a vector (X, Y, Z) from the user position 144.

(Viewpoint position table 161)
FIG. 13 shows an example of the data structure of the viewpoint position table 161.

  The viewpoint position table 161 is a data table for storing viewpoint position information indicating the viewpoint of the user on the display surface of the display unit 102.

  The viewpoint position table 161 has an area for storing a plurality of viewpoint position information 162 as shown in FIG. The viewpoint position information 162 corresponds to the viewpoint of one user.

  Each viewpoint position information 162 includes a user ID 163 and a viewpoint position 164.

  The user ID 163 is an identifier for identifying the user. The viewpoint position 164 indicates the position of the user's viewpoint on the display surface of the display unit 102 by XY coordinates.

(Display area table 181)
FIG. 14 shows an example of the data structure of the display area table 181.

  The display area table 181 is a data table for storing display area information indicating a user who is paying attention to each first display area displayed by the display unit 102.

  The display area table 181 has an area for storing a plurality of pieces of display area information 182 as shown in FIG. The display area information 182 corresponds to one piece of text data.

  Each display area information 182 includes only the data ID 183, and each display area information 182 includes the data ID 183 and one or more user IDs 184.

  The data ID 183 is an identifier for identifying text data displayed by the display unit 102. The user ID 184 is an identifier for identifying a user.

  When the display area information 182 includes only the data ID 183, it indicates that there is no user who is paying attention to the first display area where the text data indicated by the data ID 183 is displayed. Further, when the display area information 182 includes the data ID 183 and one user ID 184, it indicates that one user is paying attention to the first display area where the text data indicated by the data ID 183 is displayed. Further, when the display area information 182 includes the data ID 183 and the plurality of user IDs 184, it indicates that the plurality of users are paying attention to the first display area where the text data indicated by the data ID 183 is displayed.

(Translation text data table 201)
FIG. 15 shows an example of the data structure of the translation text data table 201.

  The translated text data table 201 is a data table for storing translation information indicating translated text (translated text data, second data).

  The translation text data table 201 has an area for storing a plurality of pieces of translation information 202 as shown in FIG. The translation information 202 corresponds to one piece of text data.

  Each translation information 202 includes a data ID 203, a translation text 204, and a language 205 to be used.

  The data ID 203 is an identifier for identifying text data displayed by the display unit 102. The translation text 204 is a character string obtained by translating the text data identified by the data ID 203 in the display data table 171. The used language 205 indicates the language used for translation.

(Dictionary database 211)
The dictionary database 211 is a database that is used by the translation unit 111 and stores, for a plurality of languages, the correspondence between terms in the first language and terms in the second language.

(2) Cameras 12a to 12d and input units 105a to 105d
The camera 12a captures an object, generates image data, and outputs the generated image data to the input unit 105a. The input unit 105a receives image data from the camera 12a, and writes the received image data as image data 111a in the storage unit 104 together with a camera ID for identifying the camera 12a.

  The cameras 12b to 12d and the input units 105b to 105d are the same as the camera 12a and the input unit 105a. The cameras 12b to 12d respectively generate image data by shooting, and the input units 105b to 105d generate the image data. The image data 111b to 111d are written into the storage unit 104 together with the camera ID.

(3) Face recognition processing unit 106
The face recognition processing unit 106 reads each image data together with the camera ID from the storage unit 104, and cuts out object data that is a user's face from the read image data. The extraction of the object data is performed, for example, by recognizing a set of eyes, nose, mouth and ears in the image data. The face recognition processing unit 106 also cuts out the eyes, nose, mouth, and ear as part data. Further, the face recognition processing unit 106 determines, for each part data, whether the part data is an eye, a nose, a mouth, or an ear, and specifies the part type.

  The face recognition processing unit 106 writes the object data and the part data cut out from each image data into the storage unit 104.

  In addition, the face recognition processing unit 106 generates an object ID for identifying the extracted object data, and generates an object range indicating the range of the extracted object data. Next, the face recognition processing unit 106 writes the object information including the camera ID, the image ID, the object ID, and the object range in the object information table 121.

  Further, the face recognition processing unit 106 determines whether or not the feature portion of the cut object data matches any one of the face image data 197a, 197b, 197c,... Stored in the storage unit 104 in advance. Judge. If it matches the characteristic part of any of the face image data, the user ID corresponding to the face image ID for identifying the face image data is extracted from the user attribute table 191. The face recognition processing unit 106 writes the extracted user ID in the object information table 121. If the characteristic portion of the extracted object data does not match the characteristic portion of any face image data, the user ID is not written in the object information table 121.

  In addition, the face recognition processing unit 106 generates a part ID for identifying the part data for each part data cut out from the object data, and generates a part indicating a range of the cut out part data. Next, the face recognition processing unit 106 writes, in the part information table 131, part information including an object ID, a part ID, a part range, and a part type for each part data.

(4) Position calculation unit 107
The position calculation unit 107 extracts an object range for each object information from the object information table 121, and displays the extracted object range and a scale included in the image data and attached to the periphery of the top plate of the translation display device 1. By comparing the position with the position, the user position indicating the standing position of the user is calculated by XYZ coordinates. As described above, the standing position of the user indicates the position of the center of the user's face standing around the translation display device 1. The position calculation unit 107 writes the user position information including the user ID for identifying the user and the calculated user position in the user position table 141.

(5) Eye-gaze viewpoint calculation unit 108
(Calculation of the horizontal orientation of the user's face)
The gaze viewpoint calculation unit 108 calculates the horizontal direction of the user's face for each user as follows.

  FIG. 16A is a schematic view of a user's head facing in three directions. That is, the user's head 363a facing the left, the user's head 363b facing the front, and the user's head 363c facing the right are shown.

  In the head 363a facing the left side, a center line 364a which is a straight line passing through the midpoint between the left eye and the right eye in the vertical direction of the head 363a is set, and a left outline 366a indicating the left outline of the head 363a is set. When the right outline 365a indicating the right outline of the head 363a is set, the center line 364a is located closer to the left outline 366a than the right outline 365a.

  In addition, when a center line 364b, a left contour line 366b, and a right contour line 365b are set in the head 363b facing the front in the same manner as the head 363a, the center line 364b becomes the right contour line 365b and the left contour line 366b. Exists in the approximate center position of.

  When a center line 364c, a left contour line 366c, and a right contour line 365c are set in the head 363c facing the right side similarly to the head 363a, the center line 364c is more likely to be a right contour line 365c than a left contour line 366c. Exists close to

  In this way, the eye gaze viewpoint calculation unit 108 specifically determines whether the center line of the head exists on the side close to the left contour line or on the side close to the right contour line. Can use the horizontal table 156 to estimate the horizontal orientation of the user's face.

  Cameras 12a, 12b, 12c, and 12d are installed at the four corners of the top plate of the translation display device 1, respectively, so that any one of the cameras is located at the head (particularly, left eye, right eye, nose, and mouth) of the user. Etc.) can be photographed. As a result, as shown in FIG. 16B, it is possible to estimate the horizontal orientations 362a, 362b, 362c where the user's heads 361a, 361b, 361c face.

  The line-of-sight viewpoint calculation unit 108 reads, for each object ID stored in the object information table 121, object data identified by one object ID from the storage unit 104. The gaze viewpoint calculation unit 108 extracts and extracts the part ID and the part range in the left-eye part information and the part ID and the part range in the right-eye part information from the part information table 131 for the object ID. The part data identified by the left eye part ID and the right eye part ID are read from the storage unit 104.

  Next, the line-of-sight viewpoint calculation unit 108 calculates the center line using the part data and part range of the left eye, the part data and part range of the right eye, and the object data. Further, the line-of-sight viewpoint calculation unit 108 calculates a left outline and a right outline using the object data. Next, the line-of-sight viewpoint calculation unit 108 calculates the ratio of the left distance between the center line and the left outline and the right distance between the center line and the right outline using the object data. For example, when the center line exists at the center between the left contour line and the right contour line, the ratio is 50%. If the center line matches the left contour line, the ratio is 0% left. If the center line coincides with the right outline, the ratio is 0% to the right. If the ratio of the left distance between the center line and the left contour line and the right distance between the center line and the right contour line is 1: 3, the ratio is left 25%. If the ratio of the left distance between the center line and the left contour line to the right distance between the center line and the right contour line is 3: 1, the ratio is 25% to the right.

  Next, the line-of-sight viewpoint calculation unit 108 extracts a horizontal angle corresponding to the calculated ratio from the horizontal direction table 156. If the horizontal angle corresponding to the calculated ratio does not exist in the horizontal table 156, the linear interpolation method is used.

  In this way, the horizontal direction in which the user's face faces is calculated.

(Calculation of the inclination of the user's face from the vertical direction)
The gaze viewpoint calculation unit 108 calculates the inclination of the user's face (head) from the vertical direction for each user as follows.

  FIG. 16C shows a user's face 375a that is not inclined with respect to the vertical direction, a user's face 375b that is slightly inclined with respect to the vertical direction, and a user's face 375c that is further inclined with respect to the vertical direction. .

  For the user's face 375a that is not inclined with respect to the vertical direction, the center line 374a maintains substantially the vertical direction. In the user's face 375b slightly inclined with respect to the vertical direction, the center line 374b is slightly inclined from the vertical direction. In the user's face 375c further inclined with respect to the vertical direction, the center line 374c is further inclined from the vertical direction.

  Accordingly, the inclination of the user's face from the vertical direction can be known by using the center line of the user's head. That is, as shown in FIG. 16D, the viewing directions 373a and 373b can be known from the inclination of the center lines 372a and 372b of the user's faces 371a and 371b from the vertical direction.

  As in the case of calculating the horizontal direction of the user's face, the line-of-sight viewpoint calculation unit 108 uses the left-eye part data and part range, the right-eye part data and part range, and the object data to set the center line. calculate.

  Next, the line-of-sight viewpoint calculation unit 108 calculates a direction perpendicular to the center line based on the calculated center line, and sets this as the inclination of the user's face from the vertical direction.

(Generation of gaze information)
The gaze viewpoint calculation unit 108 combines, for each user, the calculated horizontal direction of the user's face and the inclination of the user's face from the vertical direction to obtain direction information (X, Y, Z) to which the user's gaze is directed. ) Is calculated.

  Next, the line-of-sight viewpoint calculation unit 108 converts, for each user, line-of-sight information including a user ID for identifying the user, user position information written in the user position table 141, and direction information on the line of sight of the user. Write to table 151.

(Generation of viewpoint information)
The line-of-sight viewpoint calculation unit 108 sequentially reads out all line-of-sight information from the line-of-sight information table 151. The line-of-sight viewpoint calculation unit 108 extends the straight line from the position indicated by the user position included in the read line-of-sight information to the direction indicated by the direction information included in the read line-of-sight information. Is calculated as the viewpoint position of the user. The line-of-sight viewpoint calculation unit 108 writes viewpoint position information including a user ID for identifying a user and a viewpoint position in the viewpoint position table 161.

(6) Display area detection unit 109
The display area detection unit 109 sequentially reads out all the display data related information from the display data table 171 and writes the display area information including the data ID in the read display data related information to the display area table 181. As a result, the display area table 181 includes the same number of display area information as the display data related information included in the display data table 171.

  Next, the display area detection unit 109 sequentially reads out all viewpoint position information from the viewpoint position table 161. Each time the viewpoint position information is read, the display area detection unit 109 determines the viewpoint position included in the read viewpoint position information by the display position, width, and height indicated by any display data related information in the display data table 171. It is determined whether it exists in the first display area shown. When the viewpoint position exists in the first display area, the user ID corresponding to the viewpoint position is written in the display area information including the same data ID in the display area table 181. If the viewpoint position does not exist in the first display area, the user ID is not written.

  Thus, the user ID of the user who pays attention to the first display area can be collected for each first display area.

(7) Display control unit 110
The display control unit 110 reads out all display data related information from the display data table 171. For each read-out display data-related information, display image data that displays the text data included in the display data-related information is displayed in the first display area indicated by the display position, width, and height included in the display data-related information. Generate. Here, the language used in the display data-related information, that is, the language on which the displayed text data is based is referred to as a first language (first notation). Next, the display control unit 110 outputs the generated display image data to the combining unit 229.

  The display control unit 110 (viewpoint determining means) determines whether or not the viewpoints of a plurality of users are present in one first display area for each display area information included in the display area table 181. It is determined whether only one viewpoint exists in one first display area, and whether the user's viewpoint exists in one first display area. That is, the display control unit 110 determines whether or not a plurality of viewpoints among the respective viewpoints on the display surface of the plurality of users who gaze at the display surface exist in the first display area.

  The display control unit 110 (determination means) uses the used data ID included in the display area information to be processed only when the viewpoints of a plurality of users exist in one first display area. A language (first language) is acquired from the display data table 171 and, except for the acquired language used, the language used by a plurality of users who pays their eyes to a plurality of viewpoints existing in one first display area is represented by a user attribute table. At least one of the acquired languages used by the plurality of users acquired from 191 is determined as a second language (second notation). That is, the display control unit 110 determines, as the second language, a language (notation) compatible with a plurality of users who look at the first display area by using the acquired language. Here, the notation with affinity determined by the display control unit 110 (determination means) may be a notation registered in advance as a notation familiar to each user. The display control unit 110 controls the translation unit 111 to translate text data displayed in the first display area in the second language. The display control unit 110 (generation unit) generates position information indicating a position of a second display area attached to the first display area on the display surface. The details of the generation of the position information will be described later. The display control unit 110 (output unit) outputs the translated text data and the position information to the synthesizing unit 229.

  Here, when the viewpoints of the plurality of users are present in one first display area, the languages used by the plurality of users who pour eyes at the plurality of viewpoints existing in the first display area are all the first language. In some cases, the display of the second display area is not performed.

  On the other hand, when only one user's viewpoint is present in one first display area, the display control unit 110 matches the language used by the user with the language of the text data displayed in the first display area. It is determined whether or not.

  If they do not match, the display control unit 110 controls the translation unit 111 to translate the text data displayed in the first display area in the language used by the user. Further, the display control unit 110 (generation unit) generates position information indicating the position of the second display area attached to the first display area on the display surface. The display control unit 110 (output unit) outputs the translated text data and the position information to the synthesizing unit 229. In this case, the translated text data displayed in the second display area is displayed in halftone.

  When the language used by the user matches the language of the text data displayed in the first display area, the display control unit 110 does not perform the processing.

(8) Translation unit 111
The translation unit 111 (notation conversion unit) converts the text data written in the first display area into translated text data translated (written) in a second language.

  Specifically, under the control of the display control unit 110, the translation unit 111 translates the text data in the determined second language using the dictionary database 211, and converts the data ID and the translated text that is the translated text data. And the translation information including the language to be used is written in the translation text data table 201.

(9) Synthesizing unit 229
The combining unit 229 receives the display image data from the display control unit 110, outputs the received display image data to the display unit 102 via the output unit 103, and causes the display unit 102 to display the display image data.

  Further, the synthesizing unit 229 receives the translated text data and the position information from the display control unit 110. When the translated text data and the position information are received, the received translated text data is combined with the previously received display image data at the position indicated by the received position information, and the combined display image data is output via the output unit 103. Output to the display unit 102 to display the display image data on the display unit 102.

(10) Output unit 103 and display unit 102
The output unit 103 receives the display image data from the combining unit 229 and outputs the received display image data to the display unit 102.

  The display unit 102 receives the display image data from the output unit 103 and displays the received display image data.

  The display image data includes text data at a display position included in the display data table 171 due to its width and height. Also, at the position indicated by the position information, the translated text data is included.

(11) Main control unit 101
The main control unit 101 includes cameras 12a to 12d, a display unit 102, an output unit 103, input units 105a to 105d, a face recognition processing unit 106, a position calculation unit 107, a line-of-sight viewpoint calculation unit 108, a display area detection unit 109, and display control. The unit 110, the translation unit 111, and the synthesis unit 229 are controlled in a unified manner.

1.3 Example of Screen Displayed by Display Unit FIG. 17 shows an example of text data and translated text data displayed on the display unit 102.

  As shown in the figure, the display unit 102 displays “cause and countermeasure” in Japanese in the first display area 301. In the first display area 301, viewpoints 311c, 312c, and 313c exist.

  The viewpoint 311c stands at the position 311a, and is beyond the line of sight 311b poured by the user identified by the user ID “U001”. The language used by this user is "English". Further, the viewpoint 312c stands at the position 312a, and is ahead of the line of sight 312b poured by the user identified by the user ID “U002”. The language used by this user is "Japanese". Furthermore, the viewpoint 313c stands at the position 313a, and is beyond the line of sight 313b poured by the user identified by the user ID “U003”. The language used by this user is "Spanish".

  As described above, in the first display area 301, Japanese is displayed, and the viewpoints 311c, 312c, and 313c are present. The language used by the user who looks at the viewpoints 311c and 313c is not the displayed Japanese but the displayed language. In English and Spanish, respectively, "Cause and Action" and "Causay Medidas" are displayed in the second display areas 401a and 401b associated with the first display area 301 in English and Spanish, respectively. Have been.

  Therefore, three users who look at the viewpoint in the first display area 301 pay attention to the first display area 301 because the second display areas 401a and 401b attached to the first display area 301 are displayed. You can know the existence of other users. Also, three users who look at the viewpoint in the first display area 301 can understand the meaning of the character data displayed in the first display area 301 and the associated second display areas 401a and 401b. Communication can be achieved between two users.

  The display unit 102 displays “temperature” in Japanese in the first display area 302. In the first display area 302, a viewpoint 314c exists.

  The viewpoint 314c stands at the position 314a and is located beyond the line of sight 314b of the user identified by the user ID “U004”. The language used by this user is "German".

  As described above, in the first display area 302, Japanese is displayed and the viewpoint 314c is present, and the language used by the user who looks at the viewpoint 314c is not the displayed Japanese, but German. In the second display area 402 attached to the first display area 302, "Temperature" is displayed in German. However, since only one viewpoint 314c exists in the first display area 302, “Temperature” is displayed in halftone in the second display area 402.

  In addition, on the display unit 102, the “trajectory of the sun” is displayed in the first display area 303 in Japanese. In the first display area 303, there are a viewpoint 315c and a viewpoint 316c.

  The viewpoint 315c stands at the position 315a and is located beyond the line of sight 315b of the user identified by the user ID “U005”. The language used by this user is "English". Further, the viewpoint 316c stands at the position 316a and is located beyond the line of sight 316b of the user identified by the user ID "U006". The language used by this user is "Japanese".

  As described above, Japanese is displayed in the first display area 303, and the viewpoints 315c and 316c are present. The language used by the user who pours his or her gaze on the viewpoint 315c is not the displayed Japanese but English. Therefore, “Sun's career” is displayed in the second display area 403 attached to the first display area 303 in English.

  Therefore, the two users who look at the viewpoint in the first display area 303 are displayed by the first display area 303 and the associated second display area 403, so that the two users who pay attention to the first display area 303 are displayed. You can know its existence. Also, two users who look at the viewpoint to the first display area 303 can understand the meaning of the character data displayed in the first display area 303 and the associated second display area 403, and Communication can be achieved between users.

  In the display unit 102, “satellite photograph” is displayed in the first display area 304 in Japanese. The first display area 304 has no viewpoint. Therefore, there is no display of translated character data.

(Relationship between first display area and second display area)
The positional relationship between the first display area and the second display area attached to the first display area will be described with reference to FIGS.

  FIG. 18A shows an example in which the first display area 351a and the second display area 352a are in contact with each other. As described above, one side of the first display area 351a may be in contact with one side of the second display area 352a. Since the first display area 351a is in contact with the second display area 352a, the relationship between the first display area 351a and the second display area 352a is clear.

  In this case, the display control unit 110 (generation unit) generates position information indicating the position of the second display area that contacts the first display area on the display surface.

  FIG. 18B shows an example in which the second display area 352b exists within a predetermined distance from the first display area 351b. Thus, the second display area 352b may exist near the first display area 351b. Here, the distance between the first display area 351b and the second display area 352b may be, for example, the distance between the center of gravity of the first display area 351b and the center of gravity of the second display area 352b. The predetermined distance may be, for example, twice as long as the distance between the centers of gravity when the first display area 351b and the second display area 352b are in contact with each other. Since the second display area 352b exists within a predetermined distance from the first display area 351b, the relationship between the first display area 351b and the second display area 352b is clear.

  In this case, the display control unit 110 (generation unit) generates position information indicating the position of the second display area near the first display area on the display surface.

  FIG. 18C shows an example in which the second display area 352c exists so as to overlap the first display area 351c. In this case, it is preferable that the characters displayed in the first display area 351c do not overlap with the characters displayed in the second display area 352c. Since the second display area 352c exists so as to overlap the first display area 351c, the relationship between the first display area 351c and the second display area 352c is clear.

  In this case, the display control unit 110 (generation unit) generates position information indicating the position of the second display area overlapping the first display area on the display surface.

  FIG. 18D shows an example in which the second display area 352d is included in the first display area 351d. In this case, it is preferable that the characters displayed in the first display area 351d do not overlap with the characters displayed in the second display area 352d. Since the second display area 352d is included in the first display area 351d, the relationship between the first display area 351d and the second display area 352d is clear.

  In this case, the display control unit 110 (generation unit) generates position information indicating the position of the second display area included in the first display area on the display surface.

  Next, the correspondence between the display in the first display area and the display in the second display area will be described with reference to FIGS.

  FIG. 18E shows an example in which the hatching pattern applied to the background in the first display area 351e matches the hatching pattern applied to the background in the second display area 352e. Furthermore, the display colors of the hatching patterns may match. Thus, since the hatching patterns are the same, the relationship between the first display area 351e and the second display area 352e is clear.

  FIG. 18F shows an example in which the dot pattern applied to the background in the first display area 351f matches the dot pattern applied to the background in the second display area 352f. Further, the display colors of the dot patterns may match. As described above, since the dot patterns are the same, the relationship between the first display area 351f and the second display area 352f is clear.

  FIG. 18G shows an example in which the first display area 351 g and the second display area 352 g are linked by a line image 353. As shown in this figure, the image may be linked by a double line image, or may be linked by a line image such as a broken line, a dash-dot line, or a wavy line. As described above, since the first display area 351g and the second display area 352g are linked by the line image 353, the relationship between the first display area 351g and the second display area 352g is clear.

  FIG. 18H illustrates an example in which the display pattern of the frame of the first display area 351h matches the display pattern of the frame of the second display area 352h. Further, the display colors of the frames may match. Thus, since the display patterns of the frames are the same, the relationship between the first display area 351h and the second display area 352h is clear.

  In addition, the display color of the characters in the first display area may match the display color of the characters in the second display area (not shown). In this case, since the display colors of the characters are the same, the relationship between the inside of the first display area and the inside of the second display area is clear.

  The display color of the first display area is different from the display color of the second display area that has no correspondence with the first display area. Further, the display pattern of the first display area and the display pattern of the second display area having no correspondence are different from each other.

  In the above case, the display control unit 110 (generation unit) generates the same color pattern information indicating the display color or display pattern in the first display area and the display color or display pattern in the second display area. You may. Further, the display control unit 110 (output unit) may output color pattern information to the synthesizing unit 229 in order to display using the display color or the display pattern.

1.4 Operation The operation of the translation display device 1 will be described with reference to the flowcharts shown in FIGS.

  Under the control of the display control unit 110, the display unit 102 displays the text data of all the display data related information included in the display data table 171 at the display position by the width and the height. Here, the language on which the displayed text data is based is called the first language (step S101).

  Next, the cameras 12a to 12d respectively capture images of the target, generate image data, and write the generated image data to the storage unit 104 via the input units 105a to 105d. The face recognition processing unit 106 acquires image data from the storage unit 104 (Step S102).

  Next, the face recognition processing unit 106 detects a user's face from the acquired image data (step S103). Next, the position calculation unit 107 detects the standing position of the user from the acquired image data (Step S104). Next, the face recognition processing unit 106 specifies the user using the user attribute table 191 and the face image data of the user (step S105).

  Next, the line-of-sight viewpoint calculation unit 108 calculates the horizontal direction of the user's face (step S106), and then calculates the inclination of the user's face from the vertical direction (step S107). Subsequently, the line-of-sight viewpoint calculation unit 108 calculates the user's line of sight using the horizontal direction of the user's face and the inclination of the user's face from the vertical direction (step S108). The viewpoint of the user on the display surface is calculated (step S109).

  Next, the display control unit 110 repeats steps S111 to S124 for each display area information included in the display area table 181 (steps S110 to S125).

  The display control unit 110 determines whether or not the viewpoints of a plurality of users exist in one first display area for one first display area. Specifically, the display control unit 110 determines whether one display area information includes a plurality of user IDs (Step S111).

  When the viewpoints of a plurality of users are present in one first display area, that is, when one display area information includes a plurality of user IDs ("YES" in step S111), the display control unit 110 Are acquired from the user attribute table 191 and at least one of the acquired languages used is determined as the second language. However, when determining the second language, the first language is excluded (step S112). Next, the translation unit 111 obtains translated text data by translating the text data (step S113). Next, the display control unit 110 sets a second display area where the translated text data is to be displayed so as to accompany the first display area (step S114). Next, the display control unit 110 outputs the translated text data and the position information indicating the position of the second display area to the synthesizing unit 229 (Step S115). Next, the display unit 102 displays the display image data including the second text data in the second display area (Step S116).

  When the viewpoints of a plurality of users do not exist in one first display area, that is, when one display area information does not include a plurality of user IDs ("NO" in step S111), the display control unit 110 It is determined whether only one user's viewpoint exists in one first display area, that is, it is determined whether one display area information includes only one user ID (step S119).

  When only one user's viewpoint exists in one first display area, that is, when one display area information includes only one user ID (“YES” in step S119), the display control unit 110 Then, the language used by the one user is acquired from the user attribute table 191, and it is determined whether or not the language used (first language) of the display text data matches the language used by the user (step S120). If the language used in the display text data does not match the language used by the user ("NO" in step S120), translation section 111 translates the text data to obtain translated text data (step S121), and then displays the translated text data. The control unit 110 sets a second display area where the translated text data is to be displayed, for example, so as to accompany the first display area (step S122). Next, the display control unit 110 outputs the translated text data and the position information indicating the position of the second display area to the synthesizing unit 229 (Step S123). Next, the display unit 102 displays the display image data including the second text data in the second display area. At this time, in the second display area, the second text data is displayed in halftone (step S124).

  If the language used (first language) of the display text data matches the language used by the user ("YES" in step S120), display control unit 110 does nothing.

  When the user's viewpoint does not exist in one first display area, that is, when one display area information does not include a user ID (“NO” in step S119), the display control unit 110 do not do.

  When the repetition of steps S111 to S124 ends, the display control unit 110 shifts the control to step S102 and repeats the processing.

1.5 Summary As described above, according to the above-described embodiment, only when a plurality of user's viewpoints exist in the first display area in which text data is displayed in the first language, these viewpoints are displayed. The text data is converted in a second language (excluding the first language) used by a plurality of users who look at the line of sight, and the translated text data is displayed in a second display area attached to the first display area.

  By displaying the translated text data in the second display area attached to the first display area in this way, it can be seen that a plurality of users are paying attention to the first display area. In other words, one user who is paying attention to the first display area has another user who is paying attention to the first display area other than himself, and that user is displayed in the first display area as well as himself. You can know that you are interested in the text data. As a result, communication can be achieved between a plurality of users. Further, since the translated text data is displayed in the second display area, even a user who cannot understand the first language can know the meaning of the text data displayed in the first display area.

2. Modifications Although the present invention has been described based on the above embodiment, it is not limited to the above embodiment. The following may be performed.

2.1 Modification (1)
In the above embodiment, as shown in FIG. 5, even when the user's viewpoints 311c, 312c, and 313c are gathered in the first display area 301, the size of the first display area 301 is not changed. . However, the invention is not so limited.

  When the viewpoints of many users are gathered in the first display area of the display unit 102, the first display area may be enlarged. Further, the second display area displayed along with the first display area may be displayed large.

  FIG. 21A shows the relationship between text data displayed on the display unit 102 and the user's standing position. FIG. 21B shows an example of the text data and the translated text data which are largely displayed on the display unit 102 of the translation display device 1.

  As shown in FIG. 21A, many user's viewpoints are present in the first display area 301. On the other hand, the viewpoints of the user are not gathered for the first display areas 302, 303, and 304.

  In such a case, the first display area 301 is enlarged to be the first display area 421a as shown in FIG. 21B, and the second display areas 421b and 421c accompanying the first display area 421a are also provided. May be larger.

  In this case, the first display areas 302, 303, and 304 in FIG. 21A may be made smaller and their widths may be made smaller. As shown in FIG. 21B, the first display area 421a and the first display area 302 of FIG. , 303, 304 may be deformed and displayed as the first display areas 422, 423, 424. For example, in FIG. 21A, the first display areas 302, 303, and 304 are long in the Y-axis direction, but in FIG. 21B, the first display areas 422, 423, and 424 are It is long in the X-axis direction.

  Specifically, the display control unit 110 (viewpoint determining unit) may determine whether the number of the plurality of viewpoints in the third display area is equal to or more than a predetermined number. When the number of the plurality of viewpoints is equal to or more than a predetermined number, the display control unit 110 (generating unit) further generates enlarged position information indicating the position of the enlarged region obtained by enlarging the third display region. The display control unit 110 (output unit) further outputs the enlarged position information to the combining unit 229 in order to display the text data in the enlarged area indicated by the enlarged position information.

Further, when the number of the plurality of viewpoints is equal to or more than the predetermined number, the display control unit 110 further generates second enlarged position information indicating the position of the enlarged region of the second display region, and enlarges the translated text data. May be. The display control unit 110 further combines the second enlarged position information and the enlarged translated text data in order to display the enlarged translated text data in the enlarged second display area indicated by the second enlarged position information. Here, the number of the plurality of viewpoints in the third display area is equal to or more than the predetermined number, for example, when the number of the plurality of viewpoints in the third display area is equal to Is 25% (or may be 50%, 75%, or 90%) of the number of all viewpoints on the display surface.

  As described above, according to the modified example (1), when many user's viewpoints are gathered in the third display area, the third display area is enlarged, and the second display displayed along with the third display area. The area is enlarged, and the displayed text data and translated text data are enlarged.

  For this reason, the user who pays attention to the third display area can clearly recognize these characters with the large-displayed text data and translated text data.

2.2 Modification (2)
In the above embodiment, as shown in FIG. 17, each character in the text data displayed in the first display area 301, 302, 303, 304 is viewed from the X axis positive direction to the X axis negative direction. And erect. The same applies to the translated text data displayed in the second display areas 401a, 401b, 402, and 403. However, the invention is not so limited.

  Depending on the direction of the user's line of sight and the standing position of the user, each character in the text data displayed in the first display area and the second display area and each character in the translation text data are displayed in the first display area and the second display area, respectively. The orientation of the text data and the translated text data may be adjusted so that the characters appear to be upright.

  FIG. 22 shows an example of the relationship between the text data and translated text data displayed on the display unit 102 of the translation display device 1 and the user's standing position, and an example of the text data and translated text data displayed on the display unit 102. .

  As in FIG. 5, also in this figure, “cause and countermeasure” is displayed on the display unit 102 in the first display area 301 in Japanese. In the first display area 301, viewpoints 311c, 312c, and 313c exist. The standing position of the user who pours his or her gaze at these viewpoints is the same as in FIG.

  As described above, in the first display area 301, Japanese (first language) is displayed, and the viewpoints 311c, 312c, and 313c are present, and the language used by the user who looks at the viewpoints 311c and 313c is displayed. Since the language is English and Spanish (second language), respectively, instead of Japanese, the second display areas 433 and 432 attached to the first display area 301 are displayed in English and Spanish, respectively, as in FIG. The words are labeled “Cause and Action” and “Causey Medidas”.

  However, what is different from FIG. 5 is that the second display areas 433 and 432 make the respective display characters appear to be erect when viewed from the standing position of the user who looks at the viewpoints 311c and 313c. It is. Further, in the second display area 431 as well, the character data in Japanese is made to look upright when viewed from the standing position of the user who looks at the viewpoint 312c.

  The same applies to the second display area 434 attached to the first display area 302 and the second display area 435 attached to the first display area 303.

  Specifically, the display control unit 110 (generation unit) converts the image data in which the translated text data is rotated and arranged so that the characters constituting the translated text data appear to be upright from the user's standing position. Generate. The display control section 110 (output means) outputs image data. The display unit 102 displays image data obtained by rotating the translated text data in the second display area.

  In this way, the displayed content looks upright according to the direction of the user's line of sight and the standing position of the user, so that the user's understanding of the displayed content can be promoted.

2.3 Modification (3)
In the above embodiment, the user's gaze direction and the user's gaze direction are determined using the images of the user captured by the four cameras 12a, 12b, 12c, and 12d provided at the four corners of the top plate of the translation display device 1. The user's viewpoint on the display surface of the display unit 102 is obtained. However, the invention is not so limited.

  A translation display system 501 as a modified example (3) is a system that translates and displays text data written in a first language in a second language in a part of a first display area on a display surface. As shown in FIG. 23, it may be constituted by a monitor 511, a translation device 512, and glasses 513, 514, 515 worn on the face of the user. The monitor 511 and the translation device 512 are connected by a cable 516.

  The translation display system 501 replaces a structural part for detecting a user's viewpoint in the translation display device 1 in the embodiment with glasses 513, 514, and 515, and replaces the display unit 102 in the translation display device 1 with a monitor 511. is there.

  For this reason, the translation device 512 is configured by the structural components for detecting the user's viewpoint and the remaining components other than the display unit 102 in the translation display device 1 in the embodiment.

  The glasses 513 communicate with the translation device 512 by short-range wireless communication. Similarly to the glasses 513, the glasses 514 and 515 communicate with the translation device 512 by short-range wireless communication.

  In a frame portion of the glasses 513, a camera 513a that captures an eye direction of a user wearing the glasses 513 and a calculation unit are arranged. The camera 513a captures the direction of the user's line of sight to generate image data, and the calculation unit calculates the position of an object in the center of the generated image data as a viewpoint, and sends the calculated viewpoint to the translation device 512. Output. The glasses 514 and 515 also have the same structure as the glasses 513.

  The translation device 512 acquires, from the glasses 513, 514, and 515, respective viewpoints on the display surface of a plurality of users who gaze at the display surface of the monitor 511, and among the acquired viewpoints, a plurality of viewpoints are displayed on the first display. Judge whether or not it exists in the area, and only when the plurality of viewpoints exist in the first display area, except for the first language, a plurality of users who pour eyes at the plurality of viewpoints understand each Among the languages, at least one language is determined as a second language, and the text data written in the first display area is related to the meaning of the text data, and the translated text expressed in the second language It translates the data into data, generates position information indicating the position of the second display area associated with the first display area on the display surface, and displays the translated text data and the position information on the monitor 5. And outputs it to the 1.

  The monitor 511 has a display surface, displays text data in a first display area of the display surface, receives translated text data and position information, and displays translated text data in a second display area indicated by the position information. I do.

(Summary)
As described above, according to the modification (3), a plurality of users pay attention to the first display area by displaying the translated text data in the second display area accompanying the first display area. I understand. In other words, one user who is paying attention to the first display area has another user who is paying attention to the first display area other than himself, and that user is displayed in the first display area as well as himself. You can know that you are interested in the text data.

  As a result, communication between a plurality of users who focus on the first display area can be achieved.

2.4 Modification (4)
In the above embodiment, the user's gaze direction and the user's gaze direction are determined using the images of the user captured by the four cameras 12a, 12b, 12c, and 12d provided at the four corners of the top plate of the translation display device 1. The user's viewpoint on the display surface of the display unit 102 is obtained. However, the invention is not so limited.

  As a modification (4), a translation display system 531 whose schematic configuration is shown in FIG. 24A translates text data displayed in a first language in a part of a first display area on a display surface in a second language. The system includes a whiteboard 532, a translation device 536, and glasses 533, 534, and 535 worn on the user's face.

  The translation display system 531 is obtained by replacing the structural part for detecting the user's viewpoint and the display unit 102 in the translation display device 1 according to the embodiment with glasses 533, 534, and 535.

  For this reason, the translation device 536 is configured by the structural components for detecting the user's viewpoint and the remaining components other than the display unit 102 in the translation display device 1 in the embodiment.

  The glasses 533 communicate with the translation device 536 by short-range wireless communication. Like the glasses 533, the glasses 534 and 535 communicate with the translation device 536 by short-range wireless communication.

  The whiteboard 532 has a display surface.

  The glasses 533 are glasses-type display devices that are worn on the user's face and include a camera 533a. The camera 533a captures the direction of the user's line of sight to generate image data, calculates the position of an object in the center of the generated image data as a viewpoint, and sends the calculated viewpoint to the translation device 536 by short-range wireless communication. Send to. In addition, the glasses 533 display image data obtained by shooting.

  The glasses 534 and 535 also have the same configuration as the glasses 533.

  The translation device 536 acquires, from the glasses 533, 534, and 535, respective viewpoints of the plurality of users who gaze at the display surface of the whiteboard 532, and among the acquired viewpoints, a plurality of viewpoints are first. Judge whether or not it exists in the display area, and only when the plurality of viewpoints exist in the first display area, except for the first language, a plurality of users who look at the plurality of viewpoints and understand each other Among the languages, at least one language is determined as a second language, text data written in the first display area is translated into a second language method, and attached to the first display area on the display surface. The position information indicating the position of the second display area is generated, and the translated text data and the position information are transmitted to the glasses 533, 534, and 535 by short-range wireless communication.

  The glasses 533, 534, and 535 receive the translated text data and the position information indicating the position of the second display area from the translation device 536 by short-range wireless communication, and Display the translated text data.

(Summary)
As described above, according to the modification (4), a plurality of users pay attention to the first display area by displaying the translated text data in the second display area accompanying the first display area. I understand. In other words, one user who is paying attention to the first display area has another user who is paying attention to the first display area other than himself, and that user is displayed in the first display area as well as himself. You can know that you are interested in the text data.

  As a result, communication between a plurality of users who focus on the first display area can be achieved.

2.5 Other Modifications (1) In the above embodiment, the display unit 102 displays both text data and translated text data. That is, as shown in FIG. 17, the display unit 102 displays text data in the first display area 301, and displays translated text data in the second display areas 401a and 401b attached to the first display area 301. I have. However, this is not a limitation.

  The display unit 102 may display the translated text data in the first display area 301 instead of the text data.

  Specifically, the display control unit 110 (generation unit) generates position information indicating the position of the first display area, and the display control unit 110 (output unit) sets the second display area as the second display area. In order to display the translated text data, the generated position information may be output.

  When there are multiple users' viewpoints in the first display area, if there are no users who use the first language among the multiple users who look at these viewpoints, use the display surface effectively. Can be.

  (2) In the above embodiment, as shown in FIG. 17, the display unit 102 displays text data in the first display area 301, and displays the text data in the second display areas 401a and 401b attached to the first display area 301. The translated text data is displayed. However, this is not a limitation.

  The display control unit 110 (blank determining means) may determine whether there is a blank area large enough to display the translated text data around the first display area. That is, it is determined whether or not there is a blank area larger than the display size of the translated text data and in which no character is displayed. When determining that there is a blank area, the display control unit 110 (generation unit) generates position information indicating the position of the second display area in the blank area, and the display control unit 110 Alternatively, the translated character data may be displayed in a blank area around the first display area.

  On the other hand, if the display control unit 110 determines that there is no blank area large enough to display the translated text data around the first display area, the display control unit 110 may reduce the size of displaying the translated text data to a small extent. For example, after reducing the size by, for example, 5% to 10% in area, it is again determined whether or not there is a blank area large enough to display the reduced-sized translated text data. You may decide. When there is a blank area, translated character data having a reduced size may be displayed in the blank area.

  If the display control unit 110 determines that there is no blank area large enough to display the translated text data around the first display area, for example, the display control unit 110 sets the first display area to the first display area. After being moved to the left or right by a length of about 5% to 10% of the width of the display area, or by moving the first display area from 5% to 10% of the height of the first display area. After moving upward or downward by a length of about%, it may be determined again whether or not a blank area exists. After the display area is moved, if there is a blank area, the translated text data may be displayed in the blank area.

  By arranging the translated text data in a blank area where no character is displayed, other necessary characters and the like are not covered by the translated text data.

  (3) In the above-described embodiment and modified examples, text data expressed in a first language (first notation) is displayed in a first display area, and text data is displayed in a second language (second notation). And the translated text data is displayed in the second display area. However, the invention is not so limited.

  Standard languages used as norms nationwide may be considered as one language, and dialects that have developed differently in different regions may be considered as different languages.

  The first notation may be an expression in a standard language used nationwide as a norm, and the second notation may be an expression in a dialect that has developed differently depending on the region.

  In other words, the first language may be a standard language and the second language may be a dialect. The text data expressed by the standard language that is the first language is displayed in the first display area, the text data is translated into a dialect that is the second language, and the translated text data may be displayed in the second display area. .

  Further, the first notation may be an expression in a dialect, and the second notation may be an expression in a standard language.

  In other words, the first language may be a dialect and the second language may be a standard language. The text data expressed in the first language dialect may be displayed in the first display area, the text data may be translated into the second language standard language, and the translated text data may be displayed in the second display area. .

  According to this method, communication can be achieved between a person who understands the standard language and a person who understands the dialect.

  (4) In the above-described embodiment and modified examples, text data expressed in a first language (first notation) is displayed in a first display area, and text data is displayed in a second language (second notation). And the translated text data is displayed in the second display area. However, the invention is not so limited.

  The first notation may be an expression in kanji, and the second notation may be an expression in katakana, hiragana, or romaji.

  At this time, the text data represented by the kanji as the first notation is displayed in the first display area, and the text data is converted into katakana (or hiragana, romaji) as the second notation, and the converted text data May be displayed in the second display area.

  Further, the first notation may be expressed in katakana, hiragana, or romaji, and the second notation may be expressed in kanji.

  At this time, text data written in katakana (or hiragana, romaji) as the first notation is displayed in the first display area, and the text data is converted into kanji as the second notation, and converted text data May be displayed in the second display area.

  According to this method, communication can be achieved between a person who understands kanji and a person who understands katakana, hiragana, or romaji.

  (5) In the above embodiment and the modified example, text data expressed in the first language (first notation) is displayed in the first display area, and text data is displayed in the second language (second notation). And the translated text data is displayed in the second display area. However, the invention is not so limited.

  The first notation may be expressed by a symbol mark that symbolizes a predetermined idea, and the second notation may be expressed by a character that expresses the idea symbolized by the symbol mark. Further, the second notation may be a notation of a description of the symbol mark. Here, the symbol mark may be a graphic symbolizing a predetermined idea.

  A symbol mark represented by the first notation is displayed in the first display area, and a predetermined idea represented by the symbol mark is described as a description in the second notation, and a description is displayed in the second display area. May be. As the description, translated text data translated in each language may be used.

  The symbol mark is, for example, a company emblem, a crest, an insignia, a national emblem, a family crest symbol, a logo, a seal, and is a figure symbolizing an individual, a family, a company, an organization, and the like. In this case, the description is a character representation of a name indicating an individual, family, company, organization, or the like.

  The symbol mark is, for example, a map symbol. A specific example is a map symbol indicating a shrine, a castle ruin, a court, and a lighthouse, and is a figure indicating a route of a railway (shinkansen). Further, specific examples of the description are “shrine”, “castle ruins”, “court”, “lighthouse”, and “shinkansen”, respectively. Further, English, Spanish, and French translations may be added to these descriptions.

  The symbol mark is, for example, a symbol of an electric circuit. Specific examples are symbols indicating transformers, resistors, and transistors. Specific examples of the description are “transformer”, “resistor”, and “transistor”, respectively. Further, English, Spanish, and French translations may be added to these descriptions.

  (6) In the above embodiments and modifications, the data represented by the first notation is displayed in the first display area, the data is converted by the second notation, and the converted data is converted to the second notation. It is displayed in the display area. Thus, there is a one-to-one relationship between the data and the converted data. However, the invention is not so limited.

  In the first display area, a symbol indicating a transistor may be displayed, and in the second display area, “transistor” may be displayed as a descriptive sentence, and a graph showing electrical characteristics and temperature characteristics of the transistor may be displayed as a detailed description. .

  In the first display area, a map symbol indicating a shrine is displayed, and in the second display area, "shrine" is described as a descriptive text, and the origin, history, religion, etc. of the shrine are described as a detailed description. Is also good.

  Also, in the first display area, a figure indicating the route of the Shinkansen (railroad) is displayed, and in the second display area, "Shinkansen" is displayed as a descriptive text, and the timetable and operation status of the train are displayed as the detailed description. Is also good.

  As described above, the first notation is an expression using a symbol mark that is a figure that symbolizes a predetermined idea, and the second notation is an expression that uses a character to express the idea symbolized by the symbol mark. Good. Further, the first notation may be based on a simple expression, and the second notation may be based on an expression that explains the simple expression in detail. Here, the expression that explains the simple expression in detail may be a character or a graphic including the character.

  (7) In the above-described embodiment and the modified example, when only one viewpoint is present in the first display area, the translated text data is converted to the second display area associated with the first display area by halftone. To display. However, the invention is not so limited.

  When only one viewpoint is present in the first display area, the second display area itself may not be displayed.

  (8) In the above-described embodiment and modified examples, it is assumed that the viewpoint of the user is fixed at one point. However, the invention is not so limited.

  When the user is paying attention to the first display area, it is considered that the viewpoint of the user is not fixed at one point and moves irregularly inside and outside the first display area. In such a case, if the user's viewpoint exists inside the first display area within a period of, for example, 80% of a certain period (for example, 5 seconds), the user's viewpoint is changed to the first display. It may be considered that it exists inside the region.

  (9) In the above embodiment, the language used by the user is stored in the user attribute table 191 (FIG. 6) corresponding to the user. Has been acquired. However, the invention is not so limited.

  Such a method may be used as long as the language used by the user can be recognized from the face image of the user captured by the camera.

  (10) In the above embodiment, the display control unit 110 (determination unit) determines, from the user attribute table 191, the languages used by a plurality of users who look at a plurality of viewpoints existing in one first display area. At least one of the acquired languages used by the plurality of users is determined as the second language (second notation).

  Here, when there are a plurality of notations having an affinity for a plurality of users who focus on the viewpoint in the first display area, one notation having an affinity for the most users among the plurality of users is referred to as a second notation. You may decide as two notations.

  In addition, priority is set for each language used, and when there are a plurality of notations that are compatible with a plurality of users who focus on the viewpoint in the first display area, among the languages used by these users, the highest priority is set. The language in which the priority is set may be determined as the second notation.

  When the display control unit 110 (determination unit) determines a plurality of notations compatible with a plurality of users as the second notation, the translation unit 111 (notation conversion unit) transmits the text data (second notation). May be translated (converted) in a plurality of determined languages (notations). In this case, a plurality of translated text data is generated.

  As described above, when a plurality of translated text data is generated, the second display area may be divided into a plurality of partial areas for displaying the translated text data, according to the determined plurality of notations. . The display control unit 110 (generation unit) may generate partial area information indicating a plurality of partial areas. The display control unit 110 (output unit) may output the partial area information. The display unit 102 displays the translated text data in a plurality of partial areas in the second display area.

  Further, as described above, when a plurality of translated text data is generated, the display control unit 110 (output means) switches and displays each of the plurality of translated text data in one second display area. , And sequentially outputs a plurality of translated text data. The display unit 102 displays a plurality of translated text data in one second display area while switching.

  (11) As described above, each device is a computer system including a microprocessor and a memory. The memory may store a computer program for control, and the microprocessor may operate according to the computer program for control.

  Here, the control computer program is configured by combining a plurality of instruction codes indicating instructions to the computer in order to achieve a predetermined function.

  Further, the control computer program may be recorded on a computer-readable recording medium, for example, a flexible disk, a hard disk, an optical disk, a semiconductor memory, or the like.

  (12) The above embodiment and the above modified examples may be combined.

  The notation conversion device according to the present invention is such that, when the second data is displayed in the second display area associated with the first display area, a plurality of users who pour a viewpoint on the first display area are provided in the first display area. Knowing the existence of the user who is paying attention to, can communicate with users who understand different notations, and convert the data written in the first notation into the notation in the second notation It is useful as technology.

Reference Signs List 1 translation display device 11 legs 12a to 12d camera 101 main control unit 102 display unit 103 output unit 104 storage unit 105a to 105d input unit 106 face recognition processing unit 107 position calculation unit 108 gaze viewpoint calculation unit 109 display area detection unit 110 display Control unit 111 Translation unit 229 Synthesis unit 501 Translation display system 511 Monitor 512 Translation device 513-515 Glasses 513a-515a Camera 516 Cable 531 Translation display system 532 Whiteboard 533-535 Glasses 533a-535a Camera 533b-535b Display unit 536 Translation device

Claims (28)

A notation conversion device that converts first data written in a first notation in a first display area on a display surface into a notation in a second notation,
Among viewpoints on the display surface of a plurality of users who gaze at the display surface, viewpoint determination means for determining whether a plurality of viewpoints are present in the first display area,
Only when there are a plurality of viewpoints in the first display area, except for the first notation, a notation compatible with a plurality of users who look at the first display area, a second notation. Determining means for determining as
Notation conversion means for converting the first data described in the first display area into second data represented by the second notation,
On the display surface, generating means for generating position information indicating the position of the second display area,
An output unit that outputs the second data and the position information to an external display device in order to display the second data in the second display area. The notation conversion device according to claim 1, wherein the first notation is an expression in a first language, and the second notation is an expression in a second language. The first notation is an expression in kanji, the second notation is an expression in katakana, hiragana or romaji, or the first notation is an expression in katakana, hiragana or romaji, The notation conversion device according to claim 1, wherein the second notation is an expression using Chinese characters. The first notation is an expression in a standard language used nationwide as a norm, the second notation is an expression in a dialect developed differently depending on the region, or the first notation is a dialect The notation conversion device according to claim 1, wherein the second notation is an expression in a standard language. The said 1st notation is expression by the symbol mark which symbolizes a predetermined idea, and the said 2nd notation expresses the idea symbolized by the said symbol mark by a character. 2. The notation conversion device according to 1. The notation conversion device according to claim 1, wherein the first notation is based on a simple expression, and the second notation is based on an expression that explains the simple expression in detail. . The expression conversion device according to claim 6, wherein the expression for explaining the expression in detail is a character or a graphic including the character. The notation conversion device according to claim 1, wherein the notation having an affinity determined by the determination means is a notation registered in advance as a notation familiar to each user. . The determining means, when there are a plurality of notations having an affinity for the plurality of users, the one notation having an affinity for the most users among the plurality of users is used as the second notation. The notation conversion device according to claim 1, wherein the notation is determined. When the plurality of notations compatible with the plurality of users are determined as the second notation by the determining unit, the notation conversion unit converts the first data into the determined plurality, respectively. The notation conversion device according to claim 1, wherein the conversion is performed by the following notation. The second display area is divided into a plurality of partial areas each displaying the converted first data, by the plurality of notations,
The generating means further generates partial area information indicating the plurality of partial areas,
The notation conversion device according to claim 10, wherein the output unit further outputs the partial area information. The output means sequentially outputs a plurality of the converted first data in order to switch and display each of the plurality of converted first data in the second display area. The notation conversion device according to claim 10. The generating means generates the position information indicating the position of the first display area,
The notation conversion device according to claim 1, wherein the output unit outputs the position information in order to display the second data in the first display area as the second display area. Further, in the area around the first display area, a blank determining means for determining whether there is a blank area larger than the size of the second data and in which no data is displayed,
The notation conversion according to claim 1, wherein when it is determined that a blank area exists, the generation unit generates the position information indicating a position of the second display area in the blank area. apparatus. The viewpoint determination means may further include, among the respective viewpoints on the display surface of the plurality of users gazing at the display surface, whether the number of the viewpoints present in the third display area is equal to or greater than a predetermined number. Judge,
The generation unit, further, when the number is equal to or more than a predetermined number, generates expansion range information indicating a range of an expansion area obtained by expanding the third display area,
The notation conversion device according to claim 1, wherein the output unit further outputs the enlarged range information to the display device so as to enlarge and display the third display area. On the display surface, the second display area exists within a predetermined distance from the first display area, and the second display area exists in contact with the first display area, and overlaps with the first display area. The notation conversion device according to claim 1, wherein the second display area exists or the second display area is included in the first display area. The generating means further includes:
A display color or display pattern in the first display area and the same color pattern information indicating a display color or display pattern in the second display area are generated,
2. The notation conversion device according to claim 1, wherein the output unit further outputs the color pattern information to the display device in order to display the display using the display color or the display pattern. 3. The notation conversion device according to claim 1, further comprising a detection unit configured to detect a viewpoint on the display surface of a user who gazes at the display surface. The detecting means,
A camera for acquiring image data with the user as a subject,
Extracting the standing position of the user and the line-of-sight direction of the user from the image data, and based on the extracted standing position and the line-of-sight direction, the viewpoint on the display surface of the user who gazes at the display surface The notation conversion device according to claim 18, further comprising: a calculation unit that calculates the notation. The display device is provided in a posture in which the rectangular display surface faces upward,
The user stands around the display device,
The generating means further generates image data in which the second data is rotated and arranged so that characters constituting the second data are viewed from the user's standing position as being upright,
The output means outputs the image data as the second data,
The notation conversion device according to claim 1, wherein the display device displays the image data in the second display area. Further, a storage means for storing notation information indicating notation that is compatible with each of the plurality of users,
The determining unit acquires the notation information from the storage unit, and determines a notation compatible with the plurality of users by using a notation indicated by the acquired notation information. The notation conversion device according to claim 1, wherein The viewpoint determining means is attached to the user's face, acquires the viewpoint from glasses outside the notation conversion device including a camera and a calculation unit,
The camera generates image data by photographing the user's line of sight,
The notation conversion device according to claim 1, wherein the calculation unit calculates, as a viewpoint, a position of an object appearing in a central portion of the image data. The viewpoint determination means is attached to the user's face, acquires the viewpoint from the display device of the glasses type equipped with a camera,
The display device according to claim 1, wherein the camera captures an image of a user's line of sight with the camera to generate image data, and calculates a position of an object in a central portion of the generated image data as a viewpoint. The notation conversion device according to 1. A conversion display device that converts the first data written in the first notation into the first notation in the first notation on the display surface and converts the first data into the notation in the second notation, and
Display means for displaying the first data in the first display area,
Viewpoint acquisition means for acquiring respective viewpoints on the display surface of a plurality of users gazing at the display surface,
Judging means for judging whether or not a plurality of viewpoints are present in the first display area among the obtained viewpoints,
Only when there are a plurality of viewpoints in the first display area, except for the first notation, a notation compatible with a plurality of users who look at the first display area, a second notation. Determining means for determining as
Notation conversion means for converting the first data described in the first display area into second data represented by the second notation,
On the display surface, generating means for generating position information indicating the position of the second display area,
Output means for outputting the second data and the position information to the display means,
The conversion display device, wherein the display unit receives the second data and the position information, and displays the second data in the second display area. A conversion display system that converts the first data written in the first notation into the notation in the second notation in the first display area on the display surface and displays the data.
Including a notation conversion device, a display device, and a detection device,
The detection device is a pair of glasses mounted on a user's face and including a camera and a calculation unit,
The camera captures image data by capturing the user's line of sight,
The calculation unit calculates a position of an object appearing in a central portion of the image data as a viewpoint,
The notation conversion device,
Obtained from the detection device, among the respective viewpoints on the display surface of a plurality of users who gaze at the display surface, viewpoint determination means for determining whether a plurality of viewpoints are present in the first display area. ,
Only when there are a plurality of viewpoints in the first display area, except for the first notation, a notation compatible with a plurality of users who pour eyes at the first display area, a second notation. Determining means for determining as
Notation conversion means for converting the first data described in the first display area into second data represented by the second notation,
On the display surface, generating means for generating position information indicating the position of the second display area,
Output means for outputting the second data and the position information to the display device,
The display device has the display surface, displays the first data in the first display area of the display surface, receives the second data and the position information, in the second display area, A conversion display system for displaying second data. A conversion display system that converts the first data written in the first notation into the notation in the second notation in the first display area on the display surface and displays the data.
Equipped with a notation conversion device, a bulletin board and a detection display device,
The bulletin board has the display surface,
The detection and display device is a glasses-type display device that is mounted on a user's face and includes a camera, and the camera captures an image of a user's line of sight to acquire image data. Calculate the position of the object reflected in the center of the data as a viewpoint, display the image data,
The notation conversion device,
Viewpoint determination means for acquiring from the detection display device and determining whether or not a plurality of viewpoints are present in the first display area among respective viewpoints on the display surface of a plurality of users who gaze at the display surface. When,
Only when there are a plurality of viewpoints in the first display area, except for the first notation, a notation compatible with a plurality of users who pour eyes at the first display area, a second notation. Determining means for determining as
Notation conversion means for converting the first data described in the first display area into second data represented by the second notation,
On the display surface, generating means for generating position information indicating the position of the second display area,
Output means for outputting the second data and the position information to the detection display device,
The conversion display system, wherein the detection display device receives the second data and the position information, and displays the second data in the second display area indicated by the position information. A control method used in a notation conversion device that converts first data written in a first notation in a first display area on a display surface into a notation in a second notation,
Of the respective viewpoints on the display surface of the plurality of users who gaze at the display surface, a viewpoint determination step of determining whether a plurality of viewpoints are present in the first display region,
Only when there are a plurality of viewpoints in the first display area, except for the first notation, a notation compatible with a plurality of users who pour eyes at the first display area, a second notation. A determining step to determine as
The notation conversion step of converting the first data described in the first display area into second data represented by the second notation,
On the display surface, a generating step of generating position information indicating the position of the second display area,
Outputting the second data and the position information to an external display device in order to display the second data in the second display area. Computer readable recording a computer program for control used in a notation conversion device that converts first data written in a first notation into a second notation in a first display area on a display surface. A recording medium,
In the notation conversion device that is a computer,
Of the respective viewpoints on the display surface of the plurality of users who gaze at the display surface, a viewpoint determination step of determining whether a plurality of viewpoints are present in the first display region,
Only when there are a plurality of viewpoints in the first display area, except for the first notation, a notation compatible with a plurality of users who pour eyes at the first display area, a second notation. A determining step to determine as
The notation conversion step of converting the first data described in the first display area into second data represented by the second notation,
On the display surface, a generating step of generating position information indicating the position of the second display area,
An output step of outputting the second data and the position information to an external display device in order to display the second data in the second display area. Medium.

Images