JP2016151938A - Information processing system, terminal device, glass type device, processing mode execution method, processing execution program of terminal device side, and processing mode execution program of glass type device side - Google Patents

Abstract

PROBLEM TO BE SOLVED: To uniquely, and furthermore quickly and appropriately select an objective processing mode from a plurality of processing modes in an information processing system composed of a terminal device and a glass type device, and to execute the processing mode.SOLUTION: A menu associating a plurality of executable processing modes with index information is displayed on a lens part 2R of a glass type device 2, and users are notified of the menu. The user takes a picture of a symbol in accordance with the index information corresponding to an objective processing mode through a camera unit of the glass type device 2. A picture-taken image is transmitted to a portable terminal device 1, where the symbol in accordance with the index information undergoes image recognition, the index information corresponding to the symbol is specified, and a processing mode in accordance with the specified index information is executed.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a system including a terminal device such as a high-function mobile phone terminal and a glass device, an apparatus and method used in the system, and a program used in the apparatus.

  Conventionally, various inventions using eyeglass-type display devices have been made. For example, Patent Document 1 described later discloses an invention related to a navigation device including a glasses-type display device. Further, Patent Document 2 described later discloses an invention that translates and outputs character information captured through a camera attached to eyeglasses. When the spectacles are mounted on the user's head so as to be positioned in front of the user's eyes, the direction of the spectacles matches the direction of the user's line of sight. Moreover, when the lens part of the glasses is used as a display (display device), display information can be reliably provided to the user. In recent years, a glass (glasses) type device called a glass type smart device, which is a further development of the conventional eyeglass type display device, has been provided by taking advantage of the characteristics of such glasses.

  Recent glass-type devices can provide various processing modes (various functions) by cooperating with a terminal device such as a high-function mobile phone terminal called a smart phone or the like. For example, information on pedestrian navigation can be displayed, or character recognition can be performed to display a translation result. In addition, the glass-type device and the terminal device collaborate to display the face information by recognizing the face, the lyrics of the music being played, the contents of the incoming mail, And display information. It is also possible to shoot moving images and still images by using the camera unit mounted on the glass device. In this case, video and still images can be taken from the user's perspective, and the user does not need to hold the camera in his / her hand, so safe and free shooting is possible even when walking or moving on a bicycle, for example. It is.

JP 2005-037181 A JP 2002-207732 A

  As described above, an information processing system including a glass-type device and a terminal device such as a high-function mobile phone terminal can provide various processing modes, and the number of processing modes that can be provided will increase. For this reason, in an information processing system including a terminal device and a glass-type device, it is important to improve usability so that a user can easily and reliably select and execute a target processing mode. . There are various methods for inputting an instruction to select a processing mode to be executed for an information processing system including a terminal device and a glass-type device.

  For example, it is conceivable to input a selection instruction by a voice command, but it may be embarrassed in a situation where there are many people around and may be annoying in a quiet place. Although it is conceivable to perform a selection operation on a glass-type device, the number of physical operations is limited because a large number of operation buttons cannot be provided on the glass-type device because of its size. Although it is conceivable to use “blink” as selection information, it is necessary to provide a camera for detecting “blink”, and even if “blink” can be detected, there are only options for opening and closing.

  Of course, it is a common method to input a selection instruction to the terminal device, but the line of sight must be directed to the terminal device, which hinders the rapid use of the system composed of the terminal device and the glass device. There may be cases. Although it is conceivable to input a selection instruction through the attached remote control device, adding an additional remote control device to the information processing system composed of the portable terminal and the glass-type device leads to high costs and is used to operating the remote control device. In the meantime, the line of sight must be directed to the remote control device, and it is thought that there is a case where the rapid use of the system may be hindered.

  In addition, it is conceivable that an acceleration sensor is provided in the glass type device and a selection instruction is input depending on the state of swinging up and down and left and right of the user wearing the glass type device. The number of states that can be expressed is limited. It is also conceivable that an executable function (processing mode) can be selected according to the number of swings. However, as the number of functions that can be executed increases, a situation in which a large number of heads must be shaken in order to execute the target function may occur, so that a quick selection input may not be performed. Further, there is a possibility that a swing performed regardless of the selection input is erroneously recognized as a swing of the selection input.

  In view of the above, the present invention is an information processing system composed of a terminal device and a glass-type device. The target processing mode is uniquely, quickly and appropriately selected from a plurality of executable processing modes. The purpose is to be able to select and execute.

In order to solve the above-mentioned problem, an information processing system according to the invention described in claim 1
An information processing system formed by a terminal device and a glass-type device,
The terminal device
Terminal-side communication means for communicating with the glass-type device;
A storage means for storing a plurality of executable processing modes, index information corresponding to each of the plurality of processing modes, and a symbol corresponding to the index information;
A menu in which the processing mode and the index information are associated with each other is formed according to the information stored in the storage unit, and is controlled to be provided and displayed on the glass-type device through the terminal-side communication unit. Menu display control means;
Recognizing the symbol corresponding to the index information included in the image and identifying the index information by performing image recognition on the image transmitted from the glass device received through the terminal-side communication unit Specific means,
Mode control means for controlling to execute a processing mode corresponding to the index information specified by the specifying means, and
The glass-type device is
Glass side communication means for communicating with the terminal device;
Display processing means for displaying the menu received through the glass side communication means;
Imaging means for imaging the symbol according to the index information associated with the target processing mode in the displayed menu;
Transmission control means for controlling the image picked up by the image pickup means to be transmitted to the terminal device through the glass side communication means.

  According to the information processing system of the first aspect of the present invention, the system includes a terminal device and a glass-type device, the terminal device includes terminal-side communication means, and the glass-type device includes glass-side communication means. Can communicate with each other. The terminal device stores a plurality of processing modes that can be executed in the storage unit, index information corresponding to each of the plurality of processing modes, and a symbol corresponding to (corresponding to) the index information. doing.

  In the terminal device, the menu display control unit functions, forms a menu in which the processing mode and the index information are associated with each other according to the information stored in the storage unit, and provides and displays the menu on the glass device. The specifying means recognizes a symbol corresponding to the index information included in the image by performing image recognition on the image transmitted from the glass device, specifies the index information corresponding to the symbol, and performs mode control. The means executes a processing mode corresponding to the index information specified by the specifying means.

  On the other hand, the glass-type device notifies the user of an executable processing mode and index information by performing a process in which the display processing unit displays the received menu from the terminal device. As a result, the user takes an action so that the symbol corresponding to the index information associated with the target processing mode in the displayed menu is imaged through the imaging unit. In the glass type device, the transmission control unit transmits an image captured through the imaging unit to the terminal device.

  In this way, a menu in which a plurality of processing modes executable in the system are associated with the index information is notified to the user of the system through the glass device. The user of the system images the symbol corresponding to the index information corresponding to the target processing mode through the imaging unit of the glass device. The captured image is sent to the terminal device, and a symbol corresponding to the index information is image-recognized to identify the index information. Then, a processing mode corresponding to the specified index information is executed.

  That is, the user of the system can accurately grasp the index information corresponding to the target processing mode in the displayed menu. The user can execute the processing mode intended by the user in the system simply by imaging the symbol corresponding to the index information corresponding to the target processing mode through the imaging means of the glass-type device.

  According to the present invention, in an information processing system including a terminal device and a glass-type device, a target processing mode is uniquely selected from a plurality of executable processing modes, and is executed promptly and appropriately. it can.

It is a figure for demonstrating the whole structure of the information processing system of embodiment. 3 is a block diagram for explaining a configuration example of a mobile terminal device 1. FIG. It is a figure for demonstrating the example of the stored data of 105 A of symbol tables. It is a figure for demonstrating the example of the storage data of the processing mode table 105B. 4 is a block diagram for explaining a configuration example of a control mechanism unit 21 provided in an ear hooking part (temple part) of the glass device 2. FIG. 5 is a flowchart for explaining processing in a case where a target processing mode is executed in response to selection input from a user in the information processing system. It is a flowchart following FIG. It is a figure for demonstrating the example of the image displayed on the lens part 2R of the glass-type device 2, and the image | photographed image. It is a figure for demonstrating the example of the image displayed on the lens part 2R of the glass-type device 2, and the image | photographed image.

  Hereinafter, an embodiment of a system, apparatus, and method of the present invention will be described with reference to the drawings.

[Entire configuration of information processing system]
FIG. 1 is a diagram for explaining the overall configuration of the information processing system of this embodiment. As shown in FIG. 1, the information processing system of this embodiment includes a mobile terminal device 1 and a glass smart device (hereinafter referred to as a glass device) 2. In this embodiment, the mobile terminal device 1 and the glass device 2 can communicate with each other by short-range wireless communication.

  The mobile terminal device 1 is configured by mounting various application software on an information processing device called a high function mobile phone terminal called a smartphone or a tablet PC (Personal Computer), for example, and can execute a plurality of processing modes. It is possible. Some of the plurality of processing modes can be executed by the mobile terminal device 1 alone, while others can be executed in cooperation with the glass-type device 2.

  The glass-type device 2 has a glass (glasses) appearance, but a control mechanism unit 21 including a control unit, a switch 204A, a camera unit 207, a display processing unit, a projection unit 206, and the like is provided in an ear hooking unit (temple unit). It is configured. By providing the control unit with the glass-type device 2, the control unit can execute application software and cooperate (cooperate) with the mobile terminal device 1.

  In this embodiment, the switch 204A of the glass-type device 2 is for receiving an operation input from the user as necessary, such as a process start instruction and a confirmation input. Further, the projection unit 206 of the glass device 2 displays various display information on the specially coated lens unit 2R of the glass device 2 by displaying it. Thereby, the user wearing the glass device 2 can visually recognize the display information displayed on the lens unit.

  Moreover, in the glass type device 2 of this embodiment, in order to simplify the description, the control mechanism unit 21 has an ear hooking portion (temple portion) on the left side (the right side of the user when worn by the user). It is the composition provided in. However, it is not limited to this. For example, it is also possible to display the display information in a range that the user can visually recognize with both eyes by providing the projection unit 206 on both the left and right ear hooks and displaying the display information on both the left and right lens units 2R and 2L. It is.

  Similarly, the camera unit 207 can be provided on both the left and right ear hooks, and an image equivalent to the image visually recognized by the user with both eyes can be formed from the captured images taken by both the left and right camera units. Of course, the switch 204A may be provided in both the left and right ear hooks, and any switch 204A may be operated.

  And in the information processing system which consists of the portable terminal device 1 and the glass-type device 2, the processing mode to be executed from among a plurality of processing modes that can be executed by the portable terminal device 1 is unambiguously, quickly and appropriately. A function that allows selection is provided. The outline of the function will be described. First, a menu in which a plurality of processing modes that can be executed in the information processing system and index information corresponding to the plurality of processing modes in a one-to-one correspondence are displayed. Through the system user. In this embodiment, the index information corresponding to the plurality of processing modes on a one-to-one basis is a number (number).

  The user of the system images the symbol corresponding to the index information corresponding to the target processing mode through the imaging unit of the glass device. The captured image is sent from the glass-type device 2 to the mobile terminal device 1, and the mobile terminal device 1 recognizes an image of a symbol corresponding to the index information and specifies the index information. And the portable terminal device 1 performs the processing mode according to this specified index information.

  As described above, the user of the system simply captures an image of the symbol corresponding to the index information corresponding to the target processing mode through the camera unit 207 of the glass-type device 2, so that the user can The target processing mode can be executed.

  For this reason, each of the portable terminal device 1 and the glass-type device 2 includes a control unit including a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. Application software can be executed and linked to each other. Below, after demonstrating the structural example of the portable terminal device 1 and the glass-type device 2 which comprise the information processing system of this embodiment, the process performed with the said information processing system is demonstrated concretely.

[Configuration Example of Portable Terminal Device 1]
FIG. 2 is a block diagram for explaining a configuration example of the mobile terminal device 1 used in the information processing system of this embodiment. In FIG. 1, a short-range transmission / reception antenna 101 </ b> A and a short-range communication unit 101 are for communicating with the glass-type device 2. For this reason, in the near field communication part 101, the same communication system as the near field communication part provided in the glass type device 2 side mentioned later is used. For example, various communication methods such as a Bluetooth (registered trademark) standard that can maintain a communication distance of at least several meters can be used.

  The control unit 102 realizes a function of controlling each unit of the mobile terminal device 1. The storage device 103 includes, for example, a nonvolatile memory as a recording medium, and realizes an information storage holding function. The storage device 103 stores various programs, various data necessary for processing, output messages, and the like. The storage device 103 is also used as a work area for temporarily storing intermediate results of various processes. The operation unit 104 includes a power on / off switch, several function buttons, and the like.

  The table storage unit 105 is configured by a recording medium such as a nonvolatile memory, for example, and stores and holds a symbol table 105A and a processing mode table 105B. Here, the symbol table associates index information (numbers) associated with a plurality of processing modes executable in the mobile terminal device 1 and symbols (image data) corresponding to (corresponding to) each index information. And keep it in memory. FIG. 3 is a diagram for explaining an example of data stored in the symbol table 105A.

  In the symbol table shown in FIG. 3, columns of “No.”, “Symbol image 1”, “Symbol image 2”, “Symbol image 3”, “Symbol image 4”,. In the “No.” column, a number (number) is stored as index information corresponding to a processing mode executable in the mobile terminal device 1. Symbols (image data) corresponding to the index information “No.” are stored in the “symbol image 1”, “symbol image 2”, “symbol image 3”, “symbol image 4”,. The

  Specifically, for the number “1” as the index information, the symbol corresponding to the index information is “an image of a hand with the palm of the hand pointing up”, “number” “1 image”, “1 stick image”, and “1 Chinese character image” are associated with each other. In this specification, “fronting the palm toward the front” means the direction of the camera unit of the glass-type device 2, and “pointing the palm toward the front” means the camera of the glass-type device 2. It means to turn to the club.

  Also, for the number “2” as index information, the symbol corresponding to this index information is “an image of a hand with the forefinger and middle finger raised with the palm facing forward”, “number 2 ”,“ 2 stick images ”, and“ Chinese numeral 2 images ”are associated with each other. Similarly, for the number “3” as index information, the symbol corresponding to the index information is “an image of a hand with the first finger, middle finger, and ring finger raised with the palm facing forward”, “ “Number 3 image”, “Triangle image”, and “Chinese number 3 image” are associated with each other.

  In addition, for the number “4” as index information, a symbol corresponding to the index information is “an image of a hand with a forefinger, middle finger, ring finger, and little finger raised with the palm facing forward”, “Number 4 image”, “Rectangle image”, and “Chinese number 4 image” are associated with each other. Similarly, for the number “5” as the index information, the symbol corresponding to the index information is “the finger with the palm, the middle finger, the ring finger, the little finger, and the thumb with the thumb up. "Image", "Number 5 image", "Pentagon image", and "Chinese number 5 image" are associated with each other.

  For the number “6” as index information, the symbol corresponding to this index information is “an image of a hand with the first finger up with the back of the hand facing forward”, “number 6” “Image”, “Hexagonal image”, and “Chinese numeral 6 image” are associated with each other. In this specification, “before facing the back of the hand” means the direction of the camera part of the glass-type device 2, and “turning the back of the hand toward the front” means the camera of the glass-type device 2. It means to turn to the club.

  Also, for the number “7” as index information, the symbol corresponding to the index information “an image of a hand with the first and middle fingers raised with the back of the hand facing forward”, “number 7 ”,“ Heptagon image ”, and“ kanji number 7 image ”are associated with each other. Similarly, for the number “8” as index information, the symbol corresponding to this index information is “an image of a hand with the first finger, middle finger, and ring finger raised with the back of the hand facing forward”, “ “Number 8 image”, “octagon image”, and “Chinese number 8 image” are associated with each other.

  For the number “9” as index information, the symbol corresponding to this index information is “an image of a hand with the first finger, middle finger, ring finger, and little finger raised with the back of the hand facing forward”, “Number 9 image”, “Nutagon image”, and “Chinese number 9 image” are associated with each other. Similarly, for the number “10” as index information, the symbol corresponding to the index information is “the hand of the first finger, middle finger, ring finger, pinky finger and thumb with the back of the hand facing forward. “Image”, “10 number image”, “decagonal image”, and “10 Chinese number image” are associated with each other.

  The index information “No.” between 1 to 5 and 6 to 10 differs depending on whether the hand symbol has the palm toward the front or the back of the hand. That is, even if the finger state is the same, the difference between whether the palm is facing forward or the back of the hand is recognized, and the corresponding index information (mode No.) can be uniquely identified. Yes.

  As described above, the symbol represented by the user's hand has a correspondence relationship between the index information number and the number of standing fingers (the number represented by the finger). For this reason, the number as index information and the symbol represented by the hand can be associated intuitively.

  FIG. 4 is a diagram for explaining an example of data stored in the processing mode table 105B stored and held in the table storage unit 105. In the processing mode table, columns of “No.”, “TOP”, “1 Navi”, “2 Recognition”, “3 Information”, and “4 Shooting” are provided. In the “No.” column, a number (number) is stored as index information corresponding to a processing mode executable in the mobile terminal device 1.

  Each of “TOP”, “1 navigation”, “2 recognition”, “3 information”, and “4 shooting” corresponds to the type of menu used in the information processing system of this embodiment. That is, “TOP” is a top menu (initial menu), and “1 Navi” is a menu related to a navigation mode (navigation mode). “2 recognition” is a menu related to the image recognition mode, “3 information” is a menu related to the information providing mode, and “4 shooting” is a menu related to the shooting mode (camera mode).

  In the top menu that is initially displayed when the portable terminal device 1 and the glass device 2 constituting the information processing system of this embodiment are turned on and a predetermined menu display operation is performed, the top menu has six modes. Selection is made possible. Here, the six modes are navigation (navigation mode), recognition (image recognition mode), information (information provision mode), shooting (shooting mode), translation (translation) shown in the “TOP” column shown in FIG. Mode) and karaoke (karaoke mode).

  As can be seen from the processing mode table shown in FIG. 4, in the top menu, the number “1” is associated with the navigation (navigation mode) as index information, and the number “2” as the index information for recognition (image recognition mode). Are associated. Similarly, in the top menu, information (information provision mode) is associated with the number “3” as index information, and shooting (shooting mode) is associated with the number “4” as index information. Similarly, in the top menu, the number “5” is associated with the translation (translation mode) as index information, and the number “6” is associated with the karaoke (karaoke mode) as index information.

  When the navigation (navigation mode) whose index information is “1” is selected in the top menu, a menu related to the navigation mode corresponding to “1 navigation” described above is further displayed, and pedestrian navigation, car navigation, bicycle navigation is displayed. Any processing mode of the motorcycle navigation is made selectable. In the menu related to the navigation mode corresponding to “1 Navi”, the number “1” is associated with the pedestrian navigation mode as index information, and the number “2” is associated with the vehicle navigation mode as index information. Similarly, in the menu related to the navigation mode corresponding to “1 Navi”, the number “3” is associated with the bicycle navigation mode as index information, and the number “4” is associated with the motorcycle navigation mode as index information. Yes.

  In addition, when recognition (image recognition mode) with index information “2” is selected in the top menu, a menu related to the image recognition mode corresponding to “2 recognition” described above is further displayed, and face recognition, building recognition, and signboard are displayed. Each mode of recognition and timetable recognition can be selected. In the menu relating to the image recognition mode corresponding to “2 recognition”, the face recognition mode is associated with number “1” as index information, and the building recognition mode is associated with number “2” as index information. Similarly, in the menu related to the image recognition mode corresponding to “2 recognition”, the number “3” is associated with the signboard recognition mode as index information, and the number “4” is associated with the timetable recognition mode as index information. It has been.

  In the top menu, the index information “3” (information providing mode) and the shooting information (shooting mode) menu with index information “4” are indexed for each selectable mode belonging to those menus. Information is associated. That is, the menu to be used is determined by the index information corresponding to each mode in the top menu. The processing mode to be executed is determined by the index information corresponding to each mode in the menu used.

  In the processing mode table shown in FIG. 4, it is assumed that there is no lower menu in translation (translation mode) and karaoke (karaoke mode) which are selection items of the top menu. Therefore, when the translation (translation mode) with the index information (No.) “5” is selected in the top menu, the translation mode is executed, and the karaoke (karaoke mode) with the index information (No.) “6” is executed. ) Is selected, the karaoke mode is executed.

  Further, as shown in FIG. 2, the mobile terminal device 1 includes a map DB 106, a network DB 107, an image DB 108, a translation dictionary DB 109, a music data folder 110, and a POI (point of interest) DB 111. “DB” is an abbreviation for database. The map DB 106 stores and holds map data for displaying a map used for navigation processing. The network DB 107 stores and holds node data including node ID, latitude / longitude, and node type, and link data including link ID, node IDs at both ends of the link, link cost, link type, and the like. The network data is mainly used in route search processing.

  The map DB 106 and the network DB 107 store map data and network data according to the purpose (purpose) such as for pedestrians, cars, bicycles, and motorcycles. For this reason, the mobile terminal device 1 can execute navigation modes such as a pedestrian navigation mode, an automobile navigation mode, a bicycle navigation mode, and a motorcycle navigation mode.

  The image DB 108 stores and holds image data used when recognition (image recognition mode) is executed and data associated with the image data in association with each other. Specifically, in the image DB 108, the face image and person attribute information (profile information, etc.) of the partner who may face each other, the image of the main building in each region and the related information on the building, the main signboard in each region And the related information of the signboard, the timetable image of the main station and the related information, and the like are stored. The translation dictionary DB 109 stores and holds translation dictionary data of various languages used when the translation mode is executed. The music data folder 110 stores and holds music data of music and associated data such as lyrics data in association with each other.

  The POIDB 111 stores and holds peripheral information used when the peripheral information mode is executed. Specifically, POIDB 111 stores and holds base information that is information about various stores and various facilities specified by latitude and longitude, or various bases such as topographic feature points. For example, if there are bases such as landmark buildings or various facilities on the map, detailed information such as what kind of base the base is and what features it has is arranged Yes.

  The GPS unit 112 and the GPS antenna 112A realize a function of accurately acquiring (positioning) the current position of the own device by receiving and analyzing transmission signals (positioning information) from a plurality of artificial satellites. Although not shown, a sensor unit and a time information providing unit are also provided. The sensor unit includes various sensors such as a six-axis sensor and a geomagnetic sensor (orientation sensor), and can be used when an autonomous navigation function is used. The time information providing unit provides information related to the time such as the current date, current day of the week, and the current time, and also realizes a so-called timer function that measures the time from the instructed time.

  In addition, the transmission / reception antenna 113A and the wireless communication unit 113 realize a communication function through a mobile phone network or the Internet. Thereby, for example, the control unit 102 accesses a predetermined map server on the Internet in accordance with an instruction from the user, and can receive provision of the latest map data, the latest network data, POI data, and the like. The control unit 102 can store and use the latest map data and the latest network data provided in the map DB 106, the network DB 107, and the POIDB 111. The mobile terminal device 1 can also transmit various information acquired by the mobile terminal device 1 to a server or the like through the wireless communication unit 113 and the transmission / reception antenna 113A. Of course, it is also possible to send and receive e-mails and make voice calls.

  The touch panel 116 includes a display unit 114 such as an LCD (Liquid Crystal Display) and a touch sensor 115, and realizes a function as an information display unit and a function as an information input unit. When the navigation function is executed, a guidance map is displayed on the display unit 114, and the route obtained as a result of the route search and the current position of the own device are displayed on the map to provide route guidance (navigation). Is done.

  The sound processing unit 117 and the speaker 117S or the sound output terminal 117T realize a function of emitting an alarm sound or a voice message in accordance with the control of the control unit 202. An earphone is connected to the audio output terminal 117T. For example, when the navigation function is executed, voice message data is supplied to the voice processing unit 117 through the control unit 102, where it is converted into an analog voice signal, and a voice message is sent from the earphone connected to the speaker 117S or the voice output terminal 117T. Sound is emitted.

  When an earphone is connected to the audio output terminal 117T, the audio processing unit 117 does not supply an audio signal to the speaker 117S, and supplies an audio signal only to the earphone via the audio output terminal 117T. For this reason, the audio processing unit 117 can detect whether an earphone is connected to the audio output terminal 117T, and can also notify the control unit 102 of the detection result.

  Under the control of the control unit 102, the menu display control unit 121 forms a menu in which a plurality of processing modes that can be executed in the mobile terminal device 1 are associated with index information when an event for displaying the menu occurs. . Then, the menu display control unit 121 transmits the formed menu to the glass type device 2 through the short-range communication unit 101 and the short-range transmission / reception antenna 101A, and processes the menu to be displayed on the lens unit of the glass type device 2. . As will be described later, the menu is formed based on data stored in the processing mode table 105B of the table storage unit 105, and index information is associated with each of a plurality of executable processing modes.

  The index specifying unit 122 is photographed by the camera unit of the glass type device 2 transmitted from the glass type device 2 after a predetermined menu is provided to the glass type device 2 and displayed by the function of the menu display control unit 121. The supplied image (image data) is supplied. Then, the index specifying unit 122 performs image recognition on the supplied image, and recognizes a symbol corresponding to the index information associated with the processing mode that can be executed by the mobile terminal device 1 included in the image. Identify information. Then, the index specifying unit 122 notifies the control unit 102 of the specified index information.

  The mode control unit 123 refers to the processing mode table 105B of the table storage unit 105 based on the index information specified by the index specifying unit 122 under the control of the control unit 102, and performs processing corresponding to the specified index information. Control each part to execute the mode. The mode control unit 123 includes a processing unit corresponding to a processing mode to be executed among the navigation processing unit 131, the recognition processing unit 132, the translation processing unit 133, the music lyrics processing unit 134, the information provision processing unit 135, and the camera processing unit 136. The target processing mode is executed by controlling.

  The navigation processing unit 131 executes a navigation mode selected by the user among the pedestrian navigation mode, the car navigation mode, the bicycle navigation mode, and the motorcycle navigation mode, and includes a route search function and a route guidance function. The navigation processing unit 131 accepts information such as a departure point, a transit point, and a destination input through the touch panel 116 to perform a route search, and displays a search result on the display unit 114 to set a route. Then, a guidance message is formed using characters, arrows, etc. so that the user can move according to the set route, and the guidance message is provided to the glass device 2 for display, and route guidance is performed.

  The recognition processing unit 132 executes a recognition mode selected by the user among the above-described face recognition mode, building recognition mode, signboard recognition mode, and timetable recognition mode, and is transmitted from the glass-type device 2. An image (image data) taken through the camera unit of the glass-type device 2 is supplied. Then, the recognition processing unit 132 extracts a human face image, a building image, a signboard image, and a timetable image included in the image from the glass-type device 2, and collides with the image data stored in the image DB 108. In addition, the attribute information and the related information about the matched images are supplied to the glass device 2 and displayed.

  The translation processing unit 133 executes the above-described translation mode, and receives supply of an image (image data) captured through the camera unit of the glass device 2 transmitted from the glass device 2. Then, the translation processing unit 133 extracts character information included in the image from the glass device 2. It recognizes (identifies) what word the character information is and translates the character information into a predetermined language. Thereafter, the translation processing unit 133 performs processing for supplying the translation result to the glass-type device 2 and displaying it.

  The song lyrics processing unit 134 executes the karaoke mode (lyric display mode) described above, and reads and plays the song data instructed from the song data folder 110 in response to an instruction input from the user received through the touch panel 116. Process. The music data to be reproduced is emitted from the earphone connected to the speaker 117S and the audio output terminal 117T through the audio processing unit 117 so that the user can listen to it. At the same time, the song lyrics processing unit 134 performs processing for supplying the lyrics data of the song being played back to the glass-type device 2 for display output.

  The information provision processing unit 135 executes the weather mode, the time mode, the schedule mode, the peripheral information mode, and the incoming mail mode of the information provision mode described above. The information provision processing unit 135 acquires necessary information from information stored in the storage device 103 or POIDB, information disclosed on the Internet, information provided by the time information providing unit of the own device, and the like. And the information provision process part 135 performs the process which supplies the acquired information to the glass-type device 2, and outputs it.

  The camera processing unit 136 executes a camera mode (still image shooting mode) and a video mode (moving image shooting mode) of the above-described shooting modes. The camera processing unit 136 transmits a still image (still image data) or a moving image (moving image data) captured through the camera unit of the glass device 2 transmitted from the glass device 2 to a predetermined image in the storage device 103. Processing to record in the storage area is performed.

[Configuration Example of Control Mechanism 21 of Glass Device 2]
FIG. 5 is a block diagram for explaining a configuration example of the control mechanism unit 21 provided in the ear hooking portion (temple portion) of the glass device 2 having the appearance shown in FIG. In FIG. 5, the short-range transmission / reception antenna 201 </ b> A and the short-range communication unit 201 are for performing communication with the mobile terminal device 1. For this reason, the near field communication unit 201 has the same communication method as the near field communication unit 101 of the mobile terminal device 1 described above.

  The control unit 202 realizes a function of controlling each unit of the glass device 2. The storage device 203 includes, for example, a nonvolatile memory as a recording medium, and realizes an information storage holding function. The storage device 203 stores various programs and various data necessary for processing. The storage device 203 is also used as a work area for temporarily storing intermediate results of various processes. The operation unit 204 includes a power on / off switch, the above-described switch 204A, several function buttons, and the like.

  Under the control of the control unit 202, the display processing unit 205 forms a display signal to be supplied to the projection unit 206 from the display data provided thereto, and supplies this to the projection unit 206. The projection unit 206 performs a process of projecting display information corresponding to the display signal from the display processing unit 205 onto the lens unit 2L of the glass device 2. Thereby, display information is displayed on the lens part 2L of the glass-type device 2, and a user who wears the glass-type device 2 on the head so that the lens part is positioned in front of the eyes can be seen.

  As shown in FIG. 1, the camera unit 207 is provided at the front end portion (end portion on the lens unit side) of the ear hooking portion (temple portion) of the glass device 2. Then, when the user wears the glass device 2 on the head so that the lens portion is positioned in front of the eyes, the camera unit 207 captures an image in the user's line of sight and captures it as image data. Realize the function. A captured image (image data) captured through the camera unit 207 can be transmitted to the mobile terminal device 1 through the short-range communication unit 201 and the short-range transmission / reception antenna 201A.

  The sensor unit 208 includes at least a geomagnetic sensor (orientation sensor) for specifying the orientation of the camera unit 207. In addition to the geomagnetic sensor, for example, a 6-axis sensor may be provided.

[Processing of information processing system]
Next, the process performed in the information processing system configured by the mobile terminal device 1 described with reference to FIGS. 2 to 4 and the glass device 2 described with reference to FIG. 5 will be specifically described. As described above, the information processing system of this embodiment can execute various processing modes as described with reference to FIG. 4 by the cooperation of the mobile terminal device 1 and the glass device 2. The user can select which processing mode to execute uniquely and quickly and appropriately without hesitation.

  FIG. 6 and FIG. 7 are flowcharts for explaining processing in the case of receiving a selection input from a user and executing a target processing mode in the information processing system of this embodiment. 8 and 9 show the image displayed on the lens unit 2R of the glass-type device 2 and the camera unit of the glass-type device 2 in the processing shown in FIGS. FIG. 207 is a diagram for describing an example of a symbol represented by a user's hand photographed through 207 and a predetermined target object.

  The processing of the flowcharts shown in FIG. 6 and FIG. 7 is a form in which the glass type device 2 is controlled from the portable terminal device 1 at predetermined timings after both the portable terminal device 1 and the glass type device 2 are powered on. Is executed. And the control part 102 of the portable terminal device 1 receives the operation input which instruct | indicates a top menu display (step S101). The operation input includes an operation for the mobile terminal device 1 and an operation for the glass device 2.

  Here, the predetermined operation on the mobile terminal device 1 instructing to perform menu display is an operation on a predetermined function key of the mobile terminal device 1 or a predetermined operation key realized as a software key by the touch panel 116. Further, the predetermined operation for the glass-type device 2 instructing to perform the menu display is, for example, an operation (operation) of pressing the switch 204A or picking up and down the ear-hook portion of the glass-type device 2. The fact that the switch 204 </ b> A of the glass device 2 is pressed or that the ear hook part of the glass device 2 is picked up and down is transmitted to the mobile terminal device 1. Note that the sensor unit 208 of the glass-type device 2 can detect that the operation of picking and lifting the ear-hook portion of the glass-type device 2 has been performed.

  Then, in step S101, the control unit 102 of the mobile terminal device 1 determines whether or not a predetermined operation instructing to perform the top menu display has been performed (whether or not present) (step S102). When it is determined in the determination process in step S102 that the predetermined operation for instructing the top menu display is not performed, the control unit 102 ends the processes shown in FIGS. 6 and 7 and performs the next execution timing. Will wait. Also, assume that it is determined in the determination process in step S102 that a predetermined operation for instructing the top menu display has been performed. In this case, the control unit 102 of the portable terminal device 1 and the control unit 202 of the glass device 2 cooperate to perform processing for displaying the top menu on the lens unit 2R of the glass device 2 (step S103).

  In step S <b> 103, the menu display control unit 121 functions under the control of the control unit 102, forms top menu display data with reference to the processing mode table 105 </ b> B of the table storage unit 105, and transmits this to the glass-type device 2. To do. Upon receiving the top menu display data from the portable terminal device 1, the control unit 202 of the glass device 2 supplies the top menu display data to the display processing unit 205, and sends the top menu corresponding to the top menu display data to the lens unit 2 </ b> R through the projection unit 206. Project. As shown in FIG. 8A, a top menu in which numbers (1 to 6) that are index information and processing modes are associated with each other in a one-to-one manner is displayed on the lens unit 2R of the glass device 2.

  Thereby, the user confirms the top menu displayed on the lens unit 2R of the glass-type device 2, and displays the symbol corresponding to the index information associated with the processing mode to be executed on the camera of the glass-type device 2. Shoot through the unit 207. For example, it is assumed that the user who has confirmed the top menu shown in FIG. 8A wants to execute the recognition mode in which the number “2” is associated as index information. In this case, the use is recognized from the displayed top menu that the index information corresponding to the recognition mode is the number “2”.

  For this reason, the user places his / her hand in the state in front of the lens unit 2L of the glass device 2 so that the camera unit 207 of the glass device 2 can shoot, as shown in FIG. , Raise the forefinger and middle finger with the palm toward you. The glass-type device 2 is configured such that the camera unit 207 operates when the top menu is displayed. Through the camera unit 207, an “image with the first finger and middle finger raised with the palm facing forward (displayed by hand). Can be photographed.

  Alternatively, the user positions a description surface such as paper on which the number “2” is written in front of the lens unit 2 </ b> R of the glass device 2 so that the camera unit 207 of the glass device 2 can take a picture. Then, as shown in FIG. 9A, in order to make it possible to specify the number (character) to be image-recognized in the mobile terminal device 1, the position where the number “2” is described is indicated by, for example, a pencil or a finger. To do. As a result, the camera unit 207 can shoot with the number “2” as index information specified.

  As described above, for example, when the top menu is displayed, the glass device 2 of the information processing system according to this embodiment causes the camera unit 207 to function and transmits the captured image to the mobile terminal device 1. The control unit 102 of the mobile terminal device 1 supplies an image (image data) from the glass device 2 to the index specifying unit 122. Then, the control unit 102 controls the index specifying unit 122 to start the image recognition process, and starts the process of recognizing a symbol corresponding to the index corresponding to the processing mode included in the image (step S104).

  In this case, the index specifying unit 122 performs image recognition processing with reference to information in the symbol table 105A of the table storage unit 105. In this embodiment, as shown in FIG. 8A, the index information corresponding to the processing mode is recognized when symbols corresponding to numbers (numbers) 1 to 6 are included. Identify. The symbols corresponding to the index information corresponding to the processing mode include symbols represented by the user's hands, numeric symbols (numeric images), graphic symbols (graphics), as described with reference to FIG. Image), symbols of Chinese numerals (images of Chinese numerals), and the like. When the index specifying unit 122 recognizes (detects) a symbol corresponding to one of the numbers 1 to 6 based on the information in the symbol table described with reference to FIG. 3, the index information corresponding to the recognized symbol Is identified and notified to the control unit 102.

  Then, the control unit 102 determines whether or not the index information corresponding to the processing mode has been recognized (specified) by the index specifying unit 122 based on the notification from the index specifying unit 122 (step S105). Assume that it is determined in the determination process in step S105 that the index information corresponding to the processing mode has been recognized. In this case, the control unit 102 controls the index specifying unit 122 to stop the index information recognition process for menu selection (step S106).

  Thereafter, the control unit 102 controls the menu display control unit 121 to perform processing for notifying the user that the processing mode corresponding to the recognized index information has been selected (step S107). Specifically, in step S107, the menu display control unit 121 uses the index information recognized by the index specifying unit 122 under the control of the control unit 102 to refer to the processing mode table 105B to select which mode. Identify whether it was. The menu display control unit 121 forms the same menu as the menu that has been formed and supplied to the glass-type device 2 and shows the selected item portion indicated by the bold line CS. Then, the menu display control unit 121 transmits the formed menu to the glass device 2 and displays the menu on the lens unit 2L of the glass device 2.

  As a result, in the lens unit 2L of the glass-type device 2, as shown in FIG. 8B or FIG. 9A, the display area portion of the processing mode selected by the user is surrounded by the thick line CS. Displayed. Thereby, it is possible to clearly indicate to the user what processing mode is currently selected in the information processing system. Therefore, the user can grasp whether or not the target processing mode is selected.

  And the control part 102 of the portable terminal device 1 accepts the confirmation input from a user (step S108). This confirmation input is an operation on a predetermined operation key of the mobile terminal device 1 or a predetermined software key of the touch panel 116. Alternatively, if the selection is correct “OK”, the operation of pressing the switch 204A of the glass-type device 2 is pressed, and if the selection is not correct “NG”, the ear-hook portion of the glass-type device 2 is picked and the glass-type device 2 is For example, the operation of moving up and down. Further, when no operation is performed for a certain period of time, it may be regarded as “NG”.

  Then, the control unit 102 determines whether or not the confirmation input received in step S108 is “OK” indicating that the selected processing mode is not wrong (step S109). Assume that it is determined in the determination process in step S109 that the received confirmation input is not “OK”. In this case, the control unit 102 controls the index specifying unit 122 to restart the recognition process of the index information associated with the processing mode (Step S110), and repeats the processing from Step S105. In step S110, the menu display control unit 121 is controlled to form a menu in which the thick line CS is deleted, and the menu is supplied to the glass device 2 and displayed again on the lens unit 2R.

  If it is determined in the determination process in step S109 that the accepted confirmation input is “OK”, the control unit 102 proceeds to the process in FIG. Then, the control unit 102 determines whether or not the lower mode menu exists in the processing mode selected in the top menu based on the information in the processing mode table (FIG. 4) (step S111). That is, in the processing mode table (FIG. 4), it is managed that “1 Navi”, “2 Recognition”, “3 Information”, and “4 Shooting” belonging to the top menu have lower menus. For this reason, when “1 navigation”, “2 recognition”, “3 information”, and “4 shooting” are selected, it can be determined that a lower menu exists.

  Assume that it is determined in the determination process in step S111 that there is no lower menu. In this case, the control unit 102 controls the mode control unit 123 to execute a processing mode corresponding to the index information recognized at the present time and confirmed by the user (step S112). In step S <b> 112, the mode control unit 123 controls the shift of the processing units 131 to 137 that execute the selected processing mode according to the control of the control unit 102, and executes a target process. Then, when the executed processing mode ends, the processing shown in FIGS. 6 and 7 also ends.

  Further, it is assumed that it is determined in the determination process in step S111 that a lower menu exists. In this case, the control unit 102 of the portable terminal device 1 and the control unit 202 of the glass-type device 2 cooperate to display a lower menu on the lens unit 2R of the glass-type device 2 (step S113). ).

  In step S113, the menu display control unit 121 functions under the control of the control unit 102, and refers to the processing mode table 105B of the table storage unit 105 to select a lower menu corresponding to the processing mode selected in the top menu. Display data is formed and transmitted to the glass-type device 2. Upon receiving the lower menu display data from the portable terminal device 1, the control unit 202 of the glass-type device 2 supplies this to the display processing unit 205, and responds to the lower menu display data through the projection unit 206. The menu is projected onto the lens unit 2R.

  In this example, as shown in FIGS. 8B and 9A, it is assumed that “recognition mode” having index information “2” is selected in the top menu. Therefore, in the case of this example, as shown in FIG. 9B, the lens unit 2R of the glass-type device 2 is associated with the number (1-4) as index information and the processing mode on a one-to-one basis. A lower menu corresponding to “2, recognition mode” is displayed.

  Thereafter, the control unit 102 controls the index specifying unit 122 to restart the recognition process of the index information associated with the processing mode (Step S114), and repeats the processing from Step S105. Thereby, in the menu corresponding to the “recognition mode” selected in the top menu (menu below the top menu), the target processing mode can be selected in the same manner as the selection of the processing mode in the top menu.

  Also, assume that it is determined in the determination process of step S105 illustrated in FIG. 6 that the index information corresponding to the processing mode cannot be specified. In this case, although not shown in FIG. 2, the control unit 102 measures the processing time after the index information recognition process is started by the function of the time information providing unit provided in the mobile terminal device 1. Then, it is determined whether or not the processing time has reached a predetermined time or more, that is, whether or not a timeout has occurred (step S115).

  If it is determined in step S115 that the time-out has not occurred, for example, whether or not an event for instructing termination has occurred, for example, a predetermined end key of the mobile terminal device 1 or the glass device 2 is operated. Is determined (step S116). In the determination process of step S116, when it is determined that an event for instructing termination has not occurred, the process from step S105 is repeated.

  If it is determined in step S115 that the time-out has occurred, or if it is determined in step S116 that an event for instructing termination has occurred, predetermined termination processing is performed (step S117). 6. The process shown in FIG. The predetermined end process in step S117 is a process for returning the information processing system to the state before the menu display, such as deleting the menu displayed on the lens unit 2R of the glass device 2.

  As shown in FIGS. 8 and 9, the menu is displayed on the lower side of the lens unit 2R, and the front can be seen through the displayed menu. In other words, the menu display does not greatly obstruct the user's field of view and does not cause a problem in safety. The display position on the lens unit 2R such as a menu can be set to an appropriate position.

[Effect of the embodiment]
In the information processing system of the above-described embodiment, a menu in which a plurality of executable processing modes and index information are associated can be provided to the user through the glass device 2. In this embodiment, since the index information associated with the processing mode is a number (number), the number corresponding to the target processing mode can be recognized immediately. Then, a symbol corresponding to the index information is photographed through the camera unit 207 and the image is recognized, so that the processing mode can be selected quickly and appropriately.

  That is, as described above, the symbol corresponding to the index information is directly related to the number that is the index information. Therefore, the symbol corresponding to the index information is immediately identified, and the identified symbol can be simplified. Can shoot. As described above, since the target processing mode can be specified and executed by the symbol corresponding to the index information, the processing mode can be selected and input directly to the hardware key or software key of the portable terminal device 1 or the glass device 2. No operation is performed. Therefore, in the information processing system including the mobile terminal device 1 and the glass device 2, the target processing mode can be uniquely selected from a plurality of executable processing modes, and can be executed quickly and appropriately. .

[Modifications, etc.]
In the above-described embodiment, as described with reference to FIG. 3, the symbol corresponding to the index information is a symbol represented by the user's hand, a numerical image, a graphic image, or a Chinese numeral image. Although described as being, it is not limited to this. As a symbol corresponding to the index information, an image such as a symbol associated with the processing mode on a one-to-one basis, other characters, or various graphics can be used.

  Canceling, clearing, initial return, starting, and the like may be physical operations on the portable terminal device 1 and the glass-type device 2 or may be instructions by blinking. In addition, as described above, canceling, clearing, initial return, starting, etc. are based on recognition of a predetermined image such as “an image in which all the fingers of the hand are held with the palm toward you (Janken Guo)”. It may be an instruction.

  Further, in the information processing system of the above-described embodiment, the processing modes that can be executed in cooperation with the mobile terminal device 1 and the glass device 2 have been described using the processing mode table shown in FIG. It is not limited to. Processing modes other than those shown in the processing mode table shown in FIG. 4 may be executable.

  In the above-described embodiment, the glass-type device 2 has been described as being displayed on the lens unit 2R by the projection unit 206, but is not limited thereto. Various display information display methods (presentation methods) such as menus in the glass-type device 2 can be used.

  In the above-described embodiment, the information processing system is configured by the mobile terminal device 1 and the glass-type device 2. However, the information processing device is not limited to the mobile terminal device, and may be various information processing devices such as a personal computer. The present invention can be applied to an information processing system including a glass-type device.

[Others]
As can be seen from the description of the embodiment described above, the function of the terminal-side communication means of the terminal device is realized by the short-range communication unit 101 and the short-range transmission / reception antenna 101A, and the function of the storage means is realized by the table storage unit 105, respectively. doing. Also. The function of the menu display control means of the terminal device is realized by the menu display control section 121, the function of the specifying means is realized by the index specifying section 122, and the function of the mode control means is realized by the mode control section 123.

  Further, the functions of the glass side communication means of the glass type device are the short distance communication unit 201 and the short distance transmission / reception antenna 201A, the function of the display processing means is mainly the display processing unit 205, and the function of the imaging means is the camera unit 207. The function of the transmission control means is mainly realized by the control unit 202.

DESCRIPTION OF SYMBOLS 1 ... Portable terminal device 101 ... Short-distance communication part 101A ... Short-distance transmission / reception antenna 102 ... Control part 103 ... Memory | storage device 104 ... Operation part 105 ... Table storage part 106 ... Map DB, 107 ... Network DB 108 ... Image DB, 109 ... Translation Dictionary DB, 110 ... Music Data Folder, 111 ... POIDB, 112 ... GPS Unit, 112A ... GPS Antenna,
DESCRIPTION OF SYMBOLS 113 ... Wireless communication part, 113A ... Transmission / reception antenna, 114 ... Display part, 115 ... Touch sensor, 116 ... Touch panel, 117 ... Sound processing part, 117S ... Speaker, 117T ... Sound output terminal, 121 ... Menu display control part, 122 ... Index specifying unit, 123 ... mode control unit, 131 ... navigation processing unit, 132 ... recognition processing unit, 133 ... translation processing unit, 134 ... music lyrics processing unit, 135 ... information provision processing unit, 136 ... camera processing unit, 2 ... Glass type device, 2R, 2L ... lens unit, 201A ... near-field transmission / reception antenna, 201 ... near-field communication unit, 202 ... control unit, 203 ... storage device, 204 ... operation unit, 205 ... display processing unit, 206 ... projection unit 207: Camera unit 208: Sensor unit

Claims (8)

An information processing system formed by a terminal device and a glass-type device,
The terminal device
Terminal-side communication means for communicating with the glass-type device;
A storage means for storing a plurality of executable processing modes, index information corresponding to each of the plurality of processing modes, and a symbol corresponding to the index information;
A menu in which the processing mode and the index information are associated with each other is formed according to the information stored in the storage unit, and is controlled to be provided and displayed on the glass-type device through the terminal-side communication unit. Menu display control means;
Recognizing the symbol corresponding to the index information included in the image and identifying the index information by performing image recognition on the image transmitted from the glass device received through the terminal-side communication unit Specific means,
Mode control means for controlling to execute a processing mode corresponding to the index information specified by the specifying means, and
The glass-type device is
Glass side communication means for communicating with the terminal device;
Display processing means for displaying the menu received through the glass side communication means;
Imaging means for imaging an image including the symbol corresponding to the index information associated with the target processing mode in the displayed menu;
An information processing system comprising: a transmission control unit configured to control the image captured by the imaging unit to be transmitted to the terminal device through the glass side communication unit. The information processing system according to claim 1,
The information processing system characterized in that the symbol corresponding to the index information is an image of displayed or described information including a number, a symbol, a character, and a figure. The information processing system according to claim 1,
The information processing system according to claim 1, wherein the symbol corresponding to the index information is a symbol expressed by a user's hand. The terminal device of an information processing system formed by a terminal device and a glass-type device,
Terminal-side communication means for communicating with the glass-type device;
A storage means for storing a plurality of executable processing modes, index information corresponding to each of the plurality of processing modes, and a symbol corresponding to the index information;
A menu in which the processing mode and the index information are associated with each other is formed according to the information stored in the storage unit, and is controlled to be provided and displayed on the glass-type device through the terminal-side communication unit. Menu display control means;
Recognizing the symbol corresponding to the index information included in the image and identifying the index information by performing image recognition on the image transmitted from the glass device received through the terminal-side communication unit Specific means,
And a mode control unit that controls to execute a processing mode corresponding to the index information specified by the specifying unit. The glass-type device of the information processing system formed by the terminal device and the glass-type device,
The terminal device includes terminal-side communication means for communicating with the glass device, a plurality of executable processing modes, index information corresponding to each of the plurality of processing modes, and a symbol corresponding to the index information. A menu that associates the processing mode with the index information according to the information stored in the storage means, and stores the information in association with each other. Menu display control means for providing and controlling display on the glass-type device, and image recognition performed on the image transmitted from the glass-type device received through the terminal-side communication means. Recognizing the symbol according to the index information included to identify the index information Are those comprising a constant section, and a mode control means for controlling to execute a processing mode corresponding to the index information specified by said specifying means,
Glass side communication means for communicating with the terminal device;
Display processing means for displaying the menu received through the glass side communication means;
Imaging means for imaging an image including the symbol corresponding to the index information associated with the target processing mode in the displayed menu;
A glass-type device comprising: a transmission control unit configured to control the image captured by the imaging unit to be transmitted to the terminal device through the glass-side communication unit. A processing mode execution method used in an information processing system formed by a terminal device and a glass-type device,
In the terminal device,
The menu display control means stores in a storage means for storing a plurality of executable processing modes, index information corresponding to each of the plurality of processing modes, and a symbol corresponding to the index information so as to be associated with each other. According to the information that is being processed, a menu that associates the processing mode with the index information is formed, and is provided and displayed on the glass-type device through terminal-side communication means for communicating with the glass-type device. A menu display control process for controlling
The identification unit recognizes the symbol corresponding to the index information included in the image by performing image recognition on the image transmitted from the glass-type device received through the terminal-side communication unit, and the index information. A specific process for identifying
A mode control step for controlling the mode control means to execute a processing mode corresponding to the index information specified in the specifying step;
In the glass type device,
A display processing step for displaying the menu received through the glass side communication means for communicating with the terminal device;
An imaging step in which an imaging unit captures an image including the symbol corresponding to the index information associated with the target processing mode in the displayed menu;
And a transmission control step of controlling the transmission control unit to transmit the image captured in the imaging step to the terminal device through the glass side communication unit. A processing mode execution program executed by a computer mounted on the terminal device of the information processing system formed by the terminal device and the glass device,
The menu display control means stores in a storage means for storing a plurality of executable processing modes, index information corresponding to each of the plurality of processing modes, and a symbol corresponding to the index information so as to be associated with each other. According to the information that is being processed, a menu that associates the processing mode with the index information is formed, and is provided and displayed on the glass-type device through terminal-side communication means for communicating with the glass-type device. A menu display control step for controlling
The identification unit recognizes the symbol corresponding to the index information included in the image by performing image recognition on the image transmitted from the glass-type device received through the terminal-side communication unit, and the index information. Specific steps to identify,
The mode control means executes a mode control step for performing control so as to execute a processing mode corresponding to the index information specified in the specifying step. A processing mode execution program on the terminal device side. A processing mode execution program executed by a computer mounted on the glass type device of an information processing system formed by a terminal device and a glass type device,
In the terminal device, the menu display control means stores the plurality of executable processing modes, the index information corresponding to each of the plurality of processing modes, and the symbol corresponding to the index information in association with each other. A menu that associates the processing mode with the index information is formed according to the information stored in the storage means, and is provided to the glass-type device through a terminal-side communication means for communicating with the glass-type device. The menu display control step for controlling the display to be displayed, and the specifying unit performs image recognition on the image transmitted from the glass-type device received through the terminal-side communication unit, thereby including the image included in the image Recognize the symbol according to index information to identify the index information A constant step, the mode control means are adapted to perform a mode control step of controlling to execute a processing mode corresponding to the index information specified by the specifying step,
A display processing step for displaying the menu received through the glass-side communication means for communicating with the terminal device;
An imaging step in which an imaging unit captures an image including the symbol in accordance with index information associated with the target processing mode in the displayed menu;
A transmission control unit that performs a transmission control step of controlling the image captured in the imaging step so as to transmit the image to the terminal device through the glass side communication unit. Mode execution program.

Images