JP2018063722A - Information processing system, information processing device, control method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enhance the convenience of a system for performing an input operation related to a browsed commodity.SOLUTION: An information processing system 3000 includes a display device 3020 and an information processing device 2000. The information processing device 2000 includes a commodity recognition part 2020, a display control part 2040, and an operation recognition part 2060. The commodity recognition part 2020 recognizes a commodity. The display control part 2040 acquires commodity information on the commodity recognized by the commodity recognition part 2020. Further, the display control part 2040 allows the display device 3020 to display an operation screen including the acquired commodity information. The operation recognition part 2060 recognizes the input operation with respect to the operation screen on the basis of the position or movement of an operation body in a captured image generated by a camera.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing system, an information processing apparatus, a control method, and a program.

  A system for browsing information about products is provided. Patent Document 1 discloses an invention of a product information output device. When a user photographs a product with a camera provided in the product information output device, the product information output device acquires product information of the product shown in the image generated by the camera and displays it on the touch panel. By doing so, the user can browse the product information. In addition, the user can ask a question about the product being browsed by operating a touch panel provided in the product information output device or speaking to a microphone provided in the product information output device. .

Patent No. 5220953

  When using the product information output device described above while viewing the product at the store, the user needs to have the product information output device in his / her hand in order to browse the product information and ask questions about the product. Therefore, at least one hand of the user is blocked, and the degree of freedom of the user's action is lowered. In addition, the method of inputting information about a product using a microphone may be difficult to use in a situation where there are people around.

  The present invention has been made in view of the above problems. An object of the present invention is to provide a technique for enhancing the convenience of a system that performs an input operation related to a product being browsed.

  The information processing apparatus according to the present invention includes (1) an object recognition unit that extracts an object recognition area, which is a partial area of a captured image, and recognizes an object included in the extracted object recognition area; Display control means for acquiring object information related to the recognized object and displaying the acquired object information and an operation screen related to the object on a display device; and (3) the position and movement of the operation tool included in the captured image. Operation recognition means for recognizing an input operation on the operation screen based on at least one of them. The object recognition means extracts the object recognition area determined based on the position of the first marker included in the captured image from the captured image. The display control means causes the display device to display the operation screen when the second marker is included in the captured image. Whether at least one of the first marker and the second marker is an image displayed on a display device of a device worn by a user, or is drawn, printed, or attached to the device, It is worn by the user or is part of the user's body.

  An information processing system of the present invention includes the information processing apparatus of the present invention and the display device.

  The control method of the present invention is executed by a computer. The control method includes (1) an object recognition step of extracting an object recognition area which is a partial area of a captured image, and recognizing an object included in the extracted object recognition area; and (2) the recognized A display control step of acquiring object information related to the object and displaying the acquired object information and an operation screen related to the object on a display device; and (3) at least one of a position and a movement of the operating body included in the captured image. And an operation recognition step for recognizing an input operation on the operation screen. In the object recognition step, the object recognition area determined based on the position of the first marker included in the captured image is extracted from the captured image. In the display control step, when the second marker is included in the captured image, the operation screen is displayed on the display device. Whether at least one of the first marker and the second marker is an image displayed on a display device of a device worn by a user, or is drawn, printed, or attached to the device, It is worn by the user or is part of the user's body.

  The program of this invention makes a computer perform each step which the control method of this invention has.

  ADVANTAGE OF THE INVENTION According to this invention, the technique which raises the convenience of the system which performs input operation relevant to the goods currently browsed is provided.

1 is a block diagram illustrating an information processing system according to a first embodiment. It is a figure for demonstrating an information processing system notionally. It is a figure which illustrates the structure of the computer which implement | achieves information processing apparatus. It is a figure which illustrates the head mounted display provided with the camera. 3 is a flowchart illustrating a flow of processing executed by the information processing apparatus according to the first embodiment. It is a figure which illustrates the goods reflected in the predetermined field of a picked-up image. It is a figure which illustrates the display device with which the guide showing a predetermined area is displayed. It is a figure which illustrates merchandise information in a table format. It is a figure which illustrates the template of an operation screen. It is the 1st figure which illustrates the method of determining the position of an operation body. It is the 2nd figure which illustrates the method of determining the position of a control object. It is a figure which illustrates the picked-up image 22 in which a user's finger | toe is reflected. It is a figure which illustrates operation defined based on the position of an operation body. It is a figure which illustrates the case which recognizes input operation which inputs the shape defined by the motion of an operation body. It is a figure which illustrates gesture input. It is a figure which illustrates a mode that the position and movement of an operating body are represented by the relative position and movement in the whole captured image. FIG. 3 is a block diagram illustrating an information processing system according to a second embodiment. 6 is a flowchart illustrating a flow of processing executed by a product recognition unit according to the second embodiment. It is a figure which illustrates the device which displays the image handled as the 1st marker. It is the 1st figure which illustrates a goods recognition field. It is the 2nd figure which illustrates a goods recognition field. It is the 3rd figure which illustrates a goods recognition field. It is a figure of FIG. 4 which illustrates a goods recognition area. It is a figure which illustrates a mode that the attitude | position of a goods recognition area changes according to the inclination of a 1st marker. It is a figure which illustrates the field in which a hand is included. FIG. 6 is a block diagram illustrating an information processing system according to a third embodiment. 10 is a flowchart illustrating a flow of processing executed by the information processing apparatus according to the third embodiment. It is the 1st figure which illustrates the operation screen where a position becomes settled with the position of the 2nd marker. It is the 2nd figure which illustrates the operation screen where a position becomes settled with the position of the 2nd marker. It is the 3rd figure which illustrates the operation screen where a position becomes settled with the position of the 2nd marker. FIG. 5 is a diagram illustrating an operation screen whose position is determined by the position of a second marker. FIG. 10 is a block diagram illustrating an information processing system according to a fourth embodiment. 10 is a flowchart illustrating a flow of processing executed by the information processing apparatus according to the fourth embodiment. It is a figure which illustrates a mode that the image currently displayed on the display apparatus of a device is changed when a vibration is detected by the sensor. It is a figure which illustrates the case which makes a detection target timing the start time of detection target time. It is a figure which illustrates the case which makes detection target timing the end time of detection target time. It is a block diagram which illustrates the composition of the information processing system common to each example. It is a figure which illustrates the utilization environment of the information processing system common to each Example. FIG. 5 is a diagram illustrating an operation screen displayed on the display device in the first embodiment. It is a figure which illustrates order information in a table format. It is a figure which illustrates the structure of the concierge system which provides the concierge service of Example 2. FIG. FIG. 10 is a diagram illustrating a state in which a head mounted display is used in Example 3. It is a figure which illustrates the map of the store displayed on a display apparatus. FIG. 10 is a diagram illustrating an operation screen displayed on a display device in a fourth embodiment. It is a figure which illustrates the display apparatus on which the operation screen for contacting a sales floor person in charge was displayed. FIG. 10 is a diagram illustrating information displayed on a display device in Example 5.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, the same reference numerals are given to the same components, and the description will be omitted as appropriate. In each block diagram, unless otherwise specified, each block represents a functional unit configuration, not a hardware unit configuration.

[Embodiment 1]
FIG. 1 is a block diagram illustrating an information processing system 3000 according to the first embodiment. The information processing system 3000 includes a display device 3020 and an information processing device 2000. The information processing apparatus 2000 includes a product recognition unit 2020, a display control unit 2040, and an operation recognition unit 2060.

  The product recognition unit 2020 recognizes the product. The display control unit 2040 acquires product information related to the product recognized by the product recognition unit 2020. Further, the display control unit 2040 causes the display device 3020 to display an operation screen including the acquired product information. The operation recognizing unit 2060 recognizes an input operation on the operation screen based on the position or movement of the operating tool in the captured image generated by the camera. Here, the operation body is an arbitrary object used by the user for the operation. For example, the operating body is a part of the user's body (such as a finger) or an object (such as a pen) held by the user's body.

  FIG. 2 is a diagram for conceptually explaining the information processing system 3000. In FIG. 2, the user wears a head mounted display 100. The camera 20 is a camera that generates a captured image, and is provided in the head mounted display 100. Further, the camera 20 is provided so as to capture the direction of the user's field of view. In FIG. 2, the display device 3020 is the display device 102 provided in the head mounted display 100.

  The product recognition unit 2020 recognizes the product 10. As a result, the display control unit 2040 displays the operation screen 40 on the display device 102. The operation screen 40 includes product information of the product 10. The product information in this example indicates the product name and price of the product 10. Furthermore, the operation screen 40 includes an image for selecting whether or not to purchase a product.

  In FIG. 2, the operating body is a user's finger 30. The user holds the finger 30 at a position overlapping the YES image. Here, in this example, since the camera 20 captures a scene in the user's field of view, the finger 30 is shown in the captured image generated by the camera 20. The operation recognition unit 2060 recognizes the input operation for selecting YES based on the position of the finger 30 in the captured image. Thereby, the product 10 is registered as a purchase target.

  Here, the operation of the information processing apparatus 2000 described with reference to FIG. 2 is an example for facilitating the understanding of the information processing apparatus 2000, and the operation of the information processing apparatus 2000 is not limited to the above example. Absent. Details and variations of the operation of the information processing apparatus 2000 will be described in the following description.

<Action and effect>
According to the information processing system 3000 of the present embodiment, the operation screen 40 including product information regarding the recognized product is displayed on the display device 3020. Furthermore, an input operation on the operation screen 40 is recognized based on the position or movement of the operating tool shown in the captured image generated by the camera 20. The user can perform an input operation related to the product by moving or stopping the operating tool within the imaging range of the camera 20. Therefore, unlike the case where an input operation must be performed using an input device such as a touch panel, the user does not need to hold the input device in his hand. Therefore, the degree of freedom of the user's action when performing an input operation related to the product being browsed is increased, and the user can easily perform the input operation. Therefore, the information processing system 3000 is more convenient for the user than a system that accepts an input operation via an input device.

  Hereinafter, the information processing system 3000 according to the present embodiment will be described in more detail.

<Example of Hardware Configuration of Information Processing Device 2000>
Each functional component of the information processing apparatus 2000 may be realized by hardware (eg, a hard-wired electronic circuit) that implements each functional component, or a combination of hardware and software (eg: It may be realized by a combination of an electronic circuit and a program for controlling it). Hereinafter, the case where each functional component of the information processing apparatus 2000 is realized by a combination of hardware and software will be further described.

  The computer 1000 is a variety of computers. For example, the computer 1000 is a head mounted display, a personal computer (PC), a server machine, a tablet terminal, or a smartphone. The computer 1000 may be a dedicated computer designed for realizing the information processing apparatus 2000 or a general-purpose computer.

  FIG. 3 is a diagram illustrating a configuration of a computer 1000 that implements the information processing apparatus 2000. The computer 1000 includes a bus 1020, a processor 1040, a memory 1060, a storage 1080, and an input / output interface 1100. The bus 1020 is a data transmission path for the processor 1040, the memory 1060, and the storage 1080 to transmit / receive data to / from each other. However, the method of connecting the processors 1040 and the like is not limited to bus connection. The processor 1040 is an arithmetic processing device such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). The memory 1060 is a memory such as a RAM (Random Access Memory) or a ROM (Read Only Memory). The storage 1080 is a storage device such as a hard disk, an SSD (Solid State Drive), or a memory card. The storage 1080 may be a memory such as a RAM or a ROM.

  The input / output interface 1100 is an interface for connecting the computer 1000 and an input / output device. In FIG. 3, a camera 20 and a display device 3020 are connected to the input / output interface 1100. The camera 20 is an arbitrary camera that repeatedly captures an image and generates a captured image representing each imaging result. Note that the camera 20 may be a two-dimensional (2D) camera or a three-dimensional (3D) camera.

  The position where the camera 20 is provided is arbitrary. For example, the camera 20 is attached to an object worn by the user. The items worn by the user are, for example, a head-mounted display or an employee ID card provided from the neck.

  FIG. 4 is a diagram illustrating a head mounted display in which the camera 20 is provided. The head mounted display 100 is a glasses-type head mounted display. The lens portion of the head mounted display 100 is a display device 102. Yes. The camera 20 is provided in the vicinity of the display device 102. By doing so, the scenery that appears in the captured image generated by the camera 20 becomes the same scenery as the scenery that is in the line of sight of the user wearing the head mounted display 100.

  The position where the camera 20 is provided is not limited to what the user is wearing. For example, the camera 20 may be provided on a wall of a room where the user performs an input operation on the information processing apparatus 2000. In the latter case, it is preferable that the shooting range (shooting direction, zoom ratio, etc.) of the camera 20 can be changed by remote control using a remote controller or the like.

  Further, for example, the camera 20 may be attached to an object worn by the user. The items worn by the user include, for example, the clothes of the user and an employee ID card that the user holds from the neck.

  The display device 3020 is an arbitrary device that outputs a screen. For example, the display device 3020 is a display device that displays a screen. For example, the display device is the display device 102 provided in the head-mounted display 100 described above. The display device 3020 may be a device that projects a screen, such as a projector.

  The storage 1080 stores program modules that implement the functions of the information processing apparatus 2000. The processor 1040 implements each function corresponding to the program module by executing each program module.

  The hardware configuration of the computer 1000 is not limited to the configuration shown in FIG. For example, each program module may be stored in the memory 1060. In this case, the computer 1000 may not include the storage 1080. Further, the method of connecting the camera 20 to the computer 1000 is not limited to the method of connecting via the input / output interface 1100. For example, the camera 20 may be connected to the computer 1000 via a network. In this case, the computer 1000 has a network interface for connecting to the network.

<Process flow>
FIG. 5 is a flowchart illustrating the flow of processing executed by the information processing apparatus 2000 according to the first embodiment. The product recognition unit 2020 recognizes the product (S102). The display control unit 2040 acquires product information of the recognized product (S104). The display control unit 2040 displays an operation screen including product information on the display device 3020 (S106). The information processing apparatus 2000 acquires a captured image (S108). The operation recognizing unit 2060 recognizes an input operation on the operation screen based on the position or movement of the operating body shown in the captured image (S110).

  Note that the flow of processing executed by the information processing apparatus 2000 is not limited to the flow shown in FIG. For example, the captured image can be acquired at an arbitrary timing before S110. As will be described later, when a captured image is used for displaying the operation screen (S106), S108 is performed before S106.

<Method for recognizing product: S102>
The product recognition unit 2020 recognizes the product 10 (S102). The method by which the product recognition unit 2020 recognizes the product 10 is arbitrary. Hereinafter, a method for recognizing the product 10 will be specifically exemplified.

<< Method of using captured image >>
The product recognition unit 2020 recognizes the product 10 shown in the captured image generated by the camera 20. More specifically, the product recognition unit 2020 recognizes the product 10 by performing object recognition on the captured image. Here, various existing techniques can be used as a method of recognizing a product shown in an image. Therefore, a specific method for recognizing the product shown in the captured image by the product recognition unit 2020 is omitted.

  The product recognition unit 2020 may recognize the product 10 with respect to the entire captured image, or may recognize the product 10 with respect to a partial region in the captured image. In the latter case, for example, the product recognition unit 2020 recognizes the product 10 with respect to a predetermined area at a predetermined position of the captured image. FIG. 6 is a diagram illustrating the product 10 shown in a predetermined area of the captured image. In FIG. 6, this predetermined area is a predetermined area 24. The predetermined area 24 is a rectangle whose center position is at the center of the captured image, whose width is the predetermined width w, and whose height is the predetermined height h. Information indicating the position of the predetermined region 24 in the captured image 22 and the shape and size of the predetermined region 24 may be set in advance in the product recognition unit 2020, or a storage device accessible from the product recognition unit 2020. Or may be settable by the user.

  When the product 10 shown in the predetermined area 24 included in the captured image is recognized, a guide that visually represents the predetermined area 24 is preferably displayed on the display device 3020. By doing so, the user can easily grasp how the product 10 can be recognized by the information processing apparatus 2000 by imaging the product 10.

  FIG. 7 is a diagram illustrating a display device 3020 on which a guide representing the predetermined area 24 is displayed. In FIG. 7, the display device 3020 is the display device 102 provided in the head mounted display 100, and the camera 20 is provided in the head mounted display 100. In FIG. 7, a guide that visually represents the predetermined area 24 is a guide 26.

  In FIG. 7A, the product 10 is outside the guide 26 although it is in the user's field of view (the imaging range of the camera 20). Therefore, the product 10 is not shown in the predetermined area 24 in the captured image 22, and the product 10 is not recognized by the product recognition unit 2020.

  On the other hand, in FIG. 7B, the product 10 is inside the guide 26. Therefore, the product 10 is shown in the predetermined area 24 in the captured image 22, and the product 10 is recognized by the product recognition unit 2020. As a result, the display control unit 2040 displays the operation screen 40 on the display device 102.

  Note that the method for recognizing the product 10 for a part of the region in the captured image is not limited to the method for recognizing the product 10 from a predetermined region at a predetermined position. An example of another method will be described in an embodiment described later.

<< Method using tag of product 10 >>
The product recognition unit 2020 acquires the product identifier (hereinafter referred to as product ID (Identifier)) read from the tag of the product 10 (the tag attached to the product 10 or the tag embedded in the product 10). 10 is recognized. The tag is, for example, an RFID (radio frequency identifier) tag.

  The product recognition unit 2020 is communicably connected to various readers that read the product ID from the tag of the product 10. The user of the information processing system 3000 causes the reader to read the product ID from the tag of the product 10. The product recognition unit 2020 acquires the product ID from the reader. The reader may be a stationary type or a portable type. In the latter case, for example, the user wears and uses a wearable reader.

<< Method of using product information symbol >>
The product recognition unit 2020 recognizes the product by acquiring the product ID read from the product information symbol attached to the product. The product information symbol is a symbol representing information for specifying a product. Here, the symbol is a bar code, a two-dimensional code (such as a QR code (registered trademark)), or a character string symbol. The character string here includes a numeric string. More specifically, the product information symbol is a barcode or the like in which information including a product ID is encoded, or a character string symbol representing information including a product ID.

  The method for acquiring the product ID from the product information symbol differs depending on what is used as the product information symbol. When the product information symbol is a barcode, for example, the product ID can be read from the barcode attached to the product 10 by using a barcode reader. In this case, the information processing apparatus 2000 is communicably connected to the barcode reader.

  When the product information symbol is a two-dimensional code, for example, the product ID can be read from the two-dimensional code attached to the product 10 by using a two-dimensional code reader. In this case, the information processing apparatus 2000 is communicably connected to the two-dimensional code reader.

  When the product information symbol is a character string symbol, for example, the product ID can be acquired by analyzing an image showing the character string symbol attached to the product 10. Note that various known techniques relating to character string analysis can be used for analyzing character string symbols.

  The image including the character string symbol may be the above-described captured image captured by the camera 20 or an image captured by another camera. In the latter case, the process of analyzing the character string symbol may be performed by the product recognition unit 2020 or may be performed by an apparatus other than the information processing apparatus 2000. When the character string symbol attached to the product 10 is analyzed by a device other than the information processing device 2000, the product recognition unit 2020 acquires the product ID of the product 10 from the device that has analyzed the character string symbol. . Further, in this case, the information processing apparatus 2000 is communicably connected to an apparatus that analyzes a character string symbol attached to the product 10.

  When a plurality of types of symbols can be used as the merchandise information symbol, the information processing apparatus 2000 is communicably connected to a reader or the like for reading a merchandise ID from each of the plurality of types of symbols. For example, when a barcode and a two-dimensional code are used as the product information symbol, the information processing apparatus 2000 is connected so as to be communicable with the barcode reader and the two-dimensional code reader.

<Method for obtaining product information: S104>
The display control unit 2040 acquires the product information of the product 10 recognized by the product recognition unit 2020 (S104). When the product recognition unit 2020 recognizes the product 10 in the captured image, the display control unit 2040 acquires the product information of the product 10 using the image of the product 10. For example, the display control unit 2040 obtains the product from the storage device or database that stores the feature amount of the product 10 (information representing the shape, color, or pattern of the product 10) and the product information of the product 10 in association with each other. Get information. For example, the display control unit 2040 acquires product information associated with a feature amount whose similarity with the feature amount of the product 10 extracted from the captured image is equal to or greater than a predetermined value. Information representing the predetermined value may be set in advance in the display control unit 2040 or may be stored in a storage device accessible from the display control unit 2040.

  On the other hand, when the product recognition unit 2020 recognizes the product 10 by acquiring the product ID of the product 10, the display control unit 2040 stores the product ID of the product 10 and the product information of the product 10 in association with each other. The product information associated with the product ID of the product 10 recognized by the product recognition unit 2020 is acquired from the device or the database.

  FIG. 8 is a diagram illustrating product information associated with the feature amount and product ID of the product 10 in a table format. The table shown in FIG. 8 is referred to as a product information table 500. The product ID 502 indicates the product ID of the product 10. The feature amount 504 indicates the feature amount of the product 10. The product information 506 indicates product information of the product 10. The product information 506 includes a product name 508, a price 510, and a description 512. The product name 508 indicates the name of the product. The price 510 indicates the price of the product. The description 512 is information indicating how to use and features of the product. The information included in the product information 506 is arbitrary and is not limited to the information shown in FIG.

  Note that a storage device or database that stores product information may be provided inside the information processing apparatus 2000 or may be provided outside the information processing apparatus 2000.

<Method for Displaying Operation Screen 40: S106>
The display control unit 2040 displays the product information on the display device 3020 (S106). Therefore, the display control unit 2040 generates the operation screen 40 based on the product information of the product 10 recognized by the product recognition unit 2020.

  For example, the display control unit 2040 obtains a template of the operation screen 40 and generates the operation screen 40 by embedding product information in the template. FIG. 9 is a diagram illustrating a template of the operation screen 40. In FIG. 9, the template of the operation screen 40 is a template 200.

  Template 200 includes replacement area 202-1 and replacement area 202-2. In the template 200, the part other than the replacement area 202 is a part that is determined in advance without depending on the recognized product 10. On the other hand, the replacement area 202 is a portion determined according to the recognized product 10.

  The display control unit 2040 generates the operation screen 40 by embedding information included in the product information of the recognized product 10. In the case of FIG. 9, the display control unit 2040 embeds the product name 508 of the product information table 500 corresponding to the recognized product 10 in the replacement area 202-1. Further, the display control unit 2040 embeds the price 510 of the product information table 500 corresponding to the recognized product 10 in the replacement area 202-2.

<Method for Acquiring Captured Image: S108>
The information processing apparatus 2000 acquires a captured image generated by the camera 20 (S108). The method by which the information processing apparatus 2000 acquires the captured image is arbitrary. For example, the information processing apparatus 2000 acquires a captured image from the camera 20. In this case, the information processing apparatus 2000 and the camera 20 are connected to be communicable.

  Further, when the camera 20 stores a captured image in an external storage device, the information processing apparatus 2000 may acquire the captured image from the storage device. In this case, the information processing device 2000 is communicably connected to the storage device.

  Note that the information processing apparatus 2000 may acquire all of the captured images generated by the camera 20, or may acquire only a part thereof. In the latter case, for example, the information processing apparatus 2000 acquires only the captured image generated after the operation screen is displayed by the display apparatus 3020. However, in this case, the product recognition by the product recognition unit 2020 is a method other than the method performed by the product recognition unit 2020 using the captured image.

<About the operation tool>
There are various objects handled by the operation recognition unit 2060 as an operation body. For example, the operation recognition unit 2060 treats a part of the user's arm (such as a finger) or an object (such as a pen) held by the user's arm as an operating body. The arm portion means a hand and a portion from the hand to the shoulder. In this case, the user performs an input operation by moving a finger or a pen within the imaging range of the camera 20.

  Further, for example, the operation recognition unit 2060 may treat an object or a marker attached to the user's body as an operation body. The marker here is an arbitrary mark that can be imaged by the camera 20. In this case, the user performs an input operation by moving the marker within the imaging range of the camera 20.

  For example, the marker is attached to the user's body (such as a finger). For example, the marker is attached to an object (such as a pen) held by the user. For example, the marker is attached to an object worn by the user. The thing that the user can wear is, for example, a ring that can be worn on the user's finger or an arbitrary wearable device.

  Information indicating what the operation recognition unit 2060 handles as an operation tool may be set in advance in the operation recognition unit 2060 or may be stored in a storage device accessible from the operation recognition unit 2060.

  In addition, the thing which the operation recognition part 2060 handles as an operation body may be one type, and may be multiple types.

<Method of detecting the position of the operating tool: S110>
The operation recognizing unit 2060 detects the position of the operating body in the captured image in order to recognize the input operation on the operation screen (S108). Here, a known technique can be used as a technique for detecting a predetermined object in the image. Therefore, a specific description of a method for detecting a predetermined operation tool from the captured image is omitted.

  There are various methods by which the operation recognition unit 2060 determines the position of the operation body. For example, the operation recognizing unit 2060 determines an area representing the operating tool from the captured image. Then, the operation recognizing unit 2060 sets the point included in the determined area as the position of the operating body. At this time, the position of the operating tool may be an arbitrary point included in the region representing the operating tool.

  For example, when the operation body is a part of the user's body or an object held by the user, the operation recognition unit 2060 calculates the center of gravity of the region representing the operation body. Then, the operation recognizing unit 2060 sets the point farthest from the center of gravity of the area included in the area representing the operating body as the position of the operating body. According to this method, for example, a fingertip or a pen tip is the position of the operating body.

  FIG. 10 is a first diagram illustrating a method for determining the position of the operating tool. In FIG. 10, the operating tool is a user's hand. First, the operation recognition unit 2060 determines an area 60 representing the user's hand from the captured image. Next, the operation recognition unit 2060 calculates the center of gravity 62 of the region 60. Then, the operation recognition unit 2060 sets a point 64 that is included in the region 60 and is farthest from the center of gravity 62 as the position of the operation body. When there are a plurality of points farthest from the center of gravity 62 in the region 60, for example, the operation recognition unit 2060 sets a point farthest from the first marker 3040 as the position of the operating body among these points.

  Here, for example, when an input operation is performed with the belly of the finger, it may be preferable to treat a place slightly deviated from the fingertip as the position of the operating body. Therefore, the operation recognition unit 2060 calculates a point 64 farthest from the center of gravity of the operating body, and further sets a position slightly shifted from this point (for example, a position shifted a predetermined distance in the direction approaching the base of the finger) as the position of the operating body. Also good. Information representing the positional relationship between the point 64 farthest from the center of gravity of the operating body and the position of the operating body may be preset in the operation recognition unit 2060 or stored in a storage device accessible from the operation recognition unit 2060. Or may be set by the user.

  However, the position of the operating body is not limited to a point farthest from the center of gravity 62 or a point determined with reference to that point. For example, the center of gravity 62 may be treated as the position of the operating body.

  When the operation body is a marker attached to an object or the user's body, for example, the operation recognition unit 2060 determines an area representing the marker from the captured image, and uses the center position of the area as the position of the operation body.

  Further, it is assumed that the operation recognition unit 2060 detects an operation body using a reference image representing an object to be detected as the operation body. In this case, the position of the operation tool may be defined in advance in this reference image. The operation recognizing unit 2060 determines an area similar to the reference image from the captured image. Then, the operation recognizing unit 2060 determines a point corresponding to the position of the operating tool defined in the reference image in the area, and treats that point as the position of the operating tool.

  FIG. 11 is a second diagram illustrating a method for determining the position of the operating tool. In this example, the operating tool is a user's finger. The reference image 120 is a reference image representing the shape of the user's finger or the like. The position 121 of the operating body is a position of the operating body that is defined in advance in the reference image.

  The operation recognition unit 2060 determines an area 130 similar to the reference image 120 from the captured image 22. This area 130 represents the user's finger. Further, the operation recognizing unit 2060 determines a point 131 corresponding to the position 121 of the operating tool when the reference image 120 is mapped to the region 130. Then, the operation recognition unit 2060 treats the point 131 as the position of the operation body.

  Information indicating how the operation recognizing unit 2060 determines the position of the operating tool may be preset in the operation recognizing unit 2060 or stored in a storage device accessible from the operation recognizing unit 2060. It may be set by the user.

<Method for Detecting Movement of Operating Object: S110>
The operation recognizing unit 2060 detects the movement of the operating body in the captured image in order to recognize the input operation on the operation screen (S110). The operation recognizing unit 2060 may detect the movement of the operating body using a plurality of captured images, or may detect the movement of the operating body using one captured image. In the former case, for example, the operation recognizing unit 2060 calculates the position of the operating body in each captured image by performing image analysis on each of the plurality of captured images. The operation recognizing unit 2060 uses the information indicating the change in the position of the operating tool as information indicating the movement of the operating tool. The information indicating the change in the position of the operating tool is, for example, information in which the positions of the operating tool are arranged in time series.

  As described above, the operation recognition unit 2060 may detect the movement of the operation body using one captured image. In many cases, a moving operating body is captured in a state of being shaken by one captured image. Therefore, the operation recognizing unit 2060 calculates the movement of the operating body from the image of the operating body that is reflected in the one captured image.

  FIG. 12 is a diagram illustrating a captured image 22 in which a user's finger is blurred. In the captured image 22, the user's finger 30 is blurred as if moving from the finger 30 -A to the finger 30 -B. The operation recognition unit 2060 detects a change in the position of a feature point common to the finger 30-A and the finger 30-B as the movement of the operation body. For example, the operation recognition unit 2060 detects a movement 50 determined by a change in the position of the fingertip of the finger 30-A and the fingertip of the finger 30-B.

<Input Operation Recognized by Operation Recognizing Unit 2060: S110>
The operation recognition unit 2060 recognizes an input operation on the operation screen based on the detected position or movement of the operation body (S110). The operation recognition unit 2060 can recognize various input operations determined based on the position or movement of the operation body. Hereinafter, various input operations that can be recognized by the operation recognition unit 2060 will be exemplified.

<< Input operation determined based on the position of the operating body >>
There are various input operations recognized by the operation recognition unit 2060 based on the position of the operation body. For example, the operation recognizing unit 2060 receives an input operation (hereinafter referred to as a key input operation) for selecting an image representing a key based on the position of the operation body.

  FIG. 13 is a diagram illustrating an operation determined based on the position of the operating tool. In FIG. 13, the operating body is a finger 30. In FIG. 13A, the finger 30 is positioned on the “5” key. Therefore, the operation recognition unit 2060 receives an input operation for inputting “5”. On the other hand, in FIG. 13B, the finger 30 is positioned on the “cancel” key. Therefore, the operation recognition unit 2060 accepts an input operation for inputting “cancel”.

  Note that the input operation recognized by the operation recognition unit 2060 based on the position of the operating tool may be any input operation determined by the position of the operating tool, and is not limited to the key input operation. For example, the operation recognition unit 2060 may accept an input operation for selecting one of a plurality of photographs displayed on the display device 3020 based on the position of the operation tool.

<< Input operation determined based on movement of operation body >>
The operation recognition unit 2060 may recognize 1) an input operation for inputting a shape based on the detected movement of the operating body, or 2) recognize a predetermined input operation corresponding to the detected movement of the operating body. Good. Each will be described below.

<< Case of recognizing input operation for inputting shape based on movement of operation body >>
FIG. 14 is a diagram illustrating a case of recognizing an input operation for inputting a shape determined by the movement of the operating body. In FIG. 14A, the operation recognition unit 2060 recognizes an input operation for inputting the shape 51 represented by the motion 50-A of the operating body and the shape 52 represented by the motion 50-B. For example, this input operation is used in an input operation for performing handwritten input.

  In FIG. 14B, the operation recognition unit 2060 recognizes an input operation that inputs a shape whose shape and size are determined by the movement of the operation body, although the shape is different from the movement of the operation body. Specifically, the operation recognizing unit 2060 recognizes an input operation for inputting a rectangle 54 having both ends of the motion 50-C as opposite ends of the diagonal line and a circle 56 having both ends of the motion 50-C as both ends of the diameter. For example, this input is used when the user performs an input representing a certain range (such as a selection operation) or when drawing a predetermined figure.

  Information indicating whether to use the method shown in FIG. 14A or the method shown in FIG. 14B may be set in advance in the operation recognition unit 2060 or can be accessed from the operation recognition unit 2060. It may be stored in a storage device or may be set by the user.

<< Case where a predetermined input operation corresponding to the movement of the operating body is recognized >>
The predetermined input operation corresponding to the detected movement of the operating body is, for example, an input operation by gesture input. FIG. 15 is a diagram illustrating gesture input. FIG. 15A shows a flick operation, and FIG. 15B shows a pinch-in / pinch-out operation. The arrow represents the movement of the operating tool.

  Information associating the movement of the operating body with a predetermined input operation corresponding to the movement may be set in advance in the operation recognizing unit 2060 or stored in advance in a storage device accessible from the operation recognizing unit 2060. It may be set by the user.

<How to represent the position and movement of the operating tool>
The method of representing the position and movement of the operating body recognized by the operation recognizing unit 2060 is arbitrary. For example, the operation recognition unit 2060 represents the position and movement of the operating tool as relative positions in the entire captured image. FIG. 16 is a diagram illustrating a state in which the position and movement of the operating tool are represented by relative positions and movements in the entire captured image. FIG. 16A shows a case where the position of the operating body is recognized as an input operation. In FIG. 16A, the coordinates (x1, x1) as the position of the operating body are the origin at the upper left corner of the captured image 22, the x axis is the right direction when the captured image 22 is viewed in plan, and the captured image 22 Is the coordinate in the coordinate system with the y-axis as the downward direction when viewed in plan view.

  FIG. 16B shows a case where the movement of the operating body is recognized as an input operation. The arrow represents the trajectory of the operating body. In FIG. 16B, the position of the operating body changes in the order of (x2, y2), (x3, y3), (x4, y4). These coordinates are all the coordinates in the coordinate system described with reference to FIG. The movement of the operating body is represented by, for example, information in which these coordinates are arranged in time series.

  Note that the above-described method for representing the position and movement of the operating tool is merely an example. The method of representing the position and movement of the operating tool may be any method that can represent the position and movement of the operating tool, and is not limited to the method described above.

<< Handling of the place where the operation body is moved >>
The operation recognition unit 2060 may recognize only the movement of the operation body as an input regardless of the position where the operation of moving the operation body is performed, or 2) the movement of the operation body and the operation body. A combination with the position where the moving operation is performed may be recognized as an input. In the former case, the movement of the same operating tool represents the same input no matter where the movement of the operating tool is performed on the captured image. On the other hand, in the latter case, it is significant where the operation body is moved on the captured image. For example, when the user performs an input of encircling a specific object shown in the captured image, not only the shape of the circle but also what is surrounded by the circle has a meaning.

  In the case of 1), for example, the operation recognizing unit 2060 recognizes, as input, the shape determined by the motion of the operating body detected by the operation recognizing unit 2060 and the gesture determined by the motion of the operating body as described above. Further, for example, the operation recognition unit 2060 may recognize, as an input, the transition of the relative coordinates of the operation body based on the start point of the movement of the operation body.

  On the other hand, in the case of 2), the operation recognition unit 2060 recognizes, as an input, the transition of the coordinates of the operation body on the captured image detected by the operation recognition unit 2060. However, in the case of 1), the operation recognizing unit 2060 may recognize the transition of the coordinates of the operating body on the captured image as an input as in the case of 2).

  Whether the operation recognition unit 2060 recognizes the input by 1) or 2) may be set in advance in the operation recognition unit 2060 or stored in a storage device accessible from the operation recognition unit 2060. It may be set by the user.

<Which input operation should be recognized by the position or movement of the operating tool>
The operation recognition unit 2060 recognizes an input operation based on the position or movement of the operation body. Here, there are various methods for determining whether the input operation is recognized based on the position or movement of the operating body. For example, it is determined in advance whether the input operation is recognized by the position or movement of the operating body. Here, the information indicating whether the input operation is recognized by the position or movement of the operating tool may be set in the operation recognizing unit 2060 or stored in a storage device accessible from the operation recognizing unit 2060. Alternatively, it may be set by the user.

  Further, for example, the operation recognition unit 2060 may determine whether to recognize the input operation based on the position or movement of the operating body, depending on the degree of movement of the operating body. For example, when the size of the moving range of the operating tool within a predetermined time is less than a predetermined value, the operation recognizing unit 2060 recognizes the input operation based on the position of the operating tool. Thereby, for example, when the user holds the operation tool at a certain position, the operation recognition unit 2060 recognizes the input operation representing the position. This operation is, for example, an image operation in which a certain place is long pressed with a mouse.

  In this case, the position of the operating body recognized as the input operation by the operation recognition unit 2060 is determined by the position of the operating body within the predetermined time. For example, the operation recognition unit 2060 sets any one of the positions of the operating body within the predetermined time as the position of the operating body. Further, for example, the operation recognition unit 2060 uses a statistical value (such as an average value or a mode value) calculated from the position of the operating body within the predetermined time as the position of the operating body.

  On the other hand, when the size of the moving range of the operating tool within a predetermined time is greater than or equal to a predetermined value, the operation recognizing unit 2060 recognizes the input operation based on the movement of the operating tool.

  The information indicating the predetermined time or the predetermined value may be set in the operation recognition unit 2060, stored in a storage device accessible from the operation recognition unit 2060, or set by the user. .

[Embodiment 2]
FIG. 17 is a block diagram illustrating an information processing system 3000 according to the second embodiment. Except for the points described below, the information processing system 3000 according to the second embodiment has the same functions as the information processing system 3000 according to the first embodiment.

  The information processing system 3000 has a first marker 3040. The first marker 3040 is worn by the user or is a part of the user's body. A detailed description of the first marker 3040 will be described later.

  The product recognition unit 2020 of the second embodiment recognizes a product from the captured image generated by the camera 20. At that time, the product recognition unit 2020 recognizes the product from a part of the region included in the captured image, not from the entire captured image. This “partial region” is determined by the first marker 3040 in the captured image.

  Specifically, first, the product recognition unit 2020 extracts a product recognition region that is determined based on the position of the first marker 3040 in the captured image. And the goods recognition part 2020 recognizes the goods reflected in this goods recognition area.

<Process flow>
FIG. 18 is a flowchart illustrating the flow of processing executed by the product recognition unit 2020 according to the second embodiment. This flowchart represents one mode of a series of processes performed in S102 of FIG.

  The product recognition unit 2020 acquires a captured image (S202). The product recognition unit 2020 calculates the position of the first marker 3040 in the captured image (S204). The product recognition unit 2020 extracts a product recognition area from the captured image based on the position of the first marker 3040 (S206). The product recognition unit 2020 recognizes the product 10 in the product recognition area (S208).

<About the first marker 3040>
The first marker 3040 is an arbitrary object whose position can be specified at least in the captured image generated by the camera 20. For example, the first marker 3040 is a marker that can be used to determine a three-dimensional coordinate system. A marker that can be used for determining a three-dimensional coordinate system is, for example, an AR (Augmented Reality) marker. However, any marker that can be used to determine the three-dimensional coordinate system is not limited to the reference direction, as long as it can constantly obtain three directions orthogonal to each other from a certain reference point. Is not limited. In addition, the first marker 3040 may be anything that can identify the position in the captured image, and does not necessarily need to be used for determining a three-dimensional coordinate system.

  When the first marker 3040 is worn on the user's body, the first marker 3040 can be attached to any location on the user's body. For example, the first marker 3040 is worn on the user's arm.

  For example, in this case, the first marker 3040 is an image displayed on the display device of the device attached to the user's arm. This device is any electronic device having a function of displaying an image on a display device. The device may be directly attached to the user's arm or may be attached to the user's arm from above the clothes.

  FIG. 19 is a diagram illustrating a device that displays an image treated as the first marker 3040. In FIG. 19, the first marker 3040 is a marker image 84 displayed on the touch panel 82 of the device 80. The marker image 84 may be an image stored in advance in the device 80, or may be an image stored in a storage device outside the device 80. In the latter case, the device 80 acquires the marker image 84 from the storage device and displays it.

  The first marker 3040 is not limited to the one displayed on the device as described above. The first marker 3040 may be drawn directly on the user's arm or the like, or may be drawn on an arbitrary object on the user's arm or the like. In the latter case, for example, the first marker 3040 is drawn on a ring attached to the user's finger, a wristband attached to the user's wrist, or a sleeve of clothes worn by the user. The first marker 3040 may be drawn by hand or printed.

  Further, the first marker 3040 may be a light emitting device (such as an LED (Light Emitting Diode)) that emits light. Note that when the first marker 3040 is a light emitting device and a three-dimensional coordinate system is determined based on the first marker 3040, the first marker 3040 is configured using three or more light emitting devices. The merchandise recognition unit 2020 calculates the position of light emitted by each of these three or more light emitting devices shown in the captured image. And the goods recognition part 2020 can determine a three-dimensional coordinate system based on the position of each light emitting device. Since a known method can be used as a method for determining a three-dimensional coordinate system using three or more objects, a description of the specific method is omitted.

  Further, the first marker 3040 may be a specific part of the user's body. For example, the first marker 3040 is the back of the user's hand.

  In order to detect the first marker 3040 from the captured image, the merchandise recognizing unit 2020 identifies information to be handled as the first marker 3040 (information on the shape, size, color, etc. of the first marker 3040; Information). The marker information may be set in advance in the product recognition unit 2020, or may be stored in a storage device accessible from the product recognition unit 2020.

<Method of calculating position of first marker 3040: S204>
The product recognition unit 2020 detects the first marker 3040 in the captured image by performing image analysis on the captured image using the marker information. Then, the product recognition unit 2020 calculates the position of the detected first marker 3040. Here, various known techniques can be used as a technique for detecting a predetermined object from the image and calculating the position of the object in the image. Note that the position of the first marker 3040 is represented by, for example, a relative position (coordinates) in the entire captured image, similarly to the position of the operating tool. For example, the position of the first marker 3040 is the coordinates of the center position of the first marker 3040 in the entire captured image. However, the position of the first marker 3040 is not limited to the center position of the first marker 3040.

<Method for Extracting Product Recognition Area: S206>
The product recognition unit 2020 extracts a product recognition area from the captured image based on the calculated position of the first marker 3040 on the captured image (S106). The product recognition area is an arbitrary area that is determined based on the position of the first marker 3040. For example, the product recognition area is an area determined by a predetermined shape with the position (for example, the center position) of the first marker 3040 as the center position. This predetermined shape is an arbitrary shape such as a circle or a rectangle.

  FIG. 20 is a first diagram illustrating the product recognition area 70. In FIG. 20, the product recognition area 70 is a rectangle having the center position of the marker image 84 (first marker 3040), the width being a predetermined length w, and the height being a predetermined length h. .

  However, the position determined by the position of the first marker 3040 is not limited to the center position of the product recognition area 70. For example, the position of the upper left corner of the product recognition area 70 may be determined by the position of the first marker 3040.

  Further, for example, the product recognition area 70 may be an area having a center position or the like at a predetermined distance from the position of the first marker 3040 in a predetermined direction. FIG. 21 is a second diagram illustrating the product recognition area 70. In FIG. 21, the center position of the product recognition area 70 is a place moved from the first marker 3040 by a distance d in the direction determined by the arrow 72.

  20 and 21, the product recognition area 70 has a predetermined shape on the plane (xy plane in FIG. 16) represented by the captured image. However, the product recognition area 70 may have a predetermined shape on a plane other than the plane represented by the captured image.

  For example, the product recognition area 70 may have a predetermined shape on a plane in a three-dimensional coordinate system determined by the first marker 3040. FIG. 22 is a third diagram illustrating the product recognition area 70. Similar to the product recognition area 70 of FIG. 20, the product recognition area 70 of FIG. 22 has the center position of the marker image 84 (first marker 3040), the width is a predetermined length w, and the height is high. A rectangle with a predetermined length h. However, in FIG. 22, the product recognition area 70 is a rectangle in the xy plane of the coordinate system 87 determined by the marker image 84. More specifically, the product recognition area 70 is centered on the origin of the coordinate system 87, the length of the coordinate system 87 in the x-axis direction is w, and the length of the coordinate system 87 in the y-axis direction is h. is there.

  FIG. 23 is a diagram of the product recognition area 70 illustrated in FIG. The product recognition area 70 in FIG. 23 is similar to the product recognition area 70 in FIG. 21 and has a center position at a position moved by a distance d in the direction determined by the arrow 72 from the marker image 84, and the width is a predetermined length w. A rectangle whose height is a predetermined length h. However, in FIG. 23, the direction represented by the arrow 72 is the direction in the coordinate system 87. The product recognition area 70 has a length w in the x-axis direction of the coordinate system 87 and h in the y-axis direction of the coordinate system 87.

  Thus, by extracting the product recognition area 70 using the coordinate system determined by the first marker 3040, the attitude of the product recognition area 70 follows the change in the attitude of the first marker 3040. For example, in the example of FIG. 22, when the user twists the wrist on which the device 80 is provided, the posture of the product recognition area 70 changes according to the twist.

  FIG. 24 is a diagram illustrating a state in which the posture of the product recognition area 70 changes according to the inclination of the first marker 3040. In FIG. 24, the posture of the marker image 84 (the coordinate system 87 determined by the marker image 84) is changed as the user twists the wrist. Further, the attitude of the product recognition area 70 is also changed so as to follow the change.

  As described above, when the posture of the product recognition area 70 is changed in accordance with the change of the posture of the first marker 3040, an object in the real world to be included in the product recognition area 70 depends on the posture of the first marker 3040. The product recognition area 70 can be included. For example, in the example of FIGS. 23 and 24, by appropriately defining the shape and size of the product recognition area 70 and the position of the product recognition area 70, the user's hand can recognize the product regardless of the posture of the marker image 84. The region 70 is included. In this case, the user can cause the information processing apparatus 2000 to recognize the product 10 by holding the product 10 in his hand. Therefore, the user can easily use the information processing apparatus 2000.

  Note that the information indicating the predetermined shape and the information indicating which location is the center of the first marker 3040 (such as information on the arrow 72) may be set in advance in the product recognition unit 2020, It may be stored in a storage device accessible from the recognition unit 2020, or may be set by a user.

  When the coordinate system 87 determined based on the first marker 3040 is used, the first marker 3040 is a marker (such as an AR marker) that can be used for determining three-dimensional coordinates. Various existing techniques can be used as a technique for determining a three-dimensional coordinate system using an AR marker or the like.

  Further, the size (such as the width w and the height h described above) of the product recognition area 70 may be defined by an absolute value (number of pixels or the like) or may be defined by a relative value. In the latter case, for example, the size of the product recognition area 70 is represented by a relative value with respect to the size of the captured image. The relative value with respect to the size of the captured image is, for example, a relative value where the width or height of the captured image is 1. For example, the size of the product recognition area 70 may be represented by a relative value with respect to the size of the first marker 3040. The relative value with respect to the size of the first marker 3040 is a relative value in which the width, height, diagonal line, diameter, or the like of the first marker 3040 is 1, for example. If the size of the product recognition area 70 is defined by a relative value with respect to the size of the first marker 3040, the product recognition area 70 becomes larger when the first marker 3040 is brought closer to the camera 20, and the first marker 3040 is moved from the camera 20. The product recognition area 70 becomes smaller when it is moved away.

  In addition, the product recognition unit 2020 may extract a predetermined area determined by using the first marker 3040 as a boundary as the product recognition area 70. For example, in the situation where the device 80 is provided on the wrist of the user, the product recognition unit 2020 extracts a region including the hand as the product recognition region 70 among regions defined by the marker image 84 (first marker 3040) as a boundary. To do.

  FIG. 25 is a diagram illustrating a region including a hand. In FIG. 25, the product recognition unit 2020 divides the captured image 22 into two regions with the y axis in the coordinate system 87 determined by the marker image 84 as a boundary. Then, the product recognition unit 2020 extracts, as the product recognition region 70, the region including the hand (the right region in FIG. 25) among the two divided regions.

  Note that the positional relationship between the arm and the hand with the first marker 3040 as a boundary (which side is the arm and which side is the hand) differs depending on which arm portion the first marker 3040 is provided on. For example, the product recognition unit 2020 grasps which arm part the first marker 3040 is provided by using information indicating which arm part the first marker 3040 is provided on. This information may be set in advance in the product recognition unit 2020, may be stored in a storage device accessible from the product recognition unit 2020, or may be set by the user.

<Method for Recognizing Product in Product Recognition Area: S208>
The product recognition unit 2020 recognizes the product 10 shown in the product recognition area 70 extracted from the captured image. Here, various known techniques can be used as a method of recognizing an object reflected in a specific region of an image by image analysis.

<Hardware configuration>
The hardware configuration of a computer that implements the information processing apparatus 2000 according to the second embodiment is represented by, for example, FIG. However, the storage 1080 of the computer 1000 that implements the information processing apparatus 2000 according to the present embodiment further stores program modules that implement the functions of the information processing apparatus 2000 according to the present embodiment.

<Action and effect>
According to the present embodiment, the product recognition area 70 which is a partial area of the captured image is extracted, and the product 10 is recognized from the product recognition area 70. Therefore, the size of data to be subjected to image processing is smaller than in the case where the process of recognizing the product 10 from the entire captured image is performed. Therefore, the process for recognizing the product 10 in this embodiment is a lighter process than the process for recognizing the product 10 from the entire captured image, and the time required for the process and computer resources are reduced.

  Further, according to the present embodiment, since the product 10 is recognized from the product recognition area 70 extracted based on the first marker 3040 shown in the captured image, the product is displayed when the first marker 3040 is not shown in the captured image. 10 is not recognized, and as a result, the operation screen 40 is not displayed. Therefore, the user can easily control whether or not to display the operation screen 40 on the display device 3020 depending on whether or not the first marker 3040 is included in the imaging range of the camera 20.

  For the user, the operation screen 40 does not always have to be displayed. For example, when the display device 3020 is the display device 102 of the head mounted display 100, the user may want to display the operation screen 40 only when necessary in order to clear the field of view as much as possible. According to the information processing apparatus 2000 of this embodiment, the user can easily control whether or not to display the operation screen 40 on the display apparatus 3020 depending on whether or not the first marker 3040 is included in the imaging range of the camera 20. . More specifically, when the camera 20 is provided in the head mounted display 100, the user can display the operation screen 40 by putting the first marker 3040 in the field of view, and the first marker 3040 can be viewed from the field of view. By removing, the operation screen 40 can be prevented from being displayed. With such an easy operation, the operation screen 40 can be displayed only when necessary, so that the convenience of the information processing system 3000 is improved for the user.

  Further, since the input operation by the user is performed when the operation screen 40 is displayed, the user is willing to perform the input operation when the operation screen 40 is not displayed, that is, when the first marker 3040 is not shown in the captured image. I can say no. According to the information processing apparatus 2000 of the present embodiment, when the first marker 3040 is not shown in the captured image, the product 10 is not recognized, and as a result, the operation screen 40 is not displayed, so that the operation recognition unit 2060 recognizes the input operation. Also not done. As described above, when the user does not intend to perform the input operation, the operation recognition unit 2060 does not recognize the input operation, and thus it is possible to prevent the computer resources of the information processing apparatus 2000 from being wasted.

[Embodiment 3]
FIG. 26 is a block diagram illustrating an information processing system 3000 according to the third embodiment. Except as described below, the information processing system 3000 according to the third embodiment has the same functions as the information processing system 3000 according to the first or second embodiment.

  The information processing system 3000 according to the third embodiment includes a second marker 3060. Similar to the first marker 3040, the second marker 3060 is a marker that can be worn by the user or a part of the user's body.

  The display control unit 2040 of the third embodiment displays an operation screen on the display device 3020 when the second marker 3060 is reflected in the captured image.

<Process flow>
FIG. 27 is a flowchart illustrating the flow of processing executed by the information processing apparatus 2000 according to the third embodiment. Except as described below, the flowchart of FIG. 27 is the same as the flowchart of FIG.

  The display control unit 2040 detects the second marker 3060 from the captured image (S302). When the second marker 3060 can be detected from the captured image (S304: YES), the processing in FIG. 27 proceeds to S108. On the other hand, when the second marker 3060 cannot be detected from the captured image (S304: NO), the processing in FIG. 27 ends.

<About the second marker 3060>
What can be used as the second marker 3060 is the same as that which can be used as the first marker 3040.

  The first marker 3040 and the second marker 3060 may be realized by the same thing. For example, the information processing apparatus 2000 handles the marker image 84 displayed on the device 80 as the first marker 3040 and the second marker 3060. However, the first marker 3040 and the second marker 3060 may be different from each other.

<Method for Detecting Second Marker 3060: S302>
The method by which the display control unit 2040 detects the second marker 3060 from the captured image is the same as the method by which the product recognition unit 2020 of the second embodiment detects the first marker 3040 from the captured image. However, unless otherwise specified, the display control unit 2040 only needs to grasp that the second marker 3060 is reflected in the captured image, and does not need to grasp the position of the second marker 3060.

<Operation screen display: S108>
When the second marker 3060 is detected from the captured image (S304: YES), the display control unit 2040 displays an operation screen on the display control unit 2040 (S108). The method for displaying the operation screen on the display control unit 2040 is as described in the first embodiment.

<Display of Operation Screen 40 Using Position of Second Marker 3060>
In the present embodiment, the display control unit 2040 may determine the display position of the operation screen based on the position of the second marker 3060. In this case, the display control unit 2040 calculates the position of the second marker 3060. Similar to the method of calculating the position of the first marker 3040, the method of calculating the position of the second marker 3060 uses various known techniques for calculating the position of the object detected from the image in the image. Available.

  There are various methods for determining the position to display the operation screen based on the position of the second marker 3060. For example, the display control unit 2040 sets the center position of the operation screen 40 to the center position of the second marker 3060. FIG. 28 is a first diagram illustrating the operation screen 40 whose position is determined by the position of the second marker 3060. In FIG. 28, the center position of the operation screen 40 is the center position of the marker image 84 (second marker 3060). In FIG. 28, the shape of the operation screen 40 is a rectangle on a plane determined by the captured image 22 (for example, the xy plane of FIG. 16). Also, in FIG. 28, information included in the operation screen 40 is omitted for easy viewing of the figure.

  Further, for example, the display control unit 2040 may set the center position of the operation screen 40 at a position away from the center position of the second marker 3060 in a predetermined direction by a predetermined distance. FIG. 29 is a second diagram illustrating the operation screen 40 whose position is determined by the position of the second marker 3060. In FIG. 29, the center position of the operation screen 40 is a position moved from the center position of the marker image 84 by a predetermined distance d in the direction determined by the arrow 46. Also in FIG. 29, the shape of the operation screen 40 is a rectangle on a plane determined by the captured image 22 (for example, the xy plane of FIG. 16). Also in FIG. 29, information included in the operation screen 40 is omitted.

  However, the shape of the operation screen 40 is not limited to a rectangle. The shape of the operation screen 40 may be a rectangle or the like on a plane other than the plane determined by the captured image 22. For example, the display control unit 2040 maps the operation screen 40 onto a plane in a coordinate system determined by the second marker 3060 (for example, a coordinate system 87 determined by the marker image 84).

  FIG. 30 is a third diagram illustrating the operation screen 40 whose position is determined by the position of the second marker 3060. In FIG. 30, the center position of the operation screen 40 is the center position of the marker image 84. In FIG. 30, the operation screen 40 is mapped to the xy plane of the coordinate system 87 determined by the marker image 84. Note that information included in the operation screen 40 is also omitted in FIG.

  FIG. 31 is a diagram of FIG. 4 illustrating the operation screen 40 whose position is determined by the position of the second marker 3060. In FIG. 31, the center position of the operation screen 40 is a position moved from the center position of the marker image 84 by a predetermined distance d in the direction determined by the arrow 46. However, the arrow 46 represents the direction in the coordinate system 87 determined by the marker image 84. Further, the operation screen 40 is mapped on the xy plane of the coordinate system 87. Note that information included in the operation screen 40 is also omitted in FIG.

When the information processing apparatus 2000 according to the third embodiment has the function of the information processing apparatus 2000 according to the second embodiment and the same one (for example, the marker image 84) is used for the first marker 3040 and the second marker 3060, the product It is preferable that the direction indicated by the arrow 72 for determining the position of the recognition area 70 and the direction indicated by the arrow 46 for determining the position of the operation screen 40 are set in different directions. For example, the direction indicated by the arrow 46 in FIGS. 29 and 31 is the opposite direction to the direction indicated by the arrow 72 in FIGS. 18 and 20, respectively. By doing in this way, the goods recognition area 70 (place where a user positions the goods 10) and the place where the operation screen 40 is displayed can be prevented from overlapping. That is, the operation screen 40 does not overlap with the product 10 and the user can view both the product 10 and the operation screen 40 at the same time.

  The position of the operation screen 40 on the captured image is determined based on the position of the second marker 3060 by the various methods described above. Therefore, the display control unit 2040 displays the operation screen 40 on the display device 3020 so that the operation screen 40 is displayed at a position on the display device 3020 corresponding to the determined position on the captured image. The method includes a method of displaying the operation screen 40 without being superimposed on the captured image and a method of displaying the operation screen 40 while being superimposed on the captured image. Each will be described below.

<< Method for Displaying Operation Screen 40 without Superimposition on the Captured Image >>
It is assumed that the information processing apparatus 2000 is realized using the head mounted display 100, and the display device 102 of the head mounted display 100 is a transmissive display device (the head mounted display 100 is a transmissive head mounted display). In this case, the display device 3020 is realized as the display device 102 (lens portion) of the transmissive head mounted display 100. In this case, the camera 20 that generates the captured image is provided so as to capture a scene similar to the scene seen by the user's eyes (such as the camera 20 in FIG. 2). When such a transmissive head-mounted display 100 is used, the user can grasp the surrounding scenery by looking at the real object (real world scenery) that can be seen ahead of the display device 102, and the scenery can be captured in the captured image. The scenery will be the same as the scene. Therefore, it is not necessary to display the captured image on the display device 102.

  Therefore, the display control unit 2040 displays the operation screen 40 on the display device 3020 without being superimposed on the captured image. Specifically, the display control unit 2040 converts the position of the operation screen 40 on the captured image into a position on the display device 102, and displays the operation screen 40 at the position on the display device 102 calculated by this conversion. Let

  The correspondence between the coordinates on the captured image and the coordinates on the display device 102 can be determined based on various parameters (view angle and focal length) relating to the camera 20, the positional relationship between the display device 102 and the camera 20, and the like. it can. This correspondence may be calculated by the display control unit 2040 using these parameters or the like, or may be predetermined as a set value.

<< Method of Displaying Operation Screen 40 Overlaid on Captured Image >>
When the user cannot directly see the surrounding scenery, or when the scenery shown in the captured image is not the same as the scenery seen by the user's eyes (when the user's viewing direction is not captured by the camera 20), the user views the captured image. Perform input operations while watching. For example, the information processing device 2000 is realized using the head mounted display 100, and the display device 102 of the head mounted display 100 is a non-transmissive display device (the head mounted display 100 is a non-transmissive head mounted display). And In this case, the user cannot directly see the surrounding scenery. Therefore, the display device 102 displays a captured image generated by the camera 20 that captures the user's field of view direction. The user grasps the surrounding scenery by looking at the captured image. Therefore, the user performs an input operation while viewing the captured image.

  Further, when the camera 20 is provided on the user's employee ID card or the like, the camera 20 does not necessarily capture the user's viewing direction. In this case, the user performs an input operation while viewing the captured image generated by the camera 20. In this case, the display device 3020 is realized as, for example, a projector, a stationary display device (such as a PC display device), or a display device of a portable terminal.

  As described above, when the user performs an input operation while viewing the captured image, the captured image is displayed on the display device 3020. Therefore, the display control unit 2040 causes the display device 3020 to display the operation screen 40 by superimposing the operation screen 40 on the captured image displayed by the display device 3020. In this case, the display control unit 2040 displays the operation screen 40 at the position of the operation screen 40 (for example, the position of the operation screen 40 shown in FIGS. 28 to 31) on the captured image determined by the various methods described above. Superimpose.

<When the second marker 3060 is not detected from the captured image: S304: NO>
When the second marker 3060 is not detected from the captured image (S304: NO), the display control unit 2040 may not display the operation screen on the display device 3020 or display a part of the operation screen on the display device 3020. You may let them.

  In the latter case, for example, the display control unit 2040 does not display the interface portion (such as an image representing a keyboard) used by the user for an input operation in the operation screen on the display device 3020, but represents a portion representing product information. Is displayed on the display device 3020. By doing in this way, even if it is a case where the 2nd marker 3060 is not reflected in a picked-up image, the user can browse the information regarding goods.

<Hardware configuration>
The hardware configuration of a computer that implements the information processing apparatus 2000 according to the third embodiment is represented by, for example, FIG. However, the storage 1080 of the computer 1000 that implements the information processing apparatus 2000 according to the present embodiment further stores program modules that implement the functions of the information processing apparatus 2000 according to the present embodiment.

<Action and effect>
According to the information processing apparatus 2000 of the present embodiment, the operation screen 40 is displayed when the second marker 3060 appears in the captured image. Therefore, as in the case of the second embodiment, the operation screen 40 can be displayed only when the user needs it, so that the convenience of the information processing system 3000 is improved for the user. Further, it is possible to prevent the computer resources of the information processing apparatus 2000 from being wasted.

  Furthermore, if the position of the operation screen 40 is determined by the position of the second marker 3060, the operation screen 40 can be displayed at a natural and easy-to-see position for the user. For example, the marker image 84 displayed on the device 80 is treated as the second marker 3060 so that the user wears the device 80 on the wrist so that the operation screen 40 is displayed near the user's arm. can do. This improves the operability of the information processing apparatus 2000 for the user.

[Embodiment 4]
FIG. 32 is a block diagram illustrating an information processing system 3000 according to the fourth embodiment. Except as described below, the information processing system 3000 according to the fourth embodiment has the same function as the information processing system 3000 according to any one of the first to third embodiments.

  The information processing system 3000 according to the fourth embodiment includes a sensor 3080. The sensor 3080 is a sensor worn by the user of the information processing system 3000. For example, the sensor 3080 is a vibration sensor.

  In the present embodiment, when an input operation is recognized based on the position of the operating tool, the operation recognizing unit 2060 calculates the position of the operating tool shown in the captured image at a timing based on the detection result of the sensor 3080. Recognize input operations based on position. When the input operation is recognized based on the movement of the operation body, the operation recognition unit 2060 calculates the movement of the operation body in the captured image at a time including the timing based on the detection result of the sensor 3080. An input operation is recognized based on the calculated movement.

<Process flow>
FIG. 33 is a flowchart illustrating the flow of processing executed by the information processing apparatus 2000 according to the fourth embodiment. The processing from S102 to S108 in FIG. 33 is the same as the processing from S102 to S108 in FIG. In S402, the information processing apparatus 2000 grasps the detection result of the sensor 3080. In S110, the operation recognizing unit 2060 is based on the position of the operating tool shown in the captured image at the timing based on the detection result of the sensor 3080, or the movement of the operating tool shown in the captured image at the time including the timing. To recognize the input operation.

<About sensor 3080>
The sensor 3080 is an arbitrary sensor that can be used for grasping the timing of the input operation by the user. For example, the sensor 3080 is the vibration sensor described above. The place where the vibration sensor is provided is arbitrary. For example, the vibration sensor is provided inside the device 80 as described above. Further, for example, the vibration sensor may be affixed to the user's arm or hand, or may be provided on the user's clothes (such as clothes sleeves).

  When the sensor 3080 is a vibration sensor, the user applies vibration to a place where the sensor 3080 is worn or in the vicinity thereof at the timing when the input operation is performed or at a timing close thereto. For example, the user applies vibration to an arbitrary portion of the left arm portion using the right hand in a state where the device 80 including the vibration sensor is worn on the left wrist. As a result, vibration is transmitted to the vibration sensor, and the vibration is detected by the vibration sensor.

  In addition, for example, the user applies vibration to another place using a part on which the sensor 3080 is worn or the like at a timing when the input operation is performed or a timing close thereto. For example, the user applies vibration to an arbitrary place (for example, a desk) using the right hand in a state where the device 80 having the vibration sensor built into the right wrist is worn. As a result, the vibration is transmitted to the vibration sensor, and the vibration is detected by the vibration sensor.

  The sensor 3080 is not limited to a vibration sensor. For example, the sensor 3080 may be a pressure sensor or a capacitance sensor. The place where the pressure sensor or the capacitance sensor is provided is arbitrary. For example, a pressure sensor or a capacitance sensor is provided on the touch panel of the device 80. Further, for example, the pressure sensor or the capacitance sensor may be provided on a sheet or the like that is attached to or wrapped around the user's arm or hand. Further, for example, the pressure sensor and the capacitance sensor may be provided on the user's clothes (clothes sleeves, etc.).

  When the sensor 3080 is a capacitance sensor, the user touches a place where the sensor 3080 is worn. Thereby, a change in capacitance is detected by the capacitance sensor. Further, for example, when the sensor 3080 is a pressure sensor, the user applies pressure to a place where the sensor 3080 is worn. Thereby, the pressure is detected by the pressure sensor.

  Note that an operation (such as an operation of applying vibration) for performing detection by the sensor 3080 may be performed by an operating body, or may be performed using a device other than the operating body.

<Method of grasping detection result of sensor 3080: S402>
The information processing apparatus 2000 grasps the detection result of the sensor 3080 (S402). There are various methods for the information processing apparatus 2000 to grasp the detection result of the sensor 3080. The method will be exemplified below.

<< Use of wireless communication >>
For example, the information processing apparatus 2000 acquires information indicating the detection result of the sensor by performing wireless communication with the device 80 including the sensor 3080. Using this information, the information processing apparatus 2000 grasps the detection result of the sensor 3080.

  For example, the device 80 transmits a predetermined signal to the information processing apparatus 2000 at a timing when the sensor detects a vibration having a predetermined magnitude or more. In this case, the information processing apparatus 2000 can acquire the detection result of the sensor “vibration is detected by the sensor 3080” by receiving the predetermined signal.

  Further, for example, when the sensor 3080 detects a vibration having a predetermined magnitude or larger, the device 80 may transmit information indicating the time point when the vibration is detected to the information processing apparatus 2000.

<< Detection of Change in Appearance of Device 80 >>
The device 80 may change the appearance of the device 80 in response to the vibration detected by the sensor 3080. In this case, the information processing apparatus 2000 grasps the detection result of the sensor 3080 (vibration is detected by the sensor 3080) by detecting a change in the appearance of the device 80 using the captured image generated by the camera 20. To do.

  For example, when the device 80 includes a display device, the device 80 changes the display of the display device when vibration of a predetermined magnitude or more is detected by the sensor 3080. More specifically, the device 80 changes the image displayed on the display device of the device 80 or nothing is displayed when vibration of a predetermined magnitude or more is detected by the sensor 3080. A new image is displayed on the display device. The information processing apparatus 2000 detects that the display on the display device of the device 80 has been changed by analyzing captured images repeatedly generated by the camera 20. As a result, the information processing apparatus 2000 recognizes that vibration has been detected by the sensor. FIG. 34 is a diagram illustrating a state in which an image displayed on the display device of the device 80 is changed when vibration is detected by the sensor 3080.

  Further, for example, when vibration is detected by the sensor 3080, the device 80 may turn on or blink a light such as a backlight of the display device of the device 80 or an LED (Light Emitting Diode) light provided in the device 80. Good. The information processing apparatus 2000 detects lighting and blinking of these lights by analyzing captured images repeatedly generated by the camera 20. As a result, the information processing apparatus 2000 recognizes that vibration has been detected by the sensor 3080.

  As described above, if the information processing apparatus 2000 grasps the detection of the vibration by changing the appearance of the device 80 according to the vibration detected by the sensor 3080, wireless communication is performed between the device 80 and the information processing apparatus 2000. There is no need to do it. Therefore, the device 80 and the information processing apparatus 2000 do not have to have a wireless communication function unless there is a need for wireless communication.

  The method for the information processing apparatus 2000 to grasp the detection result by a sensor other than the vibration sensor is the same as the method described above for the vibration sensor. For example, when the sensor is a pressure sensor, the device 80 and the information processing apparatus 2000 are the same as when the vibration sensor detects a vibration of a predetermined magnitude or more when the pressure sensor detects a pressure of a predetermined magnitude or more. Perform the process. Further, for example, when the sensor is a capacitance sensor, the device 80 or the information processing apparatus 2000 has a vibration sensor of a predetermined size when the capacitance sensor detects a displacement of the capacitance of a predetermined size or more. The same processing as that when the above vibration is detected is performed.

<Calculation of the position of the operating tool>
The operation recognition unit 2060 detects the operating tool at a timing based on the detection result of the sensor 3080, and calculates the position of the detected operating tool. Hereinafter, “timing based on the detection result of the sensor 3080” is referred to as detection target timing.

  The detection target timing is a timing at which vibration or the like is detected by the sensor 3080 or a timing close thereto. For example, when a predetermined signal is transmitted from the device 80 to the information processing apparatus 2000 when vibration or the like is detected by the sensor 3080, the detection target timing is a time point when the information processing apparatus 2000 receives the predetermined signal. Further, for example, when information indicating the time when vibration or the like is detected by the sensor 3080 is transmitted from the device 80 to the information processing apparatus 2000, the detection target timing is the time indicated by this information. Further, for example, when a predetermined change is added to the appearance of the device 80 when vibration or the like is detected by the sensor 3080, the detection target timing is a point in time when the predetermined change is detected by the information processing apparatus 2000.

  In addition, the operation recognition unit 2060 may set a timing before or after a predetermined time before the various timings described above (such as when the information processing apparatus 2000 receives a predetermined signal from the device 80) as a detection target timing.

  Information indicating which time point the operation recognizing unit 2060 handles as the detection target timing and information indicating the predetermined time described above may be set in advance in the operation recognizing unit 2060 or accessed from the operation recognizing unit 2060. It may be stored in a possible storage device.

  The operation recognizing unit 2060 detects the operating tool from the captured image generated by the camera 20 at the detection target timing, and calculates the position of the operating tool. Here, in general, since the camera 20 intermittently generates a captured image (for example, a frequency of 30 fps (frame / second)), there may be no captured image generated at a time that coincides with the detection target timing. In this case, for example, the operation recognition unit 2060 uses a captured image generated immediately before or after the detection target timing.

<Method for calculating movement of operating body>
The operation recognizing unit 2060 detects the operating tool at a time including the detection target timing, and calculates the motion of the operating tool. Hereinafter, “time including detection target timing” is referred to as detection target time. The operation recognizing unit 2060 calculates the movement of the operating body using the captured image generated during the detection target time.

  The operation recognition unit 2060 determines the start time point and the end time point of the detection target time using the detection target timing. For example, the operation recognition unit 2060 sets the detection target timing as the start time of the detection target time. FIG. 35 is a diagram illustrating a case where the detection target timing is set as the start time of the detection target time. In the case of FIG. 35A, the operation recognizing unit 2060 uses the time when vibration is detected by the sensor 3080 as the detection target timing. In the case of FIG. 35B, the operation recognizing unit 2060 uses a time point a predetermined time before the time point at which the vibration is detected by the sensor 3080 as the detection target timing. In the case of FIG. 35C, the operation recognizing unit 2060 uses a time point after a predetermined time from the time point when the vibration is detected by the sensor 3080 as the detection target timing.

  For example, the operation recognition unit 2060 sets the detection target timing as the end point of the detection target time. FIG. 36 is a diagram illustrating a case where the detection target timing is set as the end point of the detection target time. In the case of FIG. 36A, the operation recognizing unit 2060 uses the time when vibration is detected by the sensor 3080 as the detection target timing. In the case of FIG. 36B, the operation recognizing unit 2060 uses a time point a predetermined time before the time point at which the vibration is detected by the sensor 3080 as the detection target timing. In the case of FIG. 36C, the operation recognizing unit 2060 uses a time point a predetermined time after the time point when the vibration is detected by the sensor 3080 as the detection target timing.

  The operation recognition unit 2060 determines the detection target time by using one or two detection target timings. When determining the detection target time using one detection target timing, for example, the operation recognition unit 2060 determines the start time of the detection target time by any of the methods in FIG. The end of time. Further, for example, the operation recognition unit 2060 determines the end point of the detection target time by any one of the methods in FIG. 36, and sets a time point before that time as the end point of the detection target time.

  When determining the detection target time using two detection target timings, the operation recognizing unit 2060 determines the start time of the detection target time by any of the methods in FIG. 35, and also detects the detection target time by any of the methods in FIG. Determine the end point of.

  In addition, when the time before the time when vibration is detected by the sensor 3080 is set as the start time of the detection target time (FIG. 35A or FIG. 35B), only one detection target timing is used, and FIG. As described above, when the end point of the detection target time is determined by the detection target timing, the camera 20 needs to generate a captured image before vibration is detected by the sensor 3080. In this case, the camera 20 starts imaging before vibration is detected by the sensor 3080. For example, the camera 20 continuously captures an image from when the user starts using the information processing apparatus 2000 to when it ends. The captured image generated by the camera 20 is stored in a storage device or the like continuously for a predetermined period.

  On the other hand, when the time point after the time point at which the vibration is detected by the sensor 3080 is set as the start point of the detection target time (FIG. 35C), the camera 20 starts imaging after the vibration is detected by the sensor 3080. May be. In this case, for example, the camera 20 receives a signal indicating that vibration is detected by the sensor 3080 from the device 80 or the information processing apparatus 2000, and starts imaging at the timing when the signal is received.

  However, when the product 10 is recognized using the captured image, the timing at which the camera 20 starts imaging starts at a timing necessary for the recognition of the product 10. For example, in this case, the camera 20 always generates a captured image from when the information processing apparatus 2000 is started to when it is stopped.

  Note that the information representing each predetermined time described above may be set in advance in the operation recognition unit 2060 or may be stored in a storage device accessible from the operation recognition unit 2060. Further, the predetermined time used for determining the start time of the detection target time and the predetermined time used for the end time of the detection target time may be the same or different.

<Hardware configuration example>
The hardware configuration of a computer that implements the information processing apparatus 2000 according to the fourth embodiment is represented by, for example, FIG. However, the storage 1080 of the computer 1000 that implements the information processing apparatus 2000 according to the present embodiment further stores program modules that implement the functions of the information processing apparatus 2000 according to the present embodiment.

<Action and effect>
If the user's input operation is to be recognized using only the position or movement of the operating tool in the captured image, the input operation is erroneously recognized even though the user has not performed the input operation. There is a possibility that the input operation is not recognized even though the input operation is performed.

  According to the information processing system 3000 of this embodiment, the position of the operating body at the timing based on the detection result of the sensor 3080 worn by the user or the movement of the operating body at the time including the timing based on the detection result of the sensor 3080. Input operation is recognized. Therefore, if the user performs an input operation such that detection by the sensor 3080 is performed during the input operation, the position and movement of the operating body based on the timing is highly likely to represent the user's input operation. Therefore, the input operation intended by the user is recognized correctly, and the input operation is erroneously recognized even when the user is not performing the input operation, or the user is performing the input operation. Nevertheless, it is possible to prevent the input operation from being recognized.

[Example]
Hereinafter, the information processing system 3000 will be described by exemplifying a more specific usage method. Each example described later is merely an example of a method of using the information processing system 3000, and the method of using the information processing system 3000 is not limited to that described in the example described later.

  FIG. 37 is a block diagram illustrating the configuration of an information processing system 3000 common to the embodiments. The information processing system 3000 includes an information processing device 2000, a display device 3020, a first marker 3040, a second marker 3060, and a sensor 3080.

  FIG. 38 is a diagram illustrating a usage environment of the information processing system 3000 common to the embodiments. In each embodiment, the information processing apparatus 2000 is mounted inside the head mounted display 100. The head mounted display 100 includes a display device 102 for realizing the display device 3020. The camera 20 is provided on the head mounted display 100.

  The user wears the device 80 on the wrist. A marker image 84 is displayed on the touch panel 82 of the device 80. The marker image 84 is handled as both the first marker 3040 and the second marker 3060 by the information processing apparatus 2000. That is, the product recognition unit 2020 recognizes the product 10 included in the product recognition area 70 determined by the marker image 84. In addition, the display control unit 2040 displays the operation screen 40 on the display device 3020 when the marker image 84 is captured.

  The display control unit 2040 determines a position where the operation screen 40 is displayed based on the position of the marker image 84. Here, the operation screen 40 includes either the input area 42 or the information area 44, or both the input area 42 and the information area 44. The input area 42 is an area including an input interface such as a keyboard or a button, and the information area 44 is an area not including an input interface.

  The display positions of the input area 42 and the information area 44 are respectively determined based on the position of the marker image 84. Specifically, the display position of the input area 42 is determined as a position superimposed on the user's arm. On the other hand, the information area 44 is displayed so that the display position of the information area 44 is located above the user's arm.

  In each embodiment, the operation body is the user's finger 30. The device 80 includes a vibration sensor 86 for realizing the sensor 3080. The operation recognition unit 2060 recognizes an input operation based on the position of the finger 30 at the detection target timing determined based on the detection result by the vibration sensor 86 or the movement of the finger 30 at the detection target time.

  For example, the user taps each key included in the input area 42 representing the keyboard with the finger 30. When the vibration of the tap is detected by the vibration sensor 86, the device 80 notifies the operation recognition unit 2060. The operation recognition unit 2060 receives this notification, determines the position of the finger 30 at the timing when vibration is detected, and recognizes an input operation for inputting a key corresponding to the position. The user moves the finger 30 after tapping the arm 32 with the finger 30, and taps the arm 32 with the finger 30 again at the timing when the operation is finished. In this way, the operation recognition unit 2060 causes the operation recognition unit 2060 to recognize the movement of the finger 30. For example, the user performs handwriting input by the movement of the finger 30.

[[Example 1]]
Example 1 is an example in which the head mounted display 100 is used by a customer who visits a store. In this store, the device 80 and the head mounted display 100 are lent to each customer who visits the store. Each head-mounted display 100 is assigned an individual ID. The customer wears the device 80 and the head-mounted display 100 and looks around the product 10 in the store.

  In this store, the customer does not have to carry the product he / she wants to buy in the product basket. Instead, the customer purchases the product 10 using the head mounted display 100. For this purpose, a sample of the product 10 is displayed at the store.

  First, the customer walks through the store and searches for a product 10 that he / she wishes to purchase. When the customer finds a product 10 that is attracting interest, the customer causes the head mounted display 100 to recognize the product 10. Specifically, the customer holds the product 10 with the hand to which the device 80 is attached and views the product 10. In this way, the product recognition unit 2020 recognizes the product 10. Further, the display control unit 2040 displays the operation screen 40 including the product information of the product 10 on the display device 102.

  FIG. 39 is a diagram illustrating an operation screen 40 displayed on the display apparatus 102 in the first embodiment. The display device 102 displays an information area 44 including a description of the product 10 and an input area 42 for inputting the number of products 10 to be purchased. The input area 42 includes a numeric key for inputting the number, an OK key for determining purchase, and a cancel key for canceling purchase.

  The user looks at the information area 44 and confirms information on the product 10 (product name and price in FIG. 39). In order to purchase the product 10, the user inputs the number of items to be purchased using the numerical keys, and then taps the OK key. On the other hand, when the product 10 is not purchased, the user taps the cancel key.

  When the OK key is tapped, the product 10 is added to the order information associated with the ID of the head mounted display 100. FIG. 40 is a diagram illustrating order information in a table format. The table shown in FIG. 40 is referred to as an order information table 700. The order information table 700 is associated with the ID (HMD_ID) of the head mounted display 100. The order information table 700 includes a product ID 702 and a number 704. The product ID 702 indicates the ID of the product 10. The number 704 indicates the number of items for purchase. When the OK key is tapped by the operation described with reference to FIG. 39, a new line is added to the order information table 700.

  When the operation described with reference to FIG. 39 is finished for each product 10 to be purchased, the user goes to the checkout counter and performs checkout. At the checkout counter, the user returns the device 80 and the head mounted display 100 to the store clerk. The store clerk who receives the head mounted display 100 reads the ID of the head mounted display 100 with a computer provided in the checkout counter. This computer acquires the order information table 700 associated with the ID of the head mounted display 100 to calculate the settlement amount, and displays the settlement amount on a display device that can be viewed by the customer. After paying the settlement amount, the user receives the product from the store clerk.

  The payment may be made without using cash. For example, the user may use the head mounted display 100 to input credit card information, and pay the price with the credit card.

  The head mounted display 100 preferably has a function corresponding to multiple languages. When the head-mounted display 100 has a function corresponding to multiple languages, for example, when the user rents the head-mounted display 100, the user tells the store clerk the language that the user wants to use. Here, it is assumed that the language that the user wants to use is English. In this case, the store clerk lends the head mounted display 100 to the user after changing the language setting of the head mounted display 100 to English. Thus, the information displayed in the information area 44 and the language of the input key displayed in the input area 42 are written in English.

  By providing the head-mounted display 100 with functions corresponding to multiple languages in this manner, people from various countries can easily use the head-mounted display 100, and consequently, it is easy to use the store where the head-mounted display 100 is introduced. . Note that the language setting of the head mounted display 100 may be changed by the user.

[[Example 2]]
The second embodiment is an example in which the head mounted display 100 is used in a concierge service via the Internet. The concierge service here means a service that accepts a request regarding the product 10 from a customer and provides the user with the product 10 that meets the request.

  FIG. 41 is a diagram illustrating a configuration of a concierge system 600 that provides the concierge service according to the second embodiment. The concierge system 600 includes a server 610, a device 80, and a head mounted display 100. In the concierge system 600, each concierge staff wears a set of the device 80 and the head mounted display 100.

  The customer terminal 620 is various terminals (for example, PC) used by customers. The customer terminal 620 is connected to the server 610 via the Internet. When the server 610 receives a service provision request from the customer terminal 620, the server 610 allocates the request to one of the head mounted displays 100. The concierge staff wearing the head mounted display 100 to which the request is assigned provides a service to the user (customer) of the customer terminal 620. Here, the head mounted display 100 is connected to the server 610 via a LAN (Local Area Network).

  The concierge staff receives a request regarding the product 10 from a customer. For example, the customer conveys the request by voice through a microphone connected to the customer terminal 620. The customer's voice is transmitted to the server 610 via the customer terminal 620. In addition, since various existing techniques can be used for the service provider to listen to the customer's voice transmitted to the server via the Internet, description of the method is omitted.

  The concierge staff finds the product 10 that meets the customer's request and views the product 10 through the head mounted display 100. By doing so, the operation screen 40 including the product information of the product 10 is displayed on the display device 102. When the customer likes the product 10, the concierge staff asks the customer how many pieces he / she wants to purchase and performs an input operation to input the number on the operation screen 40. Here, the information displayed on the display device 102 of the head mounted display 100 worn by the concierge staff is the same as the input area 42 and the information area 44 shown in FIG. 39, for example.

  The customer may receive information related to the product 10 by voice from the concierge staff, or may browse information related to the product 10 using the display device of the customer terminal 620. In the latter case, for example, the server 610 provides the customer terminal 620 with the same information as the information being browsed by the concierge staff or information similar thereto. By doing so, the same information as the information being browsed by the concierge staff or information similar thereto is displayed on the display device of the customer terminal 620, and the customer can browse the information.

  For example, the server 610 transmits an image generated by the camera 20 attached to the head mounted display 100 to the customer terminal 620. In addition, for example, the server 610 transmits the video (the video generated by the camera 20 with the operation screen 40 superimposed) reflected on the display device 102 to the customer terminal 620. Further, for example, the server 610 may transmit only the operation screen 40 to the customer terminal 620 without transmitting the video generated by the camera 20 to the customer terminal 620.

  The operation screen 40 transmitted to the customer terminal 620 may be different from the operation screen 40 being browsed by the concierge staff. For example, in the case of the present embodiment, since the customer does not perform an input operation using the head mounted display 100, it is not necessary to look at the input area 42. Therefore, the head mounted display 100 includes the input area 42 and the information area 44 in the operation screen 40 displayed on the display device 102, while the information area 44 is included in the operation screen 40 transmitted to the customer terminal 620 via the server 610. Only to be included.

  Moreover, the merchandise information browsed by the concierge staff and the merchandise information transmitted to the customer terminal 620 may be different. For example, the head mounted display 100 includes various management information necessary for product management in the product information browsed by the concierge staff, while the product information transmitted to the customer terminal 620 includes such management. Do not include information.

  As described above, when the operation screen 40 displayed on the display device 102 is different from the operation screen 40 transmitted to the customer terminal 620, the template 200 for the display device 102 is used as the template 200 of the operation screen 40 described above. Both templates for the customer terminal 620 are prepared.

[[Example 3]]
The third embodiment is an example in which the head mounted display 100 is used in an operation of replenishing products to a vending machine. An operator who replenishes products to the vending machine uses the head mounted display 100 to view various information related to the replenished product 10 and to input information related to the product 10.

  For example, the worker uses the head mounted display 100 to perform an input operation for recording the number of replenished products 10. FIG. 42 is a diagram illustrating a state in which the head mounted display 100 is used in the third embodiment. The product storage 800 is a storage in which products are stored in the vending machine. In addition, in order to make a figure legible, in FIG. 42, only a part of goods storage 800 is illustrated.

  First, the worker replenishes the product 10 and then causes the head mounted display 100 to recognize the product 10 by viewing the product 10 that is visible from the product storage 800 through the head mounted display 100.

  In this embodiment, the device 80 is mounted so that the touch panel 82 can be seen on the back of the hand. The product recognition area 70 is a rectangle that touches the upper end of the marker image 84. Therefore, the worker holds the marker image 84 near the product 10 visible from the product storage 800 so that the product 10 is included in the product recognition area 70.

  When the product 10 is recognized by the head mounted display 100, the operation screen 40 (input area 42 and information area 44) is displayed on the display device 102. The worker uses the numeric keys included in the operation screen 40 to perform an input operation for inputting the replenished number.

  Note that the operation screen 40 may include a memo field in which an operator can freely describe information. For example, for each product 10, the worker inputs information such as whether the number of products 10 to be brought at the next replenishment should be increased or decreased.

  In the present embodiment, the head mounted display 100 may have a function of specifying a work vending machine and acquiring a list of products 10 to be put into the specified vending machine. The identification of the vending machine is realized, for example, by reading an ID attached to the vending machine.

  For example, it is assumed that a QR code (registered trademark) that encodes the ID of the vending machine is attached to the vending machine. In this case, the worker views the QR code (registered trademark) via the head mounted display 100. As a result, the camera 20 generates a captured image that includes the QR code (registered trademark). The head mounted display 100 recognizes the ID of the vending machine by analyzing the captured image.

  The head mounted display 100 searches a database in which the ID of the vending machine and the list of the products 10 to be put in the vending machine specified by the ID are associated, and acquires the list of the products 10. In this case, the head mounted display 100 is communicably connected to the database. However, the list of products 10 to be put into the vending machine may be stored in advance in a storage device inside the head mounted display 100.

  Note that the identification of the vending machine to be worked may be performed manually. In this case, the worker performs an input operation to select a vending machine to be worked on the head mounted display 100. For example, the head mounted display 100 displays a list of work vending machines on the display device 102. From the list, the worker selects a vending machine that is a target of work to be performed.

  The head mounted display 100 may have a function of communicating with a vending machine. In this case, for example, the vending machine transmits information indicating the number of items 10 to be replenished to the head mounted display 100. The head mounted display 100 displays the information on the display device 102. The worker can grasp the number of products 10 to be replenished by looking at the information.

  The vending machine may have a function of counting the number of replenished products 10 and notifying the head mounted display 100 of the number of replenished products 10. In this case, recording of the number of replenished products 10 is automatically performed by the head mounted display 100. Therefore, the worker does not need to perform an input operation for inputting the number of the replenished products 10.

[[Example 4]]
The fourth embodiment is an example in which the head mounted display 100 is used for collecting products ordered by online shopping from a store shelf in a supermarket that provides online shopping services. When a customer orders a product through online shopping, a list of the ordered product is transmitted to the store server. The store clerk walks around the store and collects the products on the list from the product shelves. At that time, the store clerk wears the head mounted display 100 to perform the collection work.

  The store server acquires a list of ordered products and a map of the store. Further, the store server searches the product database and specifies the position on the store where each product listed in the ordered product list is placed. The position of each product is registered in advance in the product database. And a store server produces | generates the map on which the mark which shows the position of each goods was superimposed.

  The head mounted display 100 acquires the map from the store server and displays the map on the display device 102. FIG. 43 is a diagram illustrating an example of a store map displayed on the display device 102. In FIG. 43, a map 300 is displayed on the display device 102. A product position 302 indicating the position of each product on the list (the position of each product to be collected by the store clerk) is superimposed on the map 300. The store clerk can collect the target product easily by collecting the product while looking at the map 300.

  The store clerk looks at the map displayed on the display device 102 and goes to the product shelf where the products to be collected are displayed. When the sales clerk finds a product to be collected, the clerk looks at the product via the head mounted display 100. The head mounted display 100 displays an operation screen 40 including product information of the product on the display device 102.

  FIG. 44 is a diagram illustrating an operation screen 40 displayed on the display device 102 in the fourth embodiment. FIG. 44A illustrates a case where the product 10 taken by the store clerk is a product to be collected. The operation screen 40 displays the number of products 10 to be collected. The store clerk taps the OK button after putting the number to be collected into the basket. On the other hand, when the collection of the product 10 is postponed, the store clerk taps a cancel button.

  On the other hand, FIG. 44B illustrates a case where the product 10 taken by the store clerk is not a product to be collected. The operation screen 40 displays a message notifying that the product picked up is not the product to be collected (not the ordered product). The user who has confirmed this message taps the OK button.

  In addition, in the case of a store clerk who is unfamiliar with a store such as a new store clerk, even if the map 300 displayed on the display device 102 is viewed, the target product may not be found. In such a case, the head mounted display 100 may provide a means for contacting the sales staff at each sales floor.

  FIG. 45 is a diagram illustrating the display device 102 on which the operation screen 40 for contacting the sales staff is displayed. The information area 44 includes a picture of the salesperson in charge. The input area 42 includes a button indicating the name of the sales person in charge. The clerk taps the button with the name of the salesperson who wants to contact. Thereby, for example, a voice call is established between the store clerk and the sales representative. In this case, the head mounted display 100 provides a function of performing a voice call.

[[Example 5]]
Example 5 is a case where the head mounted display 100 is used for management of goods in a store. The head mounted display 100 is worn by a store clerk. A store clerk wearing the head mounted display 100 performs an ordering work for the product 10 using the head mounted display 100, for example.

  FIG. 46 is a diagram illustrating information displayed on the display device 102 in the fifth embodiment. The display device 102 displays an information area 44 that includes information related to the product 10 included in the product recognition area 70 and an input area 42 for an input operation. Here, the information area 44 includes an information area 44-1 and an information area 44-2. In the information area 44-1, a table showing the product name, the inventory quantity, and the order quantity of the recognized product 10 is displayed. In the information area 44-2, a large image of the recognized product 10 is displayed. Here, the image of the product 10 displayed in the information area 44-2 may be an enlarged image of the product 10 extracted from the captured image, or is registered in the product database (for example, the product information table 500). It may be an image of the product 10.

  The store clerk determines the number of orders for the product 10 by looking at the number of stocks of the product 10 indicated in the information area 44. Then, the store clerk uses the numeric keys in the input area 42 to input the number of products ordered. In FIG. 46, “21” is input as the number of orders. The clerk who has finished inputting the number of orders taps the OK key. By doing so, the product 10 is ordered.

  As mentioned above, although embodiment of this invention was described with reference to drawings, these are illustrations of this invention, The combination of said each embodiment or various structures other than the above can also be employ | adopted.

Hereinafter, examples of the reference form will be added.
1. A display device;
An information processing device,
The information processing apparatus includes:
Product recognition means for recognizing products,
Display control means for acquiring product information relating to the recognized product and displaying an operation screen including the acquired product information on the display device;
An operation recognizing means for recognizing an input operation on the operation screen based on the position or movement of the operation body shown in the captured image generated by the camera;
An information processing system.
2. Whether the product recognition means recognizes the product shown in the captured image using the captured image, or recognizes the product by reading the identifier of the product from a tag attached to the product or built in Or the product is recognized by reading the product identifier from the product information symbol attached to the product. Information processing system described in 1.
3. Having a marker that is worn by the user or that is part of the user's body;
The product recognition means
From the captured image, extract a product recognition area determined based on the position of the marker in the captured image,
1. Recognize a product shown in the extracted product recognition area. Information processing system described in 1.
4). Having a marker that is worn by the user or that is part of the user's body;
The display control means causes the display device to display the operation screen when the marker is reflected in the captured image. To 3. The information processing system according to any one of the above.
5. 3. The display control unit displays the operation screen at a position determined based on the position of the marker in the captured image. Information processing system described in 1.
6). 2. The marker is an image displayed on a display device of a device worn by the user. To 5. The information processing system according to any one of the above.
7). The display control means transmits the operation screen or the captured image on which the operation screen is superimposed via a communication line. To 6. The information processing system according to any one of the above.
8). Having a sensor worn by the user,
The operation recognizing unit detects the position of the operating body that appears in the captured image at a timing based on the detection result of the sensor, or the movement of the operating body that appears in the captured image at a time including the timing. And recognizing an input operation based on the detected position or movement. To 7. The information processing system according to any one of the above.
9. Product recognition means for recognizing products,
Display control means for acquiring product information relating to the recognized product and displaying an operation screen including the acquired product information on a display device;
An operation recognizing means for recognizing an input operation on the operation screen based on the position or movement of the operation body shown in the captured image generated by the camera;
An information processing apparatus.
10. Whether the product recognition means recognizes the product shown in the captured image using the captured image, or recognizes the product by reading the identifier of the product from a tag attached to the product or built in 8. Recognize the product by reading the product identifier from the product information symbol attached to the product. The information processing apparatus described in 1.
11. The product recognition means
From the captured image, extract a product recognition area determined based on the position of the marker in the captured image,
Recognizing the product shown in the extracted product recognition area,
9. The marker is worn by the user or is part of the user's body. The information processing apparatus described in 1.
12 The display control means displays the operation screen on the display device when a marker is reflected in the captured image,
8. the marker is worn by the user or is part of the user's body; To 11. The information processing apparatus according to any one of the above.
13. The display control means displays the operation screen at a position determined based on the position of the marker in the captured image. The information processing apparatus described in 1.
14 10. The marker is an image displayed on a display device of a device worn by the user. Thru 13. The information processing apparatus according to any one of the above.
15. 8. The display control means transmits the operation screen or the captured image on which the operation screen is superimposed via a communication line. To 14. The information processing apparatus according to any one of the above.
16. The operation recognizing unit detects the position of the operating body in the captured image at a timing based on a detection result of a sensor worn by the user, or is captured in the captured image at a time including the timing. 8. detecting the movement of the operating body and recognizing an input operation based on the detected position or movement; To 15. The information processing apparatus according to any one of the above.
17. A control method executed by a computer,
A product recognition step for recognizing the product;
A display control step of acquiring product information related to the recognized product and displaying an operation screen including the acquired product information on a display device;
An operation recognizing step for recognizing an input operation on the operation screen based on the position or movement of the operation body reflected in the captured image generated by the camera;
A control method.
18. Whether the product recognition step recognizes the product shown in the captured image using the captured image, or recognizes the product by reading the identifier of the product from a tag attached to the product or built in Or recognizing the product by reading the product identifier from the product information symbol attached to the product. The control method described in 1.
19. The product recognition step includes:
From the captured image, extract a product recognition area determined based on the position of the marker in the captured image,
Recognizing the product shown in the extracted product recognition area,
The marker is worn by the user or is a part of the user's body, 18. The control method described in 1.
20. The display control step displays the operation screen on the display device when a marker is reflected in the captured image,
16. the marker is worn by the user or is part of the user's body; Thru 19. The control method as described in any one.
21. The display control step displays the operation screen at a position determined based on the position of the marker in the captured image; The control method described in 1.
22. 18. The marker is an image displayed on a display device of a device worn by the user. Thru 21. The control method as described in any one.
23. 16. The display control step transmits the operation screen or the captured image on which the operation screen is superimposed via a communication line. Thru 22. The control method as described in any one.
24. The operation recognizing step detects the position of the operating body in the captured image at a timing based on a detection result of a sensor worn by the user, or is captured in the captured image at a time including the timing. 16. detecting the movement of the operating body and recognizing an input operation based on the detected position or movement; Thru 23. The control method as described in any one.
25. On the computer,
A product recognition step for recognizing the product;
A display control step of acquiring product information related to the recognized product and displaying an operation screen including the acquired product information on a display device;
An operation recognizing step for recognizing an input operation on the operation screen based on the position or movement of the operation body reflected in the captured image generated by the camera;
A program that executes
26. Whether the product recognition step recognizes the product shown in the captured image using the captured image, or recognizes the product by reading the identifier of the product from a tag attached to the product or built in Or recognizing the product by reading the product identifier from the product information symbol attached to the product, 25. The program described in.
27. The product recognition step includes:
From the captured image, extract a product recognition area determined based on the position of the marker in the captured image,
Recognizing the product shown in the extracted product recognition area,
The marker is worn by the user or is a part of the user's body, 26. The program described in.
28. The display control step displays the operation screen on the display device when a marker is reflected in the captured image,
The marker is worn by the user or is part of the user's body; 25. Thru 27. The program according to any one of the above.
29. The display control step displays the operation screen at a position determined based on the position of the marker in the captured image; 28. The program described in.
30. 27. The marker is an image displayed on a display device of a device worn by the user. Thru 29. The program according to any one of the above.
31. The display control step transmits the operation screen or the captured image on which the operation screen is superimposed via a communication line; 25. Thru 30. The program according to any one of the above.
32. The operation recognizing step detects the position of the operating body in the captured image at a timing based on a detection result of a sensor worn by the user, or is captured in the captured image at a time including the timing. 25. detecting the movement of the operating body and recognizing an input operation based on the detected position or movement; Thru 31. The program according to any one of the above.

10 product 20 camera 22 captured image 24 predetermined region 26 guide 30 finger 32 arm 40 operation screen 42 input area 44 information area 46 arrow 50 movement 51 shape 52 shape 54 rectangle 56 circle 60 region 62 center of gravity 64 point 70 product recognition region 72 arrow 80 Device 82 Touch panel 84 Marker image 86 Vibration sensor 87 Coordinate system 100 Head mounted display 102 Display device 120 Reference image 121 Position 130 Area 131 Point 200 Template 202 Replacement area 300 Map 302 Product position 500 Product information table 502 Product ID
504 Feature quantity 506 Product information 508 Product name 510 Price 512 Description 600 Concierge system 610 Server 620 Customer terminal 700 Order information table 702 Product ID
704 Number 800 Product storage 1000 Computer 1020 Bus 1040 Processor 1060 Memory 1080 Storage 1100 Input / output interface 2000 Information processing device 2020 Product recognition unit 2020 Recognition unit 2040 Display control unit 2060 Operation recognition unit 3000 Information processing system 3020 Display device 3040 First marker 3060 Second marker 3080 sensor

Claims (14)

An object recognition means for extracting an object recognition area, which is a partial area of the captured image, and recognizing an object included in the extracted object recognition area;
Display control means for acquiring object information relating to the recognized object, and displaying the acquired object information and an operation screen relating to the object on a display device;
An operation recognizing means for recognizing an input operation on the operation screen based on at least one of a position and a movement of the operation body included in the captured image;
Have
The object recognition means extracts the object recognition area determined based on the position of the first marker included in the captured image from the captured image;
The display control unit displays the operation screen on the display device when the captured image includes a second marker,
Whether at least one of the first marker and the second marker is an image displayed on a display device of a device worn by a user, or is drawn, printed, or attached to the device, An information processing apparatus that is worn by the user or is a part of the user's body.   The object recognition unit recognizes an object included in the captured image by analyzing the captured image, or recognizes the object by reading an identifier of the object from a symbol attached to the object. The information processing apparatus described in 1.   The information processing apparatus according to claim 1, wherein the object recognition unit extracts the object recognition area determined based on a position of a marker included in the captured image from the captured image.   The information processing apparatus according to claim 1, wherein the display control unit displays the operation screen on the display device when a marker is included in the captured image.   The information processing apparatus according to claim 4, wherein the display control unit displays the operation screen at a position determined based on a position of the marker included in the captured image.   The information processing apparatus according to claim 1, wherein the first marker and the second marker are the same.   The display control unit displays the operation screen at a position different from a position on the display screen corresponding to a position of the object recognition area in the captured image on the display screen of the display device. 6. The information processing apparatus according to any one of 6.   The information processing apparatus according to claim 1, wherein the object recognition area is a predetermined area.   The information processing apparatus according to any one of claims 1 to 8, wherein the display control unit transmits the operation screen or a captured image on which the operation screen is superimposed via a communication line.   The operation recognizing unit detects the position of the operation body included in the captured image at a timing determined based on a detection result of a sensor provided in a device worn by the user, or captures an image at a time determined based on the timing. The information processing apparatus according to claim 1, wherein a motion of the operation body included in an image is detected and an input operation is recognized based on the detected position or motion.   The information processing apparatus according to claim 1, wherein the object is a product. A display device;
An information processing device,
The information processing apparatus includes:
An object recognition means for extracting an object recognition area, which is a partial area of the captured image, and recognizing an object included in the extracted object recognition area;
Display control means for acquiring object information relating to the recognized object, and displaying the acquired object information and an operation screen relating to the object on a display device;
An operation recognizing means for recognizing an input operation on the operation screen based on at least one of a position and a movement of the operation body included in the captured image;
Have
The object recognition means extracts the object recognition area determined based on the position of the first marker included in the captured image from the captured image;
The display control unit displays the operation screen on the display device when the captured image includes a second marker,
Whether at least one of the first marker and the second marker is an image displayed on a display device of a device worn by a user, or is drawn, printed, or attached to the device, An information processing system that is worn by the user or is a part of the user's body. A control method executed by a computer,
An object recognition step of extracting an object recognition area, which is a partial area of the captured image, and recognizing an object included in the extracted object recognition area;
A display control step of acquiring object information related to the recognized object, and displaying the acquired object information and an operation screen related to the object on a display device;
An operation recognition step for recognizing an input operation on the operation screen based on at least one of the position and the movement of the operation body included in the captured image;
Have
In the object recognition step, the object recognition area determined based on the position of the first marker included in the captured image is extracted from the captured image,
In the display control step, when the second marker is included in the captured image, the operation screen is displayed on the display device,
Whether at least one of the first marker and the second marker is an image displayed on a display device of a device worn by a user, or is drawn, printed, or attached to the device, A control method that is worn by the user or is part of the user's body. On the computer,
An object recognition step of extracting an object recognition area, which is a partial area of the captured image, and recognizing an object included in the extracted object recognition area;
A display control step of acquiring object information related to the recognized object, and displaying the acquired object information and an operation screen related to the object on a display device;
An operation recognition step for recognizing an input operation on the operation screen based on at least one of the position and the movement of the operation body included in the captured image;
And execute
In the object recognition step, the object recognition area determined based on the position of the first marker included in the captured image is extracted from the captured image,
In the display control step, when the second marker is included in the captured image, the operation screen is displayed on the display device,
Whether at least one of the first marker and the second marker is an image displayed on a display device of a device worn by a user, or is drawn, printed, or attached to the device, A program that is worn by or is part of the user's body.