JP6651777B2 - Information processing apparatus, information processing method and program - Google Patents

Description

  The present invention relates to an information processing device, an information processing method, and a program.

  2. Description of the Related Art Augmented Reality (AR) technology for synthesizing and displaying a virtual object on a captured image indicating a real space captured by a camera has been conventionally known. For example, when a user captures an image of a paper or the like (AR marker) including a predetermined pattern, an AR technology that displays a virtual object three-dimensionally on a captured image in accordance with the positional relationship between the paper and the camera is known. By referring to the virtual object displayed on the captured image, the user can grasp an image when the user actually arranges the object corresponding to the virtual object.

  Patent Literature 1 discloses an invention of a mixed reality display device that presents to a user a virtual reality world in which the appearance, such as the size and color of the real object, and the surrounding environment are harmonized without placing the real object in the real space. Is disclosed.

  However, in the related art, even when a virtual object corresponding to an object is displayed on a captured image, it is difficult to grasp an image when a movable part of the object is operated.

  The present invention has been made in view of the above, and provides an information processing apparatus, an information processing method, and a program that allow a user to more clearly understand an image when a movable part of an object is operated. Aim.

In order to solve the above-described problems and achieve the object, the present invention obtains a captured image including position identification information for identifying a position at which a virtual object represented by an image having a stationary part and a movable part is displayed. An imaging unit that generates a synthesized image in which the virtual object is synthesized at a position on the captured image identified by the position identification information, and a generation unit that is included in the synthesized image displayed on a display unit. An operation receiving unit that receives an operation of a movable part, and a display that controls display of the movable part based on display setting information for displaying the virtual object when an operation of the movable part is received by the operation receiving unit. and a control unit, wherein the operation receiving unit further accepts an operation for specifying an installation restriction region showing the limit region where the object corresponding to the virtual object is placed, the generation unit, the installation system Information indicating the area, to generate the composite image further synthesized on the captured image.

  ADVANTAGE OF THE INVENTION According to this invention, there exists an effect that a user can grasp | ascertain the image at the time of operating the movable part of an object more clearly.

FIG. 1 is a diagram illustrating an example of the configuration of the information processing system according to the first embodiment. FIG. 2A is a diagram illustrating a first example of the outline of the virtual object display method according to the first embodiment. FIG. 2B is a diagram illustrating a second example of the outline of the virtual object display method according to the first embodiment. FIG. 3A is a diagram schematically illustrating a virtual object according to the first embodiment. FIG. 3B is a diagram schematically illustrating a virtual object in which the movable part according to the first embodiment is operated. FIG. 4 is a diagram illustrating an example of a configuration of the information processing apparatus according to the first embodiment. FIG. 5 is a diagram illustrating an example (in the case of a printer) of the initial display image information according to the first embodiment. FIG. 6 is a diagram illustrating an example of removable partial image information (in the case of a bank) according to the first embodiment. FIG. 7 is a diagram illustrating an example of removable image information (in the case of a finisher) according to the first embodiment. FIG. 8 is a diagram illustrating an example (in the case of the ADF) of the movable part image information according to the first embodiment. FIG. 9 is a diagram illustrating an example (in the case of a toner cover) of the movable part image information according to the first embodiment. FIG. 10 is a diagram illustrating an example (in the case of a paper tray) of movable part image information according to the first embodiment. FIG. 11 is a diagram illustrating an example of movable part image information (in the case of a double-sided unit cover) according to the first embodiment. FIG. 12 is a diagram illustrating an example of the device operation image information (in the case of a print operation) according to the first embodiment. FIG. 13 is a diagram illustrating an example of a model selection screen according to the first embodiment. FIG. 14 is a diagram illustrating an example of a region selection screen according to the first embodiment. FIG. 15 is a diagram illustrating an example of the model selection screen (when a list is displayed) according to the first embodiment. FIG. 16 is a diagram illustrating an example (in the case of the printer A) of the model display screen of the first embodiment. FIG. 17A is a diagram illustrating an example of an animation illustrating an operation of opening the toner cover according to the first embodiment. FIG. 17B is a diagram illustrating an example of an animation illustrating an operation of opening the toner cover according to the first embodiment. FIG. 17C is a diagram illustrating an example of an animation illustrating an operation of opening the toner cover according to the first embodiment. FIG. 18A is a diagram illustrating an example of an animation illustrating an operation of closing the toner cover according to the first embodiment. FIG. 18B is a diagram illustrating an example of an animation illustrating an operation of closing the toner cover according to the first embodiment. FIG. 18C is a diagram illustrating an example of an animation illustrating an operation of closing the toner cover according to the first embodiment. FIG. 19 is a diagram illustrating an example of the model display screen of the first embodiment after the ADF operation button is pressed. FIG. 20A is a diagram illustrating an example of a bank selection screen according to the first embodiment. FIG. 20B is a diagram showing an example of the model display screen of the first embodiment after a bank is selected. FIG. 21 is a diagram illustrating an example of the installation restricted area designation screen according to the first embodiment. FIG. 22 is a diagram illustrating an example of the product specification screen according to the first embodiment. FIG. 23 is a diagram illustrating an example of a map display screen according to the first embodiment. FIG. 24 is a flowchart illustrating an example of the information processing method according to the first embodiment. FIG. 25 is a diagram illustrating an example of a hardware configuration of the information processing apparatus according to the first embodiment. FIG. 26 is a diagram illustrating an example (in the case of a refrigerator) of the initial display image information according to the second embodiment. FIG. 27 is a diagram illustrating an example (in the case of a door) of movable part image information according to the second embodiment. FIG. 28 is a diagram illustrating an example of a model selection screen according to the second embodiment. FIG. 29 is a diagram illustrating an example of the model selection screen (when a list is displayed) according to the second embodiment. FIG. 30 is a diagram illustrating an example of the model display screen (in the case of the refrigerator Ra) according to the second embodiment. FIG. 31 is a diagram illustrating an example of the model display screen of the second embodiment after the first door operation button is pressed. FIG. 32 is a diagram illustrating an example (in the case of an automobile) of the initial display image information according to the third embodiment. FIG. 33 is a diagram illustrating an example of movable part image information (in the case of an opening / closing unit of an automobile) of the third embodiment. FIG. 34 is a diagram illustrating an example (in the case of a car Ca) of the virtual object according to the third embodiment. FIG. 35 is a diagram illustrating an example of the virtual object according to the third embodiment (when the driver's seat door is open).

  Embodiments of an information processing apparatus, an information processing method, and a program will be described below in detail with reference to the accompanying drawings.

(1st Embodiment)
FIG. 1 is a diagram illustrating an example of a configuration of an information processing system 1 according to the first embodiment. The information processing system 1 according to the embodiment includes an information processing device 10 and a server device 20. The information processing device 10 and the server device 20 are connected via a network 100. The communication method of the network 100 may be a wireless method or a wired method, or a combination of the wireless method and the wired method. The network 100 is, for example, the Internet.

  The information processing apparatus 10 displays a composite image in which a virtual object is composited on a captured image obtained by capturing an arbitrary place. A virtual object is a 3D model of an arbitrary object. In the description of the first embodiment, a case where an object corresponding to a virtual object is an office device will be described. In the description of the first embodiment, a case where the printer is a printer (printing apparatus) will be described as an example. The information processing apparatus 10 is, for example, a smart device with a camera.

  The server device 20 stores information such as an application (program) for displaying a composite image including a virtual object in the information processing device 10, image information indicating an AR marker, and display setting information for displaying a virtual object. I do. The AR marker is position identification information for identifying a position where the virtual object is synthesized on the captured image. Details of the display setting information will be described later with reference to FIGS. The information stored in the server device 20 is downloaded to the information processing device 10 via a website launched on the server device 20, for example.

  FIG. 2A is a diagram illustrating a first example of the outline of the method for displaying the virtual object 40 according to the first embodiment. First, the user of the information processing device 10 prints the image information indicating the AR marker 30 stored in the server device 20 or the information processing device 10, and places the printed matter indicating the AR marker 30 in a place where the virtual object 40 is displayed. . Next, the information processing device 10 downloads, from the server device 20, display setting information of a model to be displayed as the virtual object 40. Next, when the information processing apparatus 10 captures a captured image including a printed material indicating the AR marker 30, a virtual object 40 corresponding to the display setting information is displayed at the position of the printed material 30 included in the captured image.

  In FIG. 1 and FIG. 2A, the server device 20 has one configuration, but the configuration for the information processing device 10 to acquire the information stored in the server device 20 is not limited to this. .

  FIG. 2B is a diagram illustrating a second example of the outline of the method for displaying the virtual object 40 according to the first embodiment. The example of FIG. 2B shows a case where the server devices 20 are distributed on the network 100 as the server device 20a and the server device 20b. First, the application developer prepares an application having display setting information including information on image data of the virtual object 40 (model data of an image displayed as a 3D model, etc.) in advance on the server device 20a (the developer company). Server, etc.). Next, the application is released from the server device 20a to the server device 20b that operates an application sales site or the like on the network 100. Next, the information processing device 10 downloads the application from the server device 20b. Thus, even in the example of the configuration in FIG. 2B, an environment in which the information processing apparatus 10 can install the application can be configured. The information processing device 10 may download the AR marker 30 from the server device 20a as needed.

  Next, an outline of the virtual object 40 displayed on the information processing apparatus 10 of the first embodiment will be described.

  FIG. 3A is a diagram schematically illustrating the virtual object 40 according to the first embodiment. The virtual object 40 according to the first embodiment has a first area 41 representing an immobile part and a second area 42 representing a movable part. For simplicity, in the example of the virtual object 40 in FIG. 3, the number of the second region 42 representing the movable portion is one, but the number of the second regions 42 representing the movable portion may be arbitrary. The virtual object 40 in FIG. 3A is, for example, a printer, and the second area 42 is, for example, an ADF (Auto Document Feeder) portion of the printer.

  FIG. 3B is a diagram schematically illustrating the virtual object 40 of the first embodiment (when the movable part is operated). The example of FIG. 3B shows a case where the second area 42 of the virtual object 40 can be opened and closed. The virtual object 40 is first displayed on the information processing device 10 in the initial display state of FIG. 3A. Then, when the information processing apparatus 10 receives an input for operating the second area 42, the display of the virtual object 40 is changed from the state of FIG. 3A to the state of FIG. 3B. At this time, the operation of opening and closing the second area 42 is displayed as a moving image. The moving image is, for example, an animation. The display reference point P in FIG. 3A and FIG. 3B indicates a reference point that specifies a position where a moving image is displayed. The coordinates of the display reference point P are represented, for example, by a coordinate system having a predetermined point O of the virtual object 40 as an origin.

  Next, an example of the configuration of the information processing apparatus 10 according to the first embodiment will be described.

  FIG. 4 is a diagram illustrating an example of a configuration of the information processing apparatus 10 according to the first embodiment. The information processing apparatus 10 according to the first embodiment includes a storage unit 11, an imaging unit 12, a display control unit 13, a display unit 14, an operation reception unit 15, a generation unit 16, a communication unit 17, and a GPS unit 18. Although the information processing device 10 may be an arbitrary device, the description of the first embodiment will exemplify a case where the information processing device 10 is a smart device such as a smartphone and a tablet terminal. That is, the display unit 14 and the operation receiving unit 15 are liquid crystal touch panels or the like.

  The storage unit 11 stores display setting information for displaying the virtual object 40 for each type of the virtual object 40. The display setting information includes, for example, initial display image information, detachable partial image information, movable partial image information, and device operation image information. In addition, the storage unit 11 obtains the printer specification information corresponding to the virtual object 40, the address of a website from which the printer corresponding to the virtual object 40 can be obtained (purchased), and the printer corresponding to the virtual object 40 (purchase). Information such as addresses of possible places (stores) is also stored. The information in the storage unit 11 is updated by the communication unit 17 communicating with the server device 20. For example, the communication unit 17 newly acquires the display setting information of the virtual object 40 from the server device 20 and stores the display setting information in the storage unit 11 so that the new virtual object 40 such as a new product can be stored in the information processing device 10. Can be displayed.

  Here, the display setting information (initial display image information, detachable partial image information, movable partial image information, and device operation image information) will be described taking the case where the virtual object 40 is a printer as an example.

  First, an example of the initial display image information of the first embodiment will be described.

  FIG. 5 is a diagram illustrating an example (in the case of a printer) of the initial display image information according to the first embodiment. The initial display image information indicates a state when display of the virtual object 40 is started. The initial display image information includes items of a model name, a main body, a bank, and a finisher. The model name indicates the name of the model of the printer. In the first embodiment, the model name is identification information for identifying the type of the virtual object 40. The main body indicates specific information for specifying a main body image in a state where a detachable portion (optional component) is not mounted. The specific information is, for example, an image file name. The main body image is displayed at the position of the AR marker 30 included in the image captured by the information processing device 10.

  The bank is an optional component that stores paper used for printing. In the bank, numerical values such as “−”, “none”, and “1”, “2”,... Are set. "-" Indicates that the bank cannot be mounted on the printer body. “No” indicates that no bank is mounted on a model to which the bank can be mounted. “1” indicates that the bank is mounted one stage on a model to which the bank can be mounted. "2" indicates that the bank is mounted in two stages on a model to which the bank can be mounted. The image indicating the bank is specified from detachable partial image information described later.

  The finisher is an optional component for performing post-printing processing such as punching (piercing) and stapling of printed matter. In the finisher, “−”, “absent”, and “present” are set. "-" Indicates that the finisher cannot be attached to the printer body. “No” indicates that the finisher is not attached to the model to which the finisher can be attached. “Yes” indicates that the finisher is attached to a model to which the finisher can be attached. The image indicating the finisher is specified from detachable partial image information described later.

  For example, in the example of FIG. 5, when the virtual object 40 corresponding to the printer B is initially displayed on the information processing device 10, the virtual object 40 to which the bank and the finisher are not attached is displayed by the main body image B.

  Next, an example of the removable partial image information according to the first embodiment will be described.

  FIG. 6 is a diagram illustrating an example of removable partial image information (in the case of a bank) according to the first embodiment. The detachable partial image information in FIG. 6 includes items of a model name, a bank, a display reference point, and a maximum number of bank stages. The model name indicates the name of the model of the printer. The bank indicates specific information for specifying an image indicating the bank. The specific information is, for example, an image file name. If "-" is set for the bank, it indicates that the bank cannot be mounted on the printer body. The display reference point indicates a position at which an image indicating a bank is displayed. For example, the information processing apparatus 10 displays an image indicating a bank as a virtual detachable object such that an end point of the image indicating the bank is aligned with the coordinates of the display reference point. The maximum number of banks indicates the number of banks that can be mounted on the printer body. When there are a plurality of types of banks that can be mounted on the same printer main body, the storage unit 11 stores detachable partial image information for each type of bank.

  For example, in the example of the printer B in FIG. 6, the image indicating the bank of the printer B is a Bank image B. The position at which the Bank image B is displayed is specified by the coordinates (x_b6, y_b6, z_b6) of the display reference point. The number of banks that can be mounted on the printer B is three.

  FIG. 7 is a diagram illustrating an example of removable image information (in the case of a finisher) according to the first embodiment. The removable partial image information in FIG. 7 includes items of a model name, a finisher, and a display reference point. The model name indicates the name of the model of the printer. The finisher indicates identification information for identifying an image indicating the finisher. The specific information is, for example, an image file name. If the finisher is set to "-", it indicates that the finisher cannot be attached to the printer body. The display reference point indicates a position at which an image indicating the finisher is displayed. For example, the information processing apparatus 10 displays an image indicating the finisher as a virtual detachable object such that an end point of the image indicating the finisher is aligned with the coordinates of the display reference point. When there are a plurality of types of finishers that can be mounted on the same printer main body, the storage unit 11 stores detachable partial image information for each type of finisher.

  For example, in the example of the printer B in FIG. 7, the image indicating the finisher of the printer B is the Finisher image B. The position where the Finisher image B is displayed is specified by the coordinates (x_b7, y_b7, z_b7) of the display reference point.

  Next, an example of the movable part image information according to the first embodiment will be described.

  FIG. 8 is a diagram illustrating an example (in the case of the ADF) of the movable part image information according to the first embodiment. The movable part image information in FIG. 8 includes items of a model name, ADF (open), ADF (close), and a display reference point. The model name indicates the name of the model of the printer. ADF (open) indicates identification information for identifying an image (animation) indicating an operation of opening the ADF. ADF (close) indicates specific information for specifying an image (animation) indicating an operation of closing the ADF. The specific information is, for example, an image file name. If "-" is set in ADF (open) and ADF (close), it indicates that there is no ADF in the printer body. The display reference point indicates a position at which an image (animation) indicating the operation of opening the ADF and an image (animation) indicating the operation of closing the ADF are displayed.

  For example, in the example of the printer A in FIG. 8, an image (animation) indicating an operation of opening the ADF of the printer A is an ADF_open image A. An image (animation) showing the operation of closing the ADF of the printer A is an ADF_close image A. The positions where the ADF_open image A and the ADF_close image A are displayed are specified by the coordinates (x_a1, y_a1, z_a1) of the display reference point.

  FIG. 9 is a diagram illustrating an example (in the case of a toner cover) of the movable part image information according to the first embodiment. The movable part image information in FIG. 9 includes items of a model name, a toner cover (open), a toner cover (close), and a display reference point. The model name indicates the name of the model of the printer. The toner cover (open) indicates specific information for specifying an image (animation) indicating an operation of opening the toner cover. The toner cover (close) indicates specific information for specifying an image (animation) indicating an operation of closing the toner cover. The specific information is, for example, an image file name. When "-" is set in the toner cover (open) and the toner cover (close), it indicates that the printer main body does not have the toner cover. The display reference point indicates a position where an image (animation) indicating an operation of opening the toner cover and an image (animation) indicating an operation of closing the toner cover are displayed.

  For example, in the example of the printer A in FIG. 9, an image (animation) indicating an operation of opening the toner cover of the printer A is a Toner_open image A. An image (animation) showing the operation of closing the toner cover of the printer A is a Toner_close image A. The positions where the Toner_open image A and the Toner_close image A are displayed are specified by the coordinates (x_a2, y_a2, z_a2) of the display reference point.

  FIG. 10 is a diagram illustrating an example (in the case of a paper tray) of movable part image information according to the first embodiment. The movable part image information in FIG. 10 includes items of a model name, a paper tray (open), a paper tray (close), and a display reference point. The model name indicates the name of the model of the printer. The paper tray (open) indicates specific information for specifying an image (animation) indicating an operation of opening the paper tray. The paper tray (close) indicates specific information for specifying an image (animation) indicating an operation of closing the paper tray. The specific information is, for example, an image file name. When "-" is set for the paper tray (open) and the paper tray (close), it indicates that there is no paper tray in the printer body. The display reference point indicates a position at which an image (animation) indicating the operation of opening the paper tray and an image (animation) indicating the operation of closing the paper tray are displayed.

  For example, in the example of the printer A in FIG. 10, an image (animation) indicating an operation of opening the paper tray of the printer A is a Tray_open image A. An image (animation) showing the operation of closing the paper tray of the printer A is a Trace_close image A. The positions at which the Tray_open image A and the Tray_close image A are displayed are specified by the coordinates (x_a3, y_a3, z_a3) of the display reference point.

  FIG. 11 is a diagram illustrating an example of movable part image information (in the case of a double-sided unit cover) according to the first embodiment. The movable part image information in FIG. 11 includes items of a model name, a duplex unit cover (open), a duplex unit cover (close), and a display reference point. The model name indicates the name of the model of the printer. The duplex unit cover (open) indicates identification information for identifying an image (animation) indicating an operation of opening the duplex unit cover. The double-sided unit cover (close) indicates specific information for specifying an image (animation) indicating an operation of closing the double-sided unit cover. The specific information is, for example, an image file name. When “−” is set in the duplex unit cover (open) and the duplex unit cover (close), it indicates that the duplexer unit cover is not provided in the printer body. The display reference point indicates a position at which an image (animation) indicating the operation of opening the duplex unit cover and an image (animation) indicating the operation of closing the duplex unit cover are displayed.

  For example, in the example of the printer A in FIG. 11, an image (animation) showing an operation of opening the duplex unit cover of the printer A is a Cover_open image A. An image (animation) showing the operation of closing the duplex unit cover of the printer A is a Cover_close image A. The positions where the Cover_open image A and the Cover_close image A are displayed are specified by the coordinates (x_a4, y_a4, z_a4) of the display reference point.

  FIG. 12 is a diagram illustrating an example of the device operation image information (in the case of a print operation) according to the first embodiment. The device operation image information in FIG. 12 includes items of a model name, a print operation, and a display reference point. The model name indicates the name of the model of the printer. The print operation indicates specific information for specifying an image (animation) indicating a state where a printed matter is discharged from the printer when printing is performed. The specific information is, for example, an image file name. If "-" is set for the print operation, it indicates that there is no image (animation) indicating that the printed matter is discharged from the printer when printing is performed. The display reference point indicates a position where an image (animation) indicating a state in which a printed matter is discharged from the printer when printing is performed is displayed.

  For example, in the example of the printer A in FIG. 12, an image (animation) showing a state where a printed material is discharged from the printer A when printing is performed is a print image A. The position at which the print image A is displayed is specified by the coordinates (x_a5, y_a5, z_a5) of the display reference point.

  Referring back to FIG. 4, when the imaging unit 12 captures a captured image including the AR marker 30 indicating the position where the virtual object 40 is displayed, the imaging unit 12 inputs the captured image to the display control unit 13. Note that the imaging unit 12 continuously captures the captured image. That is, the captured image is input to the display control unit 13 as a moving image.

  When receiving the captured image from the imaging unit 12, the display control unit 13 displays a model selection screen on which the captured image is used as a background of the screen on the display unit 14.

  FIG. 13 is a diagram illustrating an example of a model selection screen according to the first embodiment. The model selection screen according to the first embodiment includes a model selection button 101, a setting button 102, a help button 103, and a camera button 104.

  The model selection button 101 is a drop-down list for selecting a printer model. When the model selection button 101 of the operation receiving unit 15 is pressed, the display control unit 13 displays a list of printer model names stored in the storage unit 11 as the above-described display setting information on the display unit 14.

  The setting button 102 is a button for displaying a setting screen for changing display settings such as a display language (Japanese, English, Chinese, and the like) and a character size. It should be noted that the setting screen may be an area selection screen for applying a predetermined setting by selecting an area.

  FIG. 14 is a diagram illustrating an example of a region selection screen according to the first embodiment. The example of FIG. 14 shows a case where a region is selected from a plurality of regions (Japan, China, Europe, the Middle East, Africa, Asia, the Americas, and Latin America). The display control unit 13 makes the luminance of the image area indicating the area selected by the user brighter than the luminance of the image area indicating the area not selected by the user.

  The display control unit 13 performs display control based on area setting information associated with the selected area. The region setting information is information indicating a display method according to the region. The region setting information includes, for example, display language, character size, URL information, model name, purchaseable model information, and the like. The URL information includes, for example, a URL of a website displayed when a product purchase button 112 described below is pressed. Purchasable model information indicates models that can be purchased in the region. For example, a recommended product display button 111 described later displays a model whose product specification is similar to a model displayed on a model name display unit 105 described later among models available for purchase in the region based on the available model information. Is done.

  Returning to FIG. 13, the help button 103 is a button for displaying a help screen showing an operation method.

  The camera button 104 is a button for capturing a captured image displayed as a background. When the camera button 104 of the operation receiving unit 15 is pressed while the virtual object 40 is displayed, a composite image including the virtual object 40 is stored in the storage unit 11 or the like.

  FIG. 15 is a diagram illustrating an example of the model selection screen (when a list is displayed) according to the first embodiment. The example of FIG. 15 shows a case where the printer A, the printer B, and the printer C are displayed as a list when the model selection button 101 is pressed. When the operation receiving unit 15 receives an operation of selecting a model name from the list, the generating unit 16 combines the image indicating the virtual object 40 associated with the model name at the position of the AR marker 30 on the captured image. A composite image is generated. Then, the display control unit 13 displays on the display unit 14 a screen in which UI components (various buttons, see FIG. 16) described later are added to the composite image. For example, when the printer A is selected, the virtual object 40 indicated by the main image A of the initial display image information (see FIG. 5) included in the display setting information is displayed at the position of the AR marker 30.

  At this time, the display control unit 13 and the generation unit 16 change the appearance of the virtual object 40 according to the positional relationship between the imaging unit 12 of the information processing device 10 and the AR marker 30 by using the AR technology. . Specifically, first, the display control unit 13 detects the AR marker 30 included in the captured image, and determines whether the appearance (at least one of the position and the shape on the captured image) of the AR marker 30 has been changed. Is determined. When the appearance of the AR marker 30 is changed, the generation unit 16 changes the display mode of the virtual object 40 (enlargement, reduction, movement, rotation, and the like) according to the degree of change in the appearance, The virtual object 40 generates a combined image combined with the captured image. Then, the display control unit 13 displays on the display unit 14 a screen in which UI components (various buttons, see FIG. 16) described later are added to the composite image. This allows the user of the information processing apparatus 10 to refer to the virtual object 40 stereoscopically displayed on the display unit 14 and turn the imaging unit 12 in a direction in which the user wants to refer to the virtual object 40 stereoscopically displayed. The virtual object 40 can be referred to from various directions.

  FIG. 16 is a diagram illustrating an example (in the case of the printer A) of the model display screen of the first embodiment. The model display screen of the first embodiment includes a model selection button 101, a setting button 102, a help button 103, a camera button 104, a model name display unit 105, an operation button 106, a bank selection button 107, a finisher selection button 108, an installation restricted area. It has a designation button 109, a product specification display button 110, a recommended product display button 111, a product purchase button 112, a map display button 113, and a share button 114.

  Descriptions of the model selection button 101, the setting button 102, the help button 103, and the camera button 104 are the same as those in FIG.

  The model name display section 105 displays the model name selected by the model selection button 101. Also, functions available for the model are displayed. Specifically, the display control unit 13 obtains specification information specified by the model name displayed on the model name display unit 105 from the storage unit 11 so as to display functions available for the model. 14 is displayed. The example of FIG. 16 illustrates a case where the printer A can use copy, print, scan, and FAX.

  The operation buttons 106 include an ADF operation button 106a, a toner cover operation button 106b, a paper tray operation button 106c, a duplex unit operation button 106d, and a print operation display button 106e.

  The ADF operation button 106a is a button for opening and closing the ADF. The ADF operation button 106a displays the operation indicated by the image file specified by the movable part image information (in the case of the ADF) in FIG. 8 included in the display setting information. When no image file is specified in the movable part image information (in the case of the ADF), the display control unit 13 does not display the ADF operation button 106a or displays the ADF operation button 106a with half brightness (semi-transparency) or the like.

  When the ADF operation button 106a is pressed, an image (animation) indicating an operation of opening the ADF is displayed. Specifically, when the virtual object 40 is the printer A, an image (animation) indicating an operation of opening the ADF is displayed by the ADF_open image A of the movable part image information (see FIG. 8) included in the display setting information. .

  FIG. 19 is a diagram illustrating an example of the model display screen of the first embodiment after the ADF operation button 106a is pressed. When the ADF operation button 106a is pressed again in this state, an image (animation) indicating an operation of closing the ADF is displayed. Specifically, when the virtual object 40 is the printer A, an image (animation) indicating the operation of closing the ADF is displayed by the ADF_close image A of the movable part image information (see FIG. 8) included in the display setting information, The display of the printer A returns to the state shown in FIG.

  Returning to FIG. 16, the toner cover operation button 106b is a button for opening and closing the toner cover. The toner cover operation button 106b displays the operation indicated by the image file specified by the movable part image information (in the case of the toner cover) in FIG. 9 included in the display setting information. When no image file is designated in the movable part image information (in the case of the toner cover), the display control unit 13 does not display the toner cover operation button 106b, or displays the image with semi-brightness (semi-transparency) or the like. .

  When the toner cover operation button 106b is pressed, an image (animation) indicating an operation of opening the toner cover is displayed. Specifically, when the virtual object 40 is the printer A, an image (animation) indicating the operation of opening the toner cover is displayed by the Toner_open image A of the movable part image information (see FIG. 9) included in the display setting information. You.

  17A to 17C are diagrams illustrating an example of an animation showing an operation of opening the toner cover according to the first embodiment. FIG. 17A shows a state before the toner cover operation button 106b is pressed. FIG. 17B shows a state in the middle of the animation showing the operation of opening the toner cover. FIG. 17C shows a state at the end of the animation showing the operation of opening the toner cover.

  When the toner cover operation button 106b is pressed, the display control unit 13 starts displaying an animation indicating an operation of opening the toner cover. The animation showing the operation of opening the toner cover includes an animation showing a state when removing the toner cartridge. The user can more clearly understand how to remove the toner cartridge by watching the animation indicating the operation of opening the toner cover.

  Referring back to FIG. 16, when the toner cover operation button 106b is pressed again, an image (animation) indicating an operation of closing the toner cover is displayed. Specifically, when the virtual object 40 is the printer A, an image (animation) indicating the operation of closing the toner cover is displayed by the Toner_close image A of the movable part image information (see FIG. 9) included in the display setting information. You.

  18A to 18C are diagrams illustrating an example of an animation showing an operation of closing the toner cover according to the first embodiment. FIG. 18A shows a state where the toner cover operation button 106b is pressed again while the toner cover is open. FIG. 18B shows a state in the middle of the animation showing the operation of closing the toner cover. FIG. 18C shows a state at the end of the animation showing the operation of closing the toner cover.

  When the toner cover operation button 106b is pressed again while the toner cover is open, the display control unit 13 starts displaying an animation indicating an operation of closing the toner cover. The animation showing the operation of closing the toner cover includes an animation showing a state when the toner cartridge is attached. The animation showing how the toner cartridge is attached shows, for example, the direction in which the toner cartridge is rotated when the toner cartridge is attached. The user can more clearly understand how to attach the toner cartridge by watching the animation indicating the operation of closing the toner cover.

  Note that the display control unit 13 may also display an animation showing how to take out and attach components inside the opening / closing section for the opening / closing section other than the toner cover. Thus, the user can more clearly understand the procedure for replacing parts of the object corresponding to the virtual object 40 by watching the animation.

  The paper tray operation button 106c is a button for opening and closing the paper tray. The paper tray operation button 106c displays an operation indicated by an image file specified by the movable part image information (in the case of a paper tray) in FIG. 10 included in the display setting information. When no image file is designated in the movable part image information (in the case of a paper tray), the display control unit 13 does not display the paper tray operation button 106c, or displays the paper tray with the half brightness (semi-transparent) or the like. .

  When the paper tray operation button 106c is pressed, an image (animation) indicating an operation of opening the paper tray is displayed. Specifically, when the virtual object 40 is the printer A, an image (animation) indicating the operation of opening the paper tray is displayed by the Tray_open image A of the movable part image information (see FIG. 10) included in the display setting information. You. When the paper tray operation button 106c is pressed again, an image (animation) indicating an operation of closing the paper tray is displayed. Specifically, when the virtual object 40 is the printer A, an image (animation) indicating the operation of closing the paper tray is displayed by the Tray_close image A of the movable part image information (see FIG. 10) included in the display setting information. You.

  The duplex unit operation button 106d is a button for opening and closing the duplex unit. The duplex unit operation button 106d is a button for opening and closing the duplex unit. The duplex unit operation button 106d displays an operation indicated by an image file specified by the movable part image information (in the case of a duplex unit) in FIG. 11 included in the display setting information. When no image file is specified in the movable part image information (in the case of a double-sided unit), the display control unit 13 does not display the double-sided unit operation button 106d or displays the image with half brightness (semi-transparent) or the like. .

  When the duplex unit operation button 106d is pressed, an image (animation) indicating an operation of opening the duplex unit is displayed. Specifically, when the virtual object 40 is the printer A, an image (animation) indicating the operation of opening the duplex unit is displayed by the Cover_open image A of the movable part image information (see FIG. 11) included in the display setting information. You. When the duplex unit operation button 106d is pressed again, an image (animation) indicating an operation of closing the duplex unit is displayed. Specifically, when the virtual object 40 is the printer A, an image (animation) indicating the operation of closing the duplex unit is displayed by the Cover_close image A of the movable part image information (see FIG. 11) included in the display setting information. You.

  The print operation display button 106e is a button that displays, as an animation, how the printed matter is discharged from the printer when printing is performed. The print operation display button 106e displays the operation indicated by the image file specified by the device operation image information (in the case of the print operation) in FIG. 12 included in the display setting information. When no image file is specified in the device operation image information (in the case of a print operation), the display control unit 13 does not display the print operation display button 106e or displays the print operation display button 106e with half brightness (semi-transparency) or the like. .

  When the print operation display button 106e is pressed, the display control unit 13 displays an animation corresponding to the print speed of the printer, so that the print speed of the printer can be conveyed to the user sensuously. For example, when the printing speed of the printer is 30 ppm (page per minute), the display control unit 13 displays an animation indicating that one printed material is discharged from the printer every two seconds. Further, for example, when the printing speed of the printer is 20 ppm (page per minute), the display control unit 13 displays an animation indicating that one printed material is discharged from the printer every three seconds.

  The bank selection button 107 is a button for displaying a bank selection screen. The bank selection screen is a screen for selecting a bank that can be mounted on the model displayed on the model name display unit 105.

  FIG. 20A is a diagram illustrating an example of a bank selection screen according to the first embodiment. The display control unit 13 displays on the display unit 14 a bank selection screen based on the detachable partial image information (see FIG. 6) included in the display setting information of the model displayed on the model name display unit 105. Specifically, when the model displayed on the model name display unit 105 has a plurality of types of banks, a screen for selecting the type of bank is displayed by displaying a plurality of images indicating the banks. If the model displayed on the model name display unit 105 can form multiple banks, a screen for selecting the number of banks up to the maximum number of banks shown in the removable partial image information of FIG. 6 is displayed.

  In the case of the detachable partial image information (see FIG. 6) included in the display setting information of the printer A, since there is one type of bank and the maximum number of bank stages is 1, in FIG. Is displayed.

  FIG. 20B is a diagram showing an example of the model display screen of the first embodiment after a bank is selected. When the operation receiving unit 15 receives an operation indicating the selection of a bank on the bank selection screen of FIG. 20A, as shown in FIG. 20B, the virtual object 40 indicating the main body of the printer A is replaced with the virtual detachable object 43 indicating the bank. Is added. Specifically, the generation unit 16 further synthesizes the Bank image A with the synthesized image obtained by synthesizing the captured image with the main image A, and the display control unit 13 displays the synthesized image on the display unit 14. This allows the user to more clearly understand the image when the detachable portion (optional component) is mounted on the main body of the printer A.

  Returning to FIG. 16, the finisher selection button 108 is a button for displaying a finisher selection screen. The description of the finisher selection screen is the same as the description of the bank selection screen, and a description thereof will be omitted.

  The installation restriction area designation button 109 is a button for displaying an installation restriction area designation screen. The installation restriction area designation screen is a screen for designating an installation restriction area indicating a restriction on an area where an object corresponding to the virtual object 40 is installed.

  FIG. 21 is a diagram illustrating an example of the installation restricted area designation screen according to the first embodiment. The installation restriction area designation screen of the first embodiment further includes a frame 120, a width designation button 121, a depth designation button 122, a height designation button 123, and a frame move button 124 in addition to the display of the screen of FIG.

  The frame 120 indicates the size of the installation restriction area. The width, depth, height, and display position of the frame 120 are changed according to the operation of the width designation button 121, the depth designation button 122, the height designation button 123, and the frame movement button 124.

  Specifically, when “+” of the width designation button 121 is pressed, the width of the frame 120 increases, and when “−” of the width designation button 121 is pressed, the width of the frame 120 decreases. Similarly, when “+” of the depth designation button 122 is pressed, the depth of the frame 120 increases, and when “−” of the depth designation button 122 is pressed, the depth of the frame 120 decreases. Similarly, when the “+” of the height designation button 123 is pressed, the height of the frame 120 increases, and when the “−” of the height designation button 123 is pressed, the height of the frame 120 decreases. .

  The user interface for designating the width, depth, and height of the frame 120 may be arbitrary. For example, the display control unit 13 may display a numeric input unit having a numeric keypad on the display unit 14, and the operation receiving unit 15 may receive an input of a number indicating a width, a depth, and a height.

  When the frame move button 124 is pressed, the frame 120 moves up, down, left, and right on the background image (captured image) of the installation restricted area designation screen according to the pressed direction.

  When the virtual object 40 does not fit in the installation restricted area indicated by the frame 120, the display control unit 13 changes the display mode of the frame 120 indicating the installation restricted area. The method of changing the display mode may be arbitrary. The method of changing the display mode includes, for example, changing the color of the line indicating the frame 120, changing the thickness of the line indicating the frame 120, and changing the line indicating the frame 120 to blink.

  For example, in the case of the example of FIG. 21, the display control unit 13 displays the frame 120 in yellow because the virtual object 40 indicating the printer A with the ADF closed is included in the frame 120. On the other hand, the display control unit 13 displays the frame 120 in red because the virtual object 40 indicating the printer A in which the ADF is open does not fit in the frame 120. Thus, when the user operates the movable portion of the printer A at an arbitrary installation location indicated by the frame 120, the user can more clearly understand the image of whether or not the printer A fits in the installation location.

  Returning to FIG. 16, the product specification display button 110 is a button for displaying a product specification screen for displaying specification information specified by the model name displayed on the model name display unit 105.

  FIG. 22 is a diagram illustrating an example of the product specification screen according to the first embodiment. When the operation receiving unit 15 receives an operation indicating that the product specification display button 110 is pressed, the display control unit 13 acquires the specification information of the model displayed on the model name display unit 105 from the storage unit 11, and The product specification screen including the information is displayed on the display unit 14. The specification information includes, for example, a maximum print size, a printer size, a corresponding OS, a printing speed, a resolution, a corresponding interface, and the like.

  Further, the display control unit 13 displays a detailed specification link for displaying a web page for displaying detailed specifications on the product specification screen. When the operation receiving unit 15 receives an operation indicating pressing of the detailed specification link, the communication unit 17 displays the website acquired by accessing the address indicated by the detailed specification link on the display unit 14 by the display control unit 13. indicate.

  Returning to FIG. 16, the recommended product display button 111 is a button for displaying a model display screen of a model recommended to the user. The model recommended to the user is, for example, a model whose product specifications are similar to the model displayed on the model name display unit 105. When the operation receiving unit 15 receives an operation indicating that the recommended product display button 111 is pressed, the generation unit 16 sets the main image of the initial display image information included in the display setting information of the model recommended to the user on the captured image. A synthesized image synthesized at the position of the AR marker 30 is generated. Then, the display control unit 13 displays a model display screen corresponding to the composite image on the display unit 14.

  The product purchase button 112 is a button for displaying a website where the model displayed on the model name display unit 105 can be purchased. When the operation receiving unit 15 receives an operation indicating that the product purchase button 112 is pressed, the communication unit 17 acquires from the storage unit 11 the address of a website where the model displayed on the model name display unit 105 can be purchased. . Then, the display control unit 13 displays on the display unit 14 the website acquired by the communication unit 17 accessing the address.

  The map display button 113 is a button for displaying a map display screen indicating the location of a store where the model displayed on the model name display unit 105 can be purchased.

  FIG. 23 is a diagram illustrating an example of a map display screen according to the first embodiment. When the operation receiving unit 15 receives an operation indicating that the map display button 113 has been pressed, the GPS unit 18 acquires the current position 200 of the information processing device 10. Then, when the display control unit 13 displays a map near the current position 200 on the display unit 14 and a store where the model displayed on the model name display unit 105 can be purchased exists near the current position 200, The location of the store is specified on the map display screen. The example of FIG. 23 shows a case where the positions of the store 201 and the store 202 are clearly indicated on the map display screen.

  Returning to FIG. 16, the share button 114 displays a pop-up list of share applications such as Facebook (registered trademark) and Twitter (registered trademark). When the operation receiving unit 15 receives an operation indicating selection of a share application, information and the like displayed on the display unit 14 using the share application can be shared with other users.

  Next, an example of the information processing method according to the first embodiment will be described.

  FIG. 24 is a flowchart illustrating an example of the information processing method according to the first embodiment. First, the imaging unit 12 starts capturing an image including the AR marker 30 indicating the position where the virtual object 40 is displayed (step S1).

  Next, when the display control unit 13 receives a captured image from the imaging unit 12, the display control unit 13 displays a model selection screen (see FIG. 13) in which the captured image is used as a background of the screen on the display unit 14 (step S2). Next, the operation receiving unit 15 receives an operation of selecting a model name from a drop-down list of the model selection button 101 (step S3).

  Next, the display control unit 13 displays a model display screen (see FIG. 16) of the model identified by the model name selected in step S3 on the display unit 14 (step S4). Specifically, first, the generation unit 16 generates a composite image in which an image indicating the virtual object 40 associated with the model name selected in step S3 is composited with the position of the AR marker 30 on the captured image. Then, the display control unit 13 displays on the display unit 14 a screen in which UI components such as various buttons are added to the composite image.

  Next, the display control unit 13 determines whether or not the positional relationship between the imaging unit 12 of the information processing device 10 and the AR marker 30 has been changed (Step S5). Specifically, the display control unit 13 detects the AR marker 30 included in the captured image, and determines whether the appearance (at least one of the position and the shape on the captured image) of the AR marker 30 has been changed. It is determined whether the positional relationship between the imaging unit 12 of the information processing device 10 and the AR marker 30 has been changed.

  When the positional relationship between the imaging unit 12 and the AR marker 30 has been changed (Step S5, Yes), the display control unit 13 uses the AR technology to change the virtual object 40 according to the changed positional relationship. Is changed (step S6). If the positional relationship between the imaging unit 12 and the AR marker 30 has not been changed (step S5, No), the process proceeds to step S7.

  Next, the operation receiving unit 15 determines whether a button on the model display screen (see FIG. 16) has been pressed (step S7). When the button is pressed (Step S7, Yes), the information processing device 10 performs a process corresponding to the various buttons described above (Step S8). If the button has not been pressed (step S7, No), the process proceeds to step S9.

  Next, the operation receiving unit 15 determines whether or not an operation for ending the processing related to the display of the virtual object 40 has been received (Step S9). If an operation to end has not been received (step S9, No), the process returns to step S5. If an operation to end is received (step S9, Yes), the process ends.

  Next, an example of a hardware configuration of the information processing apparatus 10 according to the first embodiment will be described.

  FIG. 25 is a diagram illustrating an example of a hardware configuration of the information processing apparatus 10 according to the first embodiment. The information processing apparatus 10 according to the first embodiment includes a control device 301, a main storage device 302, an auxiliary storage device 303, a display device 304, an input device 305, a communication device 306, an imaging device 307, and a GPS device 308. The control device 301, the main storage device 302, the auxiliary storage device 303, the display device 304, the input device 305, the communication device 306, the imaging device 307, and the GPS device 308 are connected to each other via a bus 310.

  The control device 301 executes a program read from the auxiliary storage device 303 to the main storage device 302. The main storage device 302 is a memory such as a ROM and a RAM. The auxiliary storage device 303 is a memory card or the like. The storage unit 11 in FIG. 4 corresponds to the main storage device 302 and the auxiliary storage device 303.

  The display device 304 and the input device 305 are liquid crystal touch panels having a display function and an input function. Communication device 306 communicates with other devices. The imaging device 307 captures a captured image indicating a state around the information processing device 10. The GPS device 308 obtains GPS information indicating the position of the information processing device 10 by receiving signals from a plurality of GPS satellites.

  The program executed by the information processing apparatus 10 is installed in a computer-readable storage medium such as a CD-ROM, a memory card, a CD-R, and a DVD (Digital Versatile Disk) in an installable or executable file. Stored and provided as a computer program product.

  Further, the program executed by the information processing apparatus 10 may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Further, the program executed by the information processing apparatus 10 may be provided via a network such as the Internet without being downloaded.

  Further, the program of the information processing apparatus 10 may be configured to be provided by being incorporated in a ROM or the like in advance.

  The program executed by the information processing apparatus 10 has a module configuration including, among the above-described functional blocks, functional blocks that can be realized by the program. The functional blocks that can be realized by the program are, for example, the display control unit 13, the operation reception unit 15, the generation unit 16, and the communication unit 17. The functional blocks realized by the program are loaded into the main storage device 302 as the actual hardware by the control device 301 reading the program from a storage medium such as the auxiliary storage device 303 and executing it. That is, the functional blocks realized by the program are generated on the main storage device 302.

  Note that some or all of the functional blocks that can be realized by a program may be realized by hardware such as an IC (Integrated Circuit) without being realized by a program.

  As described above, in the information processing apparatus 10 according to the first embodiment, the imaging unit 12 acquires a captured image including the AR marker 30 (position identification information) that identifies the position where the virtual object 40 is displayed. Further, the generation unit 16 generates a composite image in which the virtual object 40 is composited at a position identified by the AR marker 30 included in the captured image. Then, when an operation of the movable part of the virtual object 40 is received by the operation reception unit 15, the display of the movable part is changed according to the display setting information for displaying the virtual object 40. This allows the user to more clearly understand the image when the movable part of the object is operated.

(2nd Embodiment)
Next, a second embodiment will be described. In the description of the second embodiment, a case where the object corresponding to the virtual object 40 is a home appliance will be described. In the description of the second embodiment, a case of a refrigerator will be described as an example. In the description of the second embodiment, a description similar to that of the first embodiment will be omitted, and portions different from the first embodiment will be described.

  The configuration of the information processing system 1 of the second embodiment is the same as the configuration of the information processing system 1 of the first embodiment (see FIG. 1). The configuration of the information processing apparatus 10 of the second embodiment is the same as the configuration of the information processing apparatus 10 of the first embodiment (see FIG. 4).

  First, display setting information (initial display image information and movable portion image information) according to the second embodiment will be described.

  FIG. 26 is a diagram illustrating an example (in the case of a refrigerator) of the initial display image information according to the second embodiment. The initial display image information indicates a state when the display of the virtual object 40 indicating the refrigerator is started. The initial display image information includes items of a model name and a main body image. The model name indicates the name of the model of the refrigerator. In the second embodiment, the model name is identification information for identifying the type of the virtual object 40. The main body indicates specific information for specifying an image indicating the refrigerator. The specific information is, for example, an image file name. The main body image is displayed at the position of the AR marker 30 included in the image captured by the information processing device 10.

  For example, in the example of FIG. 26, when the virtual object 40 corresponding to the refrigerator Ra is initially displayed on the information processing device 10, the virtual object 40 is displayed by the main body image Ra.

  Next, an example of the movable part image information according to the second embodiment will be described.

  FIG. 27 is a diagram illustrating an example (in the case of a door) of movable part image information according to the second embodiment. The movable part image information in FIG. 27 includes items of a model name, a door number, a door (open), a door (close), and a display reference point. The model name indicates the name of the model of the refrigerator. The door number is a number for identifying the door of the refrigerator. The door (open) indicates specific information for specifying an image (animation) indicating an operation of opening the door. The door (close) indicates specific information for specifying an image (animation) indicating an operation of closing the door. The specific information is, for example, an image file name. The display reference point indicates a position at which an image (animation) indicating the operation of opening the door and an image (animation) indicating the operation of closing the door are displayed.

  For example, in the example of the refrigerator Ra in FIG. 27, the image (animation) showing the operation of opening the door of the refrigerator Ra having the door number 1 is the Door_open image Ra1. An image (animation) showing the operation of closing the door of the refrigerator Ra having the door number 1 is the Door_close image Ra1. The positions where the Door_open image Ra1 and the Door_close image Ra1 are displayed are specified by the coordinates (x_ra1, y_ra1, z_ra1) of the display reference point.

  Next, an example of a model selection screen according to the second embodiment will be described.

  FIG. 28 is a diagram illustrating an example of a model selection screen according to the second embodiment. The model selection screen according to the second embodiment has a model selection button 101, a setting button 102, a help button 103, and a camera button 104.

  The model selection button 101 is a drop-down list for selecting a model of the refrigerator. When the model selection button 101 of the operation receiving unit 15 is pressed, the display control unit 13 displays a list of the refrigerator model names stored in the storage unit 11 as the above-described display setting information on the display unit 14.

  The setting button 102 is a button for displaying a setting screen for changing display settings such as a display language (Japanese, English, Chinese, and the like) and a character size.

  The help button 103 is a button for displaying a help screen indicating an operation method.

  The camera button 104 is a button for capturing a captured image displayed as a background. When the camera button 104 of the operation receiving unit 15 is pressed while the virtual object 40 indicating the refrigerator is displayed at the position of the AR marker 30, the synthetic image including the virtual object 40 indicating the refrigerator is stored in the storage unit 11 or the like. Is stored.

  FIG. 29 is a diagram illustrating an example of the model selection screen (when a list is displayed) according to the second embodiment. The example of FIG. 29 illustrates a case where the refrigerator Ra, the refrigerator Rb, and the refrigerator Rc are displayed as a list when the model selection button 101 is pressed. When the operation receiving unit 15 receives an operation of selecting a model name from the list, the generating unit 16 combines the image indicating the virtual object 40 associated with the model name at the position of the AR marker 30 on the captured image. A composite image is generated. Then, the display control unit 13 displays, on the display unit 14, a screen in which UI components (various buttons and the like, see FIG. 30) described later are added to the composite image. For example, when the refrigerator Ra is selected, the virtual object 40 indicated by the main body image Ra of the initial display image information (see FIG. 26) included in the display setting information is displayed at the position of the AR marker 30.

  FIG. 30 is a diagram illustrating an example of the model display screen (in the case of the refrigerator Ra) according to the second embodiment. The model display screen of the second embodiment includes a model selection button 101, a setting button 102, a help button 103, a camera button 104, a model name display unit 105, an operation button 106, an installation restriction area designation button 109, a product specification display button 110, It has a recommended product display button 111, a product purchase button 112, a map display button 113, and a share button 114.

  Descriptions of the model selection button 101, the setting button 102, the help button 103, and the camera button 104 are the same as those in FIG.

  The model name display section 105 displays the model name selected by the model selection button 101. In the example of FIG. 30, the refrigerator Ra is displayed.

  The operation buttons 106 include a first door operation button 106f, a second door operation button 106g, and a third door operation button 106h.

  The first door operation button 106f is a button for operating the opening and closing of the door 51. The first door operation button 106f displays the operation indicated by the image file specified by the information having the door number of 1 in the movable part image information (in the case of the door) in FIG. 27 included in the display setting information.

  When the first door operation button 106f is pressed, an image (animation) indicating an operation of opening the door 51 is displayed. Specifically, when the virtual object 40 is the refrigerator Ra, an image (animation) indicating an operation of opening the door 51 is displayed by the Door_open image Ra1 of the movable part image information (see FIG. 27) included in the display setting information. You.

  FIG. 31 is a diagram illustrating an example of the model display screen of the second embodiment after the first door operation button 106f is pressed. When the first door operation button 106f is pressed again in this state, an image (animation) indicating an operation of closing the door 51 is displayed. Specifically, when the virtual object 40 is the refrigerator Ra, an image (animation) indicating the operation of closing the door 51 is displayed by the Door_close image Ra1 of the movable part image information (see FIG. 27) included in the display setting information. Then, the display of the refrigerator Ra returns to the state shown in FIG.

  Referring back to FIG. 30, the second door operation button 106g is a button for opening and closing the door 52. The second door operation button 106g displays the operation indicated by the image file specified by the information with the door number of 2 in the movable part image information (in the case of the door) in FIG. 27 included in the display setting information.

  The third door operation button 106h is a button for operating the opening and closing of the door 53. The third door operation button 106g displays the operation indicated by the image file specified by the information having the door number of 3 in the movable part image information (in the case of the door) in FIG. 27 included in the display setting information.

  The installation restriction area designation button 109 is a button for displaying an installation restriction area designation screen. The installation restriction area designation screen is a screen for designating an installation restriction area indicating a restriction on an area where an object corresponding to the virtual object 40 is installed. The installation restriction area designation screen according to the second embodiment is the same as the installation restriction area designation screen (see FIG. 21) according to the first embodiment, and thus the description is omitted.

  The product specification display button 110 is a button for displaying a product specification screen for displaying specification information specified by the model name displayed on the model name display unit 105. The product specification screen according to the second embodiment is the same as the product specification screen according to the first embodiment (see FIG. 22), and a description thereof will be omitted.

  The recommended product display button 111 is a button for displaying a model display screen of a model recommended to the user. The model recommended to the user is, for example, a model whose product specifications are similar to the model displayed on the model name display unit 105. When the operation receiving unit 15 receives an operation indicating that the recommended product display button 111 is pressed, the generation unit 16 sets the main image of the initial display image information included in the display setting information of the model recommended to the user on the captured image. A synthesized image synthesized at the position of the AR marker 30 is generated. Then, the display control unit 13 displays a model display screen corresponding to the composite image on the display unit 14.

  The product purchase button 112 is a button for displaying a website where the model displayed on the model name display unit 105 can be purchased. When the operation receiving unit 15 receives an operation indicating that the product purchase button 112 is pressed, the communication unit 17 acquires from the storage unit 11 the address of a website where the model displayed on the model name display unit 105 can be purchased. . Then, the display control unit 13 displays on the display unit 14 the website acquired by the communication unit 17 accessing the address.

  The map display button 113 displays a map display screen indicating the location of a store where the model displayed on the model name display unit 105 can be purchased. The map display screen according to the second embodiment is the same as the map display screen according to the first embodiment (see FIG. 23), and a description thereof will be omitted.

  The share button 114 displays a pop-up list of share applications such as Facebook (registered trademark) and Twitter (registered trademark). When the operation receiving unit 15 receives an operation indicating selection of a share application, information and the like displayed on the display unit 14 using the share application can be shared with other users.

  The information processing method of the second embodiment is the same as the information processing method of the first embodiment (see FIG. 24), and a description thereof will be omitted. The hardware configuration of the information processing apparatus 10 according to the second embodiment is the same as the hardware configuration of the information processing apparatus 10 according to the first embodiment (see FIG. 25), and a description thereof will not be repeated.

  As described above, in the information processing apparatus 10 of the second embodiment, when the operation receiving unit 15 receives an operation of the movable part of the virtual object 40 indicating the refrigerator, the virtual object 40 indicating the refrigerator is displayed. The display of the movable part is changed according to the display setting information for the purpose. Thus, even when the object is a refrigerator (home appliance), as in the first embodiment, the user can more clearly understand the image when the movable part of the refrigerator (home appliance) is operated. .

(Third embodiment)
Next, a third embodiment will be described. In the description of the third embodiment, a case where the object corresponding to the virtual object 40 is a vehicle will be described. In the description of the third embodiment, the case of an automobile will be described as an example. However, the vehicle is not limited to an automobile but may be any automobile. In the description of the third embodiment, a description similar to that of the first embodiment will be omitted, and portions different from the first embodiment will be described.

  The configuration of the information processing system 1 of the third embodiment is the same as the configuration of the information processing system 1 of the first embodiment (see FIG. 1). The configuration of the information processing apparatus 10 according to the third embodiment is the same as the configuration of the information processing apparatus 10 according to the first embodiment (see FIG. 4).

  First, display setting information (initial display image information and movable portion image information) according to the third embodiment will be described.

  FIG. 32 is a diagram illustrating an example (in the case of an automobile) of the initial display image information according to the third embodiment. The initial display image information indicates a state when the display of the virtual object 40 indicating the car is started. The initial display image information includes items of a vehicle type name and a vehicle body image. The vehicle type name indicates the name of the vehicle type. In the third embodiment, the vehicle type name is identification information for identifying the type of the virtual object 40. The body indicates specific information for specifying an image indicating a car. The specific information is, for example, an image file name. The body image is displayed at the position of the AR marker 30 included in the image captured by the information processing device 10.

  For example, in the example of FIG. 32, when the virtual object 40 corresponding to the car Ca is initially displayed on the information processing device 10, the virtual object 40 is displayed by the vehicle body image Ca.

  Next, an example of the movable part image information according to the third embodiment will be described.

  FIG. 33 is a diagram illustrating an example of movable part image information (in the case of an opening / closing unit of an automobile) of the third embodiment. The movable part image information in FIG. 33 includes items of a vehicle type name, an opening / closing section name, an opening / closing section (open), an opening / closing section (close), and a display reference point. The vehicle type name indicates the name of the vehicle type. The opening / closing section name indicates the name of the opening / closing section of the vehicle. The open / close unit (open) indicates specific information for specifying an image (animation) indicating an operation of opening the open / close unit. The opening / closing section (close) indicates specific information for specifying an image (animation) indicating an operation of closing the opening / closing section. The specific information is, for example, an image file name. The display reference point indicates a position at which an image (animation) indicating an operation of opening the opening / closing unit and an image (animation) indicating an operation of closing the opening / closing unit are displayed.

  For example, in the example of the automobile Ca in FIG. 33, an image (animation) showing an operation of opening the door of the driver's seat of the automobile Ca is an open image Ca2. An image (animation) showing the operation of closing the door of the driver's seat of the car Ca is a close image Ca2. The positions at which the open image Ca2 and the close image Ca2 are displayed are specified by the coordinates (x_ca2, y_ca2, z_ca2) of the display reference point.

  FIG. 34 is a diagram illustrating an example (in the case of a car Ca) of the virtual object 40 according to the third embodiment. The virtual object 40 representing the car Ca has a hood 61, a driver seat door 62, a passenger seat door 63, a right rear seat door 64, a left rear seat door 65, and a rear trunk 66. For example, when the operation button of the driver's seat door 62 is pressed on the vehicle type display screen of the car Ca, the driver's seat door 62 is opened by the open image Ca2 of the movable part image information (see FIG. 33) included in the display setting information. An image (animation) indicating the opening operation is displayed. Note that the model display screen of the car Ca of the third embodiment is the same as the model display screen of the first embodiment (FIG. 16) and the model display screen of the second embodiment (FIG. 30), and thus description thereof is omitted. .

  FIG. 35 is a diagram illustrating an example of the virtual object 40 according to the third embodiment (when the driver seat door 62 is open). When the operation button of the driver's seat door 62 is pressed again on the vehicle type display screen of the car Ca, the driver's seat door 62 is displayed by the close image Ca2 of the movable part image information (see FIG. 33) included in the display setting information. Is displayed, and the virtual object 40 returns to the state of FIG.

  The information processing method according to the third embodiment is the same as the information processing method according to the first embodiment (see FIG. 24), and a description thereof will be omitted. The hardware configuration of the information processing apparatus 10 according to the third embodiment is the same as the hardware configuration of the information processing apparatus 10 according to the first embodiment (see FIG. 25), and a description thereof will not be repeated.

  As described above, in the information processing apparatus 10 according to the third embodiment, when the operation receiving unit 15 receives an operation on the movable part of the virtual object 40 indicating the car, the virtual object 40 indicating the car is displayed. The display of the movable part is changed according to the display setting information for the purpose. As a result, even when the object is a car (vehicle), the user can more clearly understand the image when the movable part of the car (vehicle) is operated, as in the first and second embodiments. Can be.

  In the description of the first to third embodiments, the case where the object corresponding to the virtual object 40 is a printer (office device), a refrigerator (home electric appliance), and an automobile (vehicle) has been described as an example. The object may be arbitrary. For example, the object corresponding to the virtual object 40 may be furniture such as a closet.

Reference Signs List 1 information processing system 10 information processing device 11 storage unit 12 imaging unit 13 display control unit 14 display unit 15 operation reception unit 16 generation unit 17 communication unit 18 GPS unit 20 server device 30 AR marker 40 virtual object 100 network 301 control device 302 main Storage device 303 Auxiliary storage device 304 Display device 305 Input device 306 Communication device 307 Imaging device 308 GPS device

Japanese Patent No. 4032776

Claims (15)

An imaging unit that acquires a captured image including position identification information that identifies a position at which a virtual object represented by an image having an immovable part and a movable part is displayed,
A generation unit configured to generate a combined image in which the virtual object is combined at a position on the captured image identified by the position identification information;
An operation receiving unit that receives an operation of the movable part included in the composite image displayed on the display unit,
A display control unit that controls display of the movable part based on display setting information for displaying the virtual object when an operation of the movable part is received by the operation reception unit ,
The operation receiving unit further receives an operation of specifying an installation restricted area indicating a restriction on an area where an object corresponding to the virtual object is installed,
The generation unit, the information indicating the installation restricted area, generates the combined image further combined with the captured image,
Information processing device. A communication unit that receives the display setting information;
A storage unit for storing the display setting information,
The information processing apparatus according to claim 1, further comprising: The display control unit changes a display mode of information indicating the installation restricted area when the virtual object does not fit in the installation restricted area,
The information processing device according to claim 1 . The operation receiving unit further receives an operation indicating selection of a virtual detachable object that is detachable from the virtual object,
The generating unit generates the composite image in which the virtual object including the virtual detachable object is composited with the captured image.
The information processing apparatus according to any one of claims 1 to 3. The operation receiving unit receives an operation of selecting identification information of the virtual object from identification information of the plurality of virtual objects,
The generation unit generates a synthesized image in which the virtual object identified by the selected identification information is synthesized at a position identified by the position identification information included in the captured image,
The information processing apparatus according to any one of claims 1 to 4. The generation unit performs at least one of enlargement, reduction, movement, and rotation on the virtual object in accordance with at least one of a position and a shape of the position identification information on the captured image, Synthesizing a synthesized image in which the virtual object whose display mode has been changed is synthesized,
The display control unit displays, on a display unit, a combined image in which the virtual object whose display mode is changed is combined.
The information processing apparatus according to any one of claims 1 to 5. The display control unit displays a site on a network where an object corresponding to the virtual object is available on the display unit,
The information processing apparatus according to any one of claims 1 to 6. The display control unit displays a map indicating a location where an object corresponding to the virtual object is available on the display unit,
The information processing apparatus according to any one of claims 1 to 7. The display control unit displays information of an object corresponding to the virtual object on the display unit,
The information processing apparatus according to any one of claims 1 to 8. The display control unit displays information on a similar object having information similar to information on an object corresponding to the virtual object on the display unit.
The information processing apparatus according to any one of claims 1 to 9. The display control unit displays a moving image of the virtual object indicating a procedure for replacing parts of the object corresponding to the virtual object,
The information processing apparatus according to any one of claims 1 to 10. The object corresponding to the virtual object includes a printing device,
The display control unit displays a moving image of the virtual object indicating a state in which a printed material is discharged from the printing device when printing is performed, according to a printing speed of a printing device corresponding to the virtual object. ,
The information processing apparatus according to any one of claims 1 to 11. The display control unit displays a region selection screen for selecting a region from a plurality of regions on the display unit, and performs display control based on region setting information associated with the selected region.
The information processing apparatus according to any one of claims 1 to 12. The imaging unit acquires a captured image including position identification information for identifying a position at which a virtual object represented by an image having an immovable part and a movable part is displayed,
A generating unit that generates a synthesized image in which the virtual object is synthesized at a position on the captured image identified by the position identification information;
An operation receiving unit that receives an operation of the movable part included in the composite image displayed on the display unit;
When the operation of the movable part is received by the step of receiving the operation, a display control unit controls the display of the movable part based on display setting information for displaying the virtual object ,
The accepting step further accepts an operation of specifying an installation restricted area indicating a restriction on an area where an object corresponding to the virtual object is installed,
The generating step generates the combined image in which the information indicating the installation restricted area is further combined with the captured image,
Information processing method. An information processing apparatus including an imaging unit that acquires a captured image including position identification information that identifies a position at which a virtual object represented by an image having an immovable part and a movable part is displayed,
A generation unit configured to generate a combined image in which the virtual object is combined at a position on the captured image identified by the position identification information;
An operation receiving unit that receives an operation of the movable part included in the composite image displayed on the display unit,
When the operation of the movable part is received by the operation receiving unit, the display control unit controls the display of the movable part based on display setting information for displaying the virtual object ,
The operation receiving unit further receives an operation of specifying an installation restricted area indicating a restriction on an area where an object corresponding to the virtual object is installed,
The generation unit, the information indicating the installation restricted area, generates the combined image further combined with the captured image,
program.

Images