JP2012155678A - Display system, display processing apparatus, display method and display program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a display system in which an object can be displayed easily to point in virtual reality where image information representing a virtual world is superimposed on a photographic image and displayed while being linked on real coordinates.SOLUTION: In a display system, an image of a virtual object, a display position set on a three-dimensional coordinate system set to a photographic space and processing to be executed selectively are specified for each virtual object. The three-dimensional coordinate system is set to an image photographed by a camera, and a display image for displaying the image of the virtual object is generated at the specified position. When a position designation from a user is received in the display image, that position is detected as a position in the three-dimensional coordinate system and if a distance between that position and the display position of the virtual object is smaller than a threshold value, a size of any other virtual objects than that virtual object is reduced.

Description

  The present invention relates to a display system, a display processing device, a display method, and a display program, and more particularly to a display system, a display processing device, a display method, and a display program for displaying selectable objects on a captured image.

  A technique has been proposed in which image information representing a virtual world is superimposed on a captured image and displayed to display virtual reality. In the virtual reality display, a photographed image representing the real world and image information representing the virtual world have a layer structure, and the real space and the virtual space are linked by real coordinates.

  Since a system using virtual reality has such a display configuration, it is difficult to grasp the sense of depth in the three-dimensional space when specifying the position of the object in the virtual space, and it is difficult to specify the position of the object. was there.

  In order to solve this problem, there is an example in which the color of at least one of the instruction area or the object is changed according to the distance between the instruction area of the user and the object. As a specific example, Japanese Patent Application Laid-Open No. 5-342322 (hereinafter referred to as Patent Document 1) discloses a color of a pointer or an object in a stepwise manner in a three-dimensional space based on the distance between the designated area coordinates and the coordinates associated with the object. The technology which changes is disclosed.

  As another example, Japanese Patent Application Laid-Open No. 2002-109568 (hereinafter referred to as Patent Document 2) discloses an object other than an object serving as a mark in order to make it easier to grasp a target object when there are a plurality of objects in a virtual space. Is disclosed. Japanese Patent Laying-Open No. 2003-337959 (hereinafter, Patent Document 3) discloses a technique for changing the display form of an object based on the distance from the viewpoint to the object.

JP-A-5-342322 JP 2002-109568 A JP 2003-337959 A

However, the techniques disclosed in these patent documents have a problem that it is difficult to grasp the sense of depth in the three-dimensional space and it is difficult to make a selection while visually recognizing the three-dimensional display. In particular,
When there are a large number of objects close to each other, there is a problem that it is difficult to select and specify a specific object from among them if it is difficult to grasp the sense of depth.

  The present invention has been made in view of such problems, and it is possible to display an object that is easy to point in virtual reality in which image information representing a virtual world is superimposed on a captured image and displayed by linking with real coordinates. An object of the present invention is to provide a display system, a display processing device, a display method, and a display program.

  In order to achieve the above object, according to an aspect of the present invention, a display system is a display system for displaying an image of a virtual object, and includes a camera, a display, and a display processing device. The display processing apparatus includes, as information about the virtual object, image information for displaying the virtual object in three dimensions, and the virtual object represented in a three-dimensional coordinate system set for the captured space. An acquisition means for acquiring from the storage device information defining the display position and the action executed by selecting the virtual object, and a three-dimensional coordinate system for the captured image by the camera, A processing unit for generating a display image for displaying an image of the virtual object at a display position defined for the virtual object and displaying the image on the display, and a three-dimensional coordinate system specified by the user And detecting means for detecting the position at. The processing means identifies one virtual object of the virtual objects as a virtual target object based on the relationship between the display position specified for the virtual object in the captured image and the specified position, and the virtual target of the virtual object A process of changing an image of a virtual non-target object is executed using a virtual object other than the object as a virtual non-target object.

  Preferably, the changing process includes a process of reducing the image of the virtual non-target object, a process of changing the shape of the image of the virtual non-target object, a process of changing the color of the image of the virtual non-target object, and the virtual non-target object. At least one of the process of changing the transparency of the image and the process of changing the display position of the image of the virtual non-target object.

  Preferably, the processing means sets, as a virtual target object, a virtual object in which a distance between a display position specified for a virtual object in a captured image and a specified position is smaller than a first reference distance specified in advance. Identify.

  More preferably, the processing unit reduces the image of the virtual non-target object according to the distance between the designated position and the virtual target object.

  Preferably, the processing means identifies a virtual object whose display position corresponds to a direction in which the position specified based on a change amount of the position specified by the user is a virtual target object.

  More preferably, the processing unit changes the display position of the virtual non-target object in a direction away from the virtual target object according to the distance between the designated position and the virtual target object.

  Preferably, the processing means changes the image of the virtual non-target object displayed overlapping the virtual target object based on the positional relationship between the camera position and the virtual object, and the virtual non-target object image not displayed overlapping the virtual target object. Change it more than the change.

  More preferably, when a change in the position of the camera is detected, the processing unit displays a change in the image of the virtual non-target object identified as being displayed on the virtual target object in accordance with the position of the camera before the change. Keep the camera position after moving.

  Preferably, the processing means is a virtual object in which a distance between the display position specified for the virtual object in the photographed image and the designated position is smaller than the second reference distance that is larger than the first reference distance. Thus, virtual objects other than the virtual object identified as the virtual target object are identified as non-virtual target objects.

  Preferably, the display transmits light from a surface opposite to the display surface, and displays the display image generated by the display processing unit with transparency on the display surface.

  According to another aspect of the present invention, the display processing device is a display processing device for displaying an image of a virtual object on a display, and includes input means for accepting an input of a captured image, and information related to the virtual object. The image information for displaying the virtual object in three dimensions, the display position of the virtual object represented in the three-dimensional coordinate system set for the captured space, and the virtual object are selected. Acquisition means for acquiring information defining the action executed by the storage device, a three-dimensional coordinate system is set for the captured image, and the virtual object is set at the display position specified for the virtual object. Processing means for generating a display image for displaying the image and displaying the image on the display; And a detection means for detecting the position in the specified 3-dimensional coordinate system Te. The processing means identifies one virtual object of the virtual objects as a virtual target object based on the relationship between the display position specified for the virtual object in the captured image and the specified position, and the virtual target of the virtual object A process of changing an image of a virtual non-target object is executed using a virtual object other than the object as a virtual non-target object.

  According to still another aspect of the present invention, the display method is a method of displaying an image of a virtual object on a display, the step of receiving an input of a captured image from a camera, and a three-dimensional coordinate system is set for the captured image A step of detecting a position in a three-dimensional coordinate system designated by a user, image information for displaying the virtual object in three dimensions as information on the virtual object, and a captured space Acquiring from the storage device information defining the display position of the virtual object represented in the set three-dimensional coordinate system and the action executed when the virtual object is selected; Based on the relationship between the specified display position and the specified position for the virtual object in One virtual objects among the objects identified as virtual object as a virtual object outside the object virtual object other than the virtual object of a virtual object, and a step of changing the image of the virtual object outside the object.

  According to still another aspect of the present invention, the display program is a program for causing a display processing device to execute processing for displaying an image of a virtual object on a display, the step of receiving an input of a captured image from a camera, and a captured image A step of setting a three-dimensional coordinate system for the image, a step of detecting a position in the three-dimensional coordinate system designated by the user, and an image for displaying the virtual object in three dimensions as information on the virtual object Information defining information, a display position of the virtual object represented in a three-dimensional coordinate system set for the imaged space, and an action to be executed when the virtual object is selected The step of obtaining from the storage device and the virtual object in the captured image Based on the relationship between the indicated position and the specified position, one virtual object of the virtual objects is identified as a virtual target object, and a virtual object other than the virtual target object of the virtual objects is defined as a virtual non-target object. And causing the display processing device to execute a step of changing the image of the non-target object.

  According to the present invention, it is possible to more intuitively and easily select an object as an option to be displayed in a virtual reality displayed by superimposing image information representing a virtual world on a captured image and linking with real coordinates. Is possible.

It is a figure which shows the specific example of a structure of the display system concerning this Embodiment. It is a figure which shows the other specific example of a structure of the display system concerning this Embodiment. It is a block diagram which shows the specific example of the apparatus structure of the display contained in a display system. It is a block diagram which shows the specific example of the apparatus structure of the pointer contained in a display system in the case of being a pointing device. It is a figure explaining an example of how to use the display system concerning this embodiment. It is a figure explaining an example of how to use the display system concerning this embodiment. It is a block diagram which shows the specific example of a function structure of the processing apparatus contained in a display system. It is a flowchart which shows the specific example of basic display operation. It is a figure which shows the specific example of the image memorize | stored as a marker. It is a figure which shows the example of a detection of the marker in a picked-up image, and the example of a display of the virtual object based on the detection result. It is a figure which shows an example of the display in a display system. It is a figure which shows the specific example of the coordinate value of the virtual object memorize | stored. It is a flowchart which shows the specific example of the operation | movement which calculates the virtual object-pointer distance. It is a figure explaining the 1st example of the display process using the distance between a pointer position and the position of a virtual object. It is a flowchart which shows the specific example of the operation | movement for the display process concerning a 1st example. It is a figure explaining the modification of the 1st example of a display process. It is a figure explaining the modification of the 1st example of a display process. It is a figure explaining the modification of the 1st example of a display process. It is a figure explaining the modification of the 1st example of a display process. It is a figure explaining the 2nd example of the display process using the distance between a pointer position and the position of a virtual object. It is a flowchart which shows the specific example of the operation | movement for the display process concerning a 2nd example. It is a figure explaining the modification of the 2nd example of a display process. It is a figure explaining the modification of the 2nd example of a display process. It is a figure explaining the modification of the 2nd example of a display process.

  Embodiments of the present invention will be described below with reference to the drawings. In the following description, the same parts and components are denoted by the same reference numerals. Their names and functions are also the same.

<System configuration>
FIG. 1 is a diagram illustrating a specific example of the configuration of the display system according to the present embodiment.

  Referring to FIG. 1, a display system according to the present embodiment includes a processing device 100, a pointer 200, a camera 300, a display 400 that is a wearable display, a server 500, and a portable terminal 600 that is an object setting device. Including.

  The processing device 100 is electrically connected to the camera 300, the display 400, and the server 500, respectively. This connection may be a connection via a specific network such as a LAN (Local Area Network) or a connection via a public line such as the Internet. Moreover, either wireless or wired may be used.

  The processing device 100 and the server 500 may be configured by a general personal computer (PC) or the like.

  The portable terminal 600 may be configured with a mobile phone, a portable PC, or the like, or may be configured with a normal, general PC.

  The camera 300 may be any device as long as it can be electrically connected to the processing device 100 and has a function of transmitting a captured image to the processing device 100.

  The display 400 is a wearable display integrated with the camera 300. The display 400 includes a display 401 (see FIG. 3) as a display unit having transparency. The display 400 is electrically connected to the processing apparatus 100 and receives image data transmitted from the processing apparatus 100 as a display unit 401. And has a function of displaying with transparency.

  The pointer 200 has a predetermined shape or pattern. The shape or pattern is registered in advance in the processing apparatus 100 and used as a marker. That is, the processing device 100 specifies the position of the pointer 200 as a marker present in the image photographed by the camera 300 by image processing. In the example of FIG. 1, the pointer 200 has a cylindrical shape that is attached to the user's fingertip, and the processing device 100 stores the cylindrical shape or a pattern applied to the surface thereof in advance. And However, the pointer 200 is not limited to the one attached to a part of the user's body, and may be, for example, an indicator bar having a shape or a pattern stored in advance. Further, the pointer 200 is not used, and the shape of a part of the body that is used for directing the user's fingertip or the like or the pattern provided on the surface thereof is stored in advance in the processing apparatus 100 as a marker, and is photographed by the camera 300. The position of the marker existing in the processed image may be specified by image processing in the processing apparatus 100.

  As another example of the pointer 200, a user operation that can be electrically connected to the processing apparatus 100, is equipped with sensors such as a position sensor, an acceleration sensor, and an inclination sensor, and designates a position using the pointer 200. A pointing device having a function of receiving and detecting the position and the like and transmitting the detected position to the processing apparatus 100 may be used.

  Furthermore, as another example of the pointer 200, a microphone that can be electrically connected to the processing apparatus 100 and receives voice input may be used. The user can instruct the position to be designated using the microphone, for example, by coordinates. In this case, the processing apparatus 100 has a voice analysis function, and analyzes the voice signal from the microphone to identify the designated position.

  The processing device 100 performs processing based on the image received from the camera 300. And the process for displaying the processing result on the display 400 is performed.

  The portable terminal 600 includes at least an operation unit for receiving an input operation from a user and a communication unit for communicating with the server 500. An input of information related to a virtual object, which will be described later, is received by an operation on the operation unit, and the information is transmitted to the server 500. The server 500 stores information on the input virtual object, and provides the information to the processing device 100 in response to a request from the processing device 100.

  When the display system according to the present embodiment has the configuration shown in FIG. 1, the display unit of the display 400, in which a user displays an image that is displayed with transparency on the display 400 that is a wearable display, is used. Pointing using the pointer 200 attached to the fingertip while looking at the scenery that can be seen through the display 401 (see FIG. 3).

  In the display system illustrated in FIG. 1, a captured image for displaying on the display 400 from the camera 300 and for specifying the position of the pointer 200 is input to the processing device 100. However, a camera for obtaining a captured image for specifying the position of the pointer 200 may be provided separately from the camera 300.

<Other examples of system configuration>
In the following description, the display system according to the present embodiment has the configuration shown in FIG. 1, and the user projects an image displayed with transparency on the display 400 that is a wearable display. It is assumed that the user points to the scenery using the pointer 200 attached to the fingertip while viewing the scenery through the display 401. However, as another example, the display 400 may not be a wearable display, but may be constituted by a PC electrically connected to the camera 300 and the pointer 200 as shown in FIG. When the connection between the camera 300 and the PC is via the Internet, the camera 300 may be a WEB camera, for example. On the display 400 configured by the PC, the captured image from the camera 300 and the image from the processing apparatus 100 are combined and displayed. In this case, the user can point using the pointer 200 attached to the fingertip while viewing the display.

<Device configuration>
Since the processing device 100, the server 500, and the mobile terminal 600 may be configured by a general PC, a mobile phone, or the like as described above, the device configuration is not limited to a specific configuration. That is, the processing device 100, the server 500, and the portable terminal 600 store at least a CPU (Central Processing Unit) for controlling the whole, a program executed by the CPU, image information necessary for displaying virtual reality, and the like. For example, a ROM (Read Only Memory) or a RAM (Random Access Memory), and a communication interface (I / F) for electrical connection with other devices to exchange data. In addition, as described above, the mobile terminal 600 further includes an operation unit that receives an operation input from the user.

  The camera 300 includes a camera unit, a memory for storing captured images and programs, a CPU for executing processing for transmitting the captured images to the processing device 100, and a sensor for detecting the position of the camera 300. And a communication I / F for electrically connecting to the processing apparatus 100 to transmit a captured image.

  FIG. 3 is a block diagram illustrating a specific example of the device configuration of the display 400. Referring to FIG. 3, display 400 is electrically connected to display 401 having transparency as a display unit, CPU 402 for executing processing for displaying a display image on display 401, and processing device 100. And a communication I / F 403 for receiving a display image.

  The CPU 402 obtains display data transmitted from the processing device 100 among data (packets) on the network received by the communication I / F 403.

  FIG. 4 is a block diagram showing a specific example of the device configuration of the pointer 200 in the case of a pointing device. Referring to FIG. 4, pointer 200 executes, as an example, a sensor such as position detection sensor 201, acceleration sensor 202, directionality detection sensor (tilt sensor) 203, and processing for detecting position information and the like using these sensors. And a communication I / F 205 for electrically connecting to the processing apparatus 100 and transmitting a detection result. Furthermore, a memory (not shown) for storing the detection result of the sensor and the program may be included.

<Overview of operation>
In the display system according to the present embodiment, an image representing a virtual object (virtual object) generated in advance by computer graphics or the like is displayed in accordance with a display position set on mobile terminal 600 with respect to a captured image representing the real world. The display 400 is displayed with transparency. As a result, for a user wearing the display 400 that is a wearable display, the virtual object appears to be floating in the real world where the virtual object can be visually recognized via the display 400. At this time, in order to display the virtual object on the display 400 so as to actually float, the depth is set with respect to the display position of the virtual object on the portable terminal 600, and the virtual object is displayed in three dimensions according to the set depth. indicate. As a result, the virtual object is displayed so as to float at a certain depth in the real space that can be viewed through the display 400.

  A “virtual object” refers to a three-dimensional image that is an option associated with content in a captured image representing the real world. For each virtual object, image information for displaying the virtual object, and a display position (hereinafter also referred to as virtual object coordinates) in a three-dimensional coordinate system set for the shooting space represented by the shot image. The operation to be executed by being selected is defined. Information regarding the virtual object is set by the portable terminal 600 and stored in the server 500.

  The processing apparatus 100 sets a three-dimensional coordinate system set for the photographing space in the photographed image input from the camera 300 and then sets a display position and a depth in the same three-dimensional coordinate system. Specify the display position of the object. Then, the image of the virtual object is processed so as to be displayed at the position, and the processed display image is transmitted to the display 400.

  The processing device 100 identifies the position designated by the pointer 200 by executing image processing on the photographed image, and converts the position into a position in a three-dimensional coordinate system set for the photographing space. . Then, a prescribed action is performed on the virtual object corresponding to the designated position. Alternatively, a control signal is output to a necessary device (not shown) to execute the operation.

  An example of how this display system is used is as shown in FIG. In FIG. 5, the image information is displayed on the display 400 with transparency so that the image information that is virtual reality is projected onto the real space in the room that is visually recognized through the display 400 that is a wearable display. 1 shows an example of display on a display system. In FIG. 5, a three-dimensional sphere represents a virtual object.

  The display of FIG. 5 is a virtual object displayed on the display 400 with transparency when the user wearing the display 400, which is a wearable display, stands at the right rear of the user facing the desk and faces the room. Is a display representing a landscape projected by the user.

  In the example of FIG. 5, the options represented by virtual objects are associated with the actual coordinates at the upper left position of the calendar or user as the contents in the room that is the real space. Therefore, the virtual object is displayed based on a predetermined display position, in the example of FIG. 5, in the example of FIG. ing.

  When the user wearing the wearable display while visually recognizing the display makes the fingertip or the like wearing the pointer 200 the position of the virtual object displayed on the display 400, the user performs the following movements as defined for the virtual object: The associated data is displayed and can be viewed, and the corresponding device can be operated like a remote control. As an example, when a virtual object displayed in the vicinity of the calendar in FIG. 5 is touched with the pointer 200, schedule information of a user registered in advance is displayed. As another example, when a user touches the virtual object at the upper left, a control signal is output to an air conditioner that is associated with the position in advance, and the air conditioning temperature is changed.

  Another example of how this display system is used is as shown in FIG. FIG. 6 shows the display 400 with transparency on the display 400 so that the image information of the virtual reality is projected onto the real space in the factory visually recognized through the display 400 which is a wearable display. In addition, an example of display on the display system is shown. Also in FIG. 6, a three-dimensional sphere represents a virtual object.

  The display in FIG. 6 shows that the display 400 has transparency when the user wearing the display 400 that is a wearable display stands further to the right of the two users facing the desk and faces the factory from the room. The displayed virtual object is projected and displayed as a scene visually recognized by the user.

  In the example of FIG. 6, options represented by virtual objects are associated with real coordinates such as equipment as content in a factory that is a real space. Therefore, the virtual object is displayed in the vicinity of the content associated with the option, such as the vicinity of each device in the factory, in the example of FIG. 6 based on the display position defined in advance. .

  When the user wearing the wearable display while visually recognizing the display makes the fingertip or the like wearing the pointer 200 the position of the virtual object displayed on the display 400, the virtual object is associated as an example of an action defined for the virtual object. You can access information (progress management, error information, handling method) sent from each device in the factory. More preferably, in the display system, the color and shape of the virtual object may be changed based on a preset correspondence relationship according to the urgency of the error and the type of accessible data. This makes it possible to inform the user of the urgency level of the error, the type of data that can be accessed, and the like in an easily understandable manner.

  More preferably, the information on the virtual object may be associated with each user or a group (department or the like) to which the user belongs. Thus, by logging in to the display system in advance, the virtual object corresponding to the user or the group (operable or permitted) is displayed according to the authority given to the user or the group to which the user belongs. The

  As shown in FIGS. 5 and 6, in the display on the display system, the virtual object is three-dimensionally displayed on the display 400 which is a two-dimensional display device. Therefore, if it is difficult to visually recognize the sense of depth of the virtual object, it is difficult for the user to specify the desired virtual object. In particular, when a plurality of virtual objects having different depths are displayed close to each other on the display 400, the virtual objects are overlapped on the two-dimensional plane display screen unless the sense of depth is visually recognized. However, it becomes difficult to specify a desired virtual object.

  Therefore, the display system according to the present embodiment executes display processing according to the distance between the display position (three-dimensional coordinates) defined for the virtual object and the position (three-dimensional coordinates) specified by the pointer 200. Thus, even in such a case, the virtual object desired by the user is displayed so that it can be easily specified.

  Specifically, the processing device 100 can designate a virtual object within a predetermined range from the pointer position on the pointer 200 based on the relationship between the display position of the virtual object and the designated position on the pointer 200. The virtual object is identified as a virtual object that is not likely to be designated. This virtual object that is unlikely to be designated is referred to as a “virtual object” in the following description.

  The processing apparatus 100 makes it easy to specify a virtual object that is likely to be specified to the user by making the display mode of the virtual non-target object different from the display mode of the virtual object that is not so.

<Functional configuration>
FIG. 7 is a block diagram showing a specific example of a functional configuration of the processing apparatus 100 for the display system according to the present embodiment to perform the operation described in the above operation outline. FIG. 7 mainly shows a specific example of a functional configuration when the CPU functions as an image processing control unit by reading and executing a program stored in a memory by a CPU (not shown). The functions shown in FIG. 7 are mainly configured on the CPU. However, a part of it may be realized by a hardware configuration.

  Referring to FIG. 7, the CPU of processing apparatus 100 functioning as an image processing control unit receives a camera image input unit 101 </ b> A for receiving an input of a captured image from camera 300 and an input signal from pointer 200. Pointer data input unit 101B, virtual object information input unit 101C for accepting input of virtual object information provided from server 500, point position on pointer 200 and display area of virtual object based on these input information By performing image recognition processing on the input captured image, the pointer object extraction unit 102 for detecting the above and the like on the captured image such as the point position on the extracted pointer 200 and the display area of the virtual object Image recognition processing unit for specifying the position (coordinates) 03, the pointer shape storage unit 104 for storing the shape of the pointer 200, and the display data for displaying the pointer 200 on the captured image are generated using the shape of the pointer 200 stored as necessary. In order to calculate the distance between the display position on the pointer 200 (hereinafter also referred to as the pointer position) and the display position of the virtual object based on the processing results of the pointer display data generation unit 105 and the image recognition processing unit 103. Virtual object-pointer distance determination unit 106, a virtual non-object-to-pointer distance determination unit 107 for calculating a distance from the display position of the virtual non-target object, and a display position for each virtual non-target object for each object. A virtual non-target object coordinate storage unit 108 for storing A virtual non-target object display data generating unit 109 for generating display data for displaying virtual non-target objects by performing the processing described above, and a virtual non-target object shape storage unit for storing the shapes of virtual non-target objects to be displayed 110 and a display processing unit 111 for generating a display image for displaying the generated display data on the display 400 with transparency.

  Further, referring to FIG. 7, pointer object extraction unit 102 includes a pointer extraction unit 1021, a virtual object extraction unit 1022, and a virtual non-target object extraction unit 1023.

  The pointer extraction unit 1021 is a function for extracting the position designated by the pointer 200 from the captured image. Specifically, the shape of the pointer 200 stored in the pointer shape storage unit 104 is extracted from the input captured image.

  The virtual object extraction unit 1022 and the virtual non-target object extraction unit 1023 are functions for extracting a region where a virtual object and a virtual non-target object are displayed in the captured image, respectively.

  Specifically, the display position of the virtual object in the three-dimensional coordinate system is stored in the server 500 as information preliminarily defined for the virtual object, and the virtual object extraction unit 1022 performs processing based on the information. An area including the display position of the virtual object is extracted as a display area. Further, the display position of the virtual non-target object in the three-dimensional coordinate system is stored in the virtual non-target object coordinate storage unit 108 as information preliminarily defined for the virtual non-target object, and the virtual non-target object extraction unit 1023 is stored. Extracts a region including the display position of the virtual non-target object to be processed as a display region based on the stored information.

  Further, referring to FIG. 7, the image recognition processing unit 103 includes a pointer coordinate detection unit 1031, a virtual object display area detection unit 1032, and a virtual non-target object display area detection unit 1033.

  The pointer coordinate detection unit 1031 is a function for specifying a position indicated by the pointer 200 extracted by the pointer object extraction unit 102 on the captured image. Specifically, the position and depth on the captured image of the position indicated by the pointer 200 extracted by the pointer object extraction unit 102 is specified. And it converts into the coordinate (henceforth a pointer coordinate) showing the three-dimensional position in the three-dimensional coordinate system set to this picked-up image.

  When the pointer 200 is the above-described pointing device, the processing apparatus 100 accepts an input of a detection result from the pointer 200 that is a pointing device instead of this function, and determines the position on the captured image based on the detection result. Has a function to identify.

  The virtual object display area detection unit 1032 and the virtual non-target object display area detection unit 1033 are set as captured images of the virtual object display area and the virtual non-target object display area extracted by the pointer object extraction unit 102, respectively. An area in the three-dimensional coordinate system is detected as a display area.

  In addition, the virtual object display area detection unit 1032 and the virtual non-target object display area detection unit 1033 display the post-change when the display position defined in advance for the virtual object or the virtual non-target object is changed by the display process described later. Specify the display position.

  The virtual object-pointer distance determination unit 106 and the non-virtual object-pointer distance determination unit 107 obtain distances between the pointer coordinates and the virtual object coordinates expressed in a three-dimensional coordinate system, respectively.

  Further, the virtual object-pointer distance determination unit 106 stores a reference distance in advance, and when the calculated distance is larger than the reference distance, that is, a predetermined range from the position defined as the display position of the virtual object. When a position farther than is specified, it is determined that the virtual object is a virtual non-target object that is unlikely to be specified, and the result is input to the virtual non-target object display data generation unit 109. Similarly, the virtual non-object-to-pointer distance determination unit 107 also determines that the target virtual object is unlikely to be designated as a virtual non-target object when the calculated distance is larger than the reference distance. The result is input to the virtual non-object display data generation unit 109.

  The virtual non-object display data generation unit 109 generates display data for displaying the virtual object as a non-target object according to the determination result. This process will be specifically described in each of the following embodiments.

<Basic display operation flow>
In the display system according to the present embodiment, an operation for superimposing and displaying image information representing a virtual object (virtual object) drawn by computer graphics or the like on a photographed image representing the real world seen by the user will be described. Here, as a specific example, an operation realized using a program as a software library developed at the University of Washington HIT Laboratory for augmented reality research will be described. The program used is not limited to this program.

  FIG. 8 is a flowchart showing a specific example of the basic display operation. The operation shown in the flowchart of FIG. 8 is realized by a CPU (not shown) of the processing apparatus 100 reading and executing the above-described program stored in the memory.

  Referring to FIG. 8, first, when the process starts, the CPU functioning as the image processing control unit executes an initialization process in step S <b> 101. One of the initialization processes here includes a process of initializing a coordinate system set for a captured image stored in a memory (not shown).

  In step S <b> 103, the CPU receives an input of a captured image from the camera 300 and acquires a captured image. The CPU identifies the display position of the virtual object based on the photographed image every time a photographed image (one frame) is input, performs a process corresponding to the display position on the image data of the virtual object, and displays the display. The operation of transmitting the display image to 400 is repeated. Thereby, according to the captured image continuously input with the movement of the camera 300, a three-dimensional virtual object is continuously displayed on the display 400 with transparency.

  In step S105, the CPU draws the input photographed image in a drawing memory (not shown), and in step S107, the photographed image representing the content specifying the position serving as the reference of the coordinate system from the images developed in the memory. Are detected as markers. The CPU stores in advance an image to be a marker in a reference state and a relationship between the marker and coordinates in a three-dimensional coordinate system set in the imaging space.

  As an example, it is assumed that the CPU stores the image of FIG. 9 as a marker image. As an example, the position of the plate-like content represented in the image of this marker in the shooting space is set as the origin of the three-dimensional coordinate system, and the x-axis, y-axis, z in three directions with respect to this content. It is assumed that the correspondence relationship with the three-dimensional coordinate system of setting an axis is stored.

  In this case, in step S107, the CPU retrieves the image of FIG. 9 from the captured image developed in the memory. When the image or an image similar to the image is detected, it is recognized that a marker exists in the captured image, and the CPU compares the reliability in step S109. As an example of the process in step S109, the CPU compares the marker image stored in advance with the detected image, and determines whether or not the degree of correlation satisfies a criterion set as a reference. Compare

  Thereafter, in step S111, the CPU 300 captures the captured image based on the position of the marker image detected from the captured image and the comparison between the detected marker image and the stored image. Calculate the position and direction. In step S113, based on the calculated position and direction of the camera 300, with respect to the captured image, the position of the image as a marker in advance is used as the origin, and the x-axis and y-axis are set according to the inclination of the image as the marker. , Set the z-axis.

  Further, based on the stored virtual object information, each virtual object coordinate stored as a display position in three-dimensional coordinates is specified in a three-dimensional coordinate system set in the photographed image, and the depth of the photographed image or The image of the virtual non-target object is converted (deformed) according to the tilt, and is displayed with transparency at the specified position. As a result, the virtual object is three-dimensionally displayed at the specified display position in a shape corresponding to the shooting position and shooting direction.

  For example, when the image of FIG. 9 is stored as a marker image, and the image of FIG. 10A is detected as a marker in the captured image, the shooting position of the camera 300 is the content represented by the marker. The angle at which the shooting direction looks down on the content and the position and direction of the camera 300 are calculated. The CPU sets a three-dimensional coordinate system defined in advance in relation to the marker for the captured image based on the imaging position and the imaging direction.

  Furthermore, in the information about the virtual object, when there is a virtual object whose display position is the origin where the content as the marker shown in the image of FIG. 9 is displayed, and the shape of the virtual object is a cube, the CPU Is deformed based on the distance and direction from the camera 300 to the content, and the like is drawn with transparency at a specified display position. As a result, as shown in FIG. 10B, a three-dimensional virtual object deformed as seen from the same position and direction as the content represented by the marker in the vicinity of the marker image as the origin. Are displayed with transparency.

  In step S115, the CPU transmits the image data of the virtual object obtained by the above processing to the display 400 together with a control signal for display. On the display 400, processing for displaying image data of the virtual object is executed. As a result, as shown in FIG. 10B, the virtual object is viewed in a state where the virtual object is floated in the real space viewed through the display 400 that is a wearable display.

  The CPU repeats the above processing every time a photographed image is input from the camera 300 until a signal for requesting the end of display is input. When an input of a signal requesting the end of display is received (YES in step S117), the series of operations is ended.

<Flow of calculating virtual object-pointer distance>
As described above, the CPU of the processing apparatus 100 functioning as the image processing control unit determines the position on the three-dimensional coordinate system where the virtual object is displayed and the position specified by the pointer 200 on the display screen of the display 400. Display processing corresponding to the distance from the position converted into the three-dimensional coordinate system is executed. Therefore, a method for calculating the distance in the virtual object-pointer distance determination unit 106 will be described. This calculation method is the same as the distance calculation method in the non-virtual object / pointer distance determination unit 107.

  FIG. 11 is a diagram illustrating an example of display on the display system. FIG. 11 shows a landscape visually recognized by a user wearing a display 400 that is a wearable display, and an image displayed with transparency on the display 400 can be viewed through the display 400. It is projected on. In the following examples, the display examples similarly indicate the scenery visually recognized by the user wearing the display 400 that is a wearable display.

  Referring to FIG. 11, it is assumed that virtual object A and virtual object B are displayed on display 400 so as to be projected in real space. In this case, the server 500 stores a coordinate value of a three-dimensional coordinate system indicating the display position of each virtual object. As an example, as illustrated in FIG. 12, a three-dimensional coordinate value indicating a display position set for each virtual object is stored in a table format.

  The CPU further detects the position of the pointer 200 in the captured image every time the captured image is input, converts the position into a position on the 3D coordinate system set for the captured image, and the three-dimensional coordinate system. Get the pointer coordinates of.

  FIG. 13 is a flowchart illustrating a specific example of the operation of calculating the virtual object-pointer distance.

  Referring to FIG. 13, the CPU of processing apparatus 100 functioning as the image processing control unit refers to the pointer coordinate value and the coordinate value of each virtual object stored on the three-dimensional coordinate system stored in step S201. In step S203, as an example, the square root of the sum of the squares of the difference between the respective coordinate values is calculated to obtain the Euclidean distance, thereby setting each captured image between the pointer position and the display position of each virtual object. The distance in the determined three-dimensional coordinate system is calculated.

  Hereinafter, an example of display processing based on the relationship between the calculated pointer position and the display position of the virtual object will be described.

<Display processing example 1>
FIG. 14 is a diagram illustrating a first example of display processing based on the relationship between the pointer position and the virtual object position. Referring to FIG. 14, in the first example, the CPU of processing apparatus 100 functioning as an image processing control unit has a virtual object (hereinafter also referred to as a virtual target object) within a first range defined in advance from the pointer position. ) Is included, a virtual object other than the virtual target object is selected as a virtual non-target object, and a virtual non-target object is selected according to the distance between the display position of the virtual target object and the pointer position. Image information set in advance for the non-virtual target object is converted so that the possible range is farther than the selectable range of the virtual target object.

  Here, it is assumed that the image information of the virtual object is a sphere centered on a specific point, and the range of the sphere in which the sphere is three-dimensionally displayed on the two-dimensional display 400 is a selectable range of the virtual object. It is prescribed that

  Specifically, the reduction of the size of the virtual non-target object corresponds to a modification that makes the selectable range of the virtual non-target object farther than the selectable range of the virtual target object. That is, as shown in FIG. 14A, when the distance between the position of the pointer 200 attached to the fingertip and one displayed virtual object becomes a predetermined distance, FIG. ), The virtual object is set as a virtual target object, and the other virtual objects are set as non-virtual target objects so that the displayed shape of the virtual non-target object is reduced.

  In order to perform this display process, the virtual object-pointer distance determination unit 106 stores a reference distance that defines the first range. Then, the virtual object-pointer distance determining unit 106 identifies the virtual target object by comparing with the reference distance.

  In addition, the virtual non-target object display data generation unit 109 stores in advance the degree of deformation of the virtual non-target object for each range of the distance D between the position of the pointer 200 and the virtual target object. As an example, a relationship is stored such that the display magnification of the non-virtual target object decreases as the distance D between the position of the pointer 200 and the virtual target object decreases. Then, display data is generated by updating display data defined for the virtual non-target object based on the correspondence.

  FIG. 15 is a flowchart illustrating a specific example of the operation for the first display process. Referring to FIG. 15, the CPU of processing apparatus 100 functioning as an image processing control unit in step S301 calculates a distance D between the pointer position and the display position of the virtual object. Details of the operation in step S301 have been described with reference to FIG.

  When the calculated distance D is smaller than the threshold value Dt, which is a reference distance that defines the first range (YES in step S303), that is, the virtual object to be processed within the first range from the pointer position. If there is an object display position, the CPU extracts a virtual object outside the first range as a virtual non-target object in step S305, and in step S307, the CPU determines the distance from the virtual target object calculated in step S301. Based on D, the display magnification of the virtual non-target object extracted in step S305 is determined based on the stored correspondence described above and reduced.

  By displaying this image on the display 400 with transparency and projecting it onto the real space visually recognized through the display 400, the display as shown in FIG. 14 is realized. Thereby, a virtual object within a predetermined range from the pointer position is set as a virtual target object that is highly likely to be specified, and other virtual objects are displayed as small objects as non-virtual target objects. Therefore, a user wearing the pointer 200 can easily grasp the distance from the pointer 200 of a plurality of displayed virtual objects, and can easily specify the virtual object closest to the pointer 200.

<Modification 1>
In the example of FIG. 14, the shape of the virtual non-target object is reduced by a predetermined magnification as a modification that makes the selectable range of the virtual non-target object farther than the selectable range of the virtual target object. However, other processing may be used. As another example, as shown in FIG. 16, for example, the shape of the virtual non-target object may be deformed into a vertically elongated shape, expanded, or irregularly changed.

  Alternatively, as another example, for example, as illustrated in FIG. 17, the virtual non-target object may be made transparent or the color may be changed. Furthermore, other changes of the non-virtual object include, for example, changing from a chromatic color to an achromatic color, or changing from a three-dimensional display to a two-dimensional display. In this case, it is assumed that the transparent part and the achromatic part of the display image of the non-virtual object are in a non-selectable range. That is, even if the portion is designated by the pointer 200, processing is performed so that the virtual non-target object is not selected.

<Modification 2>
In the above description, if the display position of the virtual object is within a predetermined range from the pointer position, the shape of the virtual object other than the virtual object is reduced. In addition, the display position of the virtual object (virtual target object) is within the first range, and the display positions of the plurality of virtual objects (non-virtual target) are within the second range wider than the first range from the pointer position. If there is, the shape of the virtual non-target object may be reduced. The same applies to the following display processing examples.

  By doing this, a plurality of virtual objects are displayed within the second range from the pointer position, and even if the virtual objects that the user intends to specify overlap and are difficult to specify, the Since a virtual object close to the pointer position, that is, a virtual object other than a virtual object that is highly likely to be designated by the user is reduced (or deformed) as a virtual non-target object, the user wearing the pointer 200 designates the virtual target object. It becomes easy to do.

<Modification 3>
In the above example, a virtual object within the first range from the pointer position is set as a virtual target object, and all other virtual objects are set as non-virtual target objects and the size is reduced. As another more preferable example, only the virtual object having the display position in the second range wider than the first range may be used as the non-virtual target object.

  In this case, the virtual non-object-to-pointer distance determination unit 107 stores a reference distance that defines the second range. Then, the non-virtual target object-pointer distance determination unit 107 identifies the virtual non-target object by comparing with the reference distance. In step S305, the CPU extracts a virtual object that is a virtual object outside the first range and that is a reference distance that defines the second range as a virtual non-target object.

  By doing so, the number of virtual objects to be deformed can be reduced, and the processing in the CPU can be facilitated.

<Modification 4>
In the above example, it is assumed that the display magnification for deforming the non-virtual target object according to the distance D between the pointer position and the display position of the virtual target object is stored. The same applies to other modified examples. However, as another example, a deformation ratio according to the distance between the virtual non-target object to be deformed and the pointer position is stored, and the virtual non-target object may be deformed according to the ratio. As an example, if a display magnification is stored such that the distance between the pointer position and the display position of the virtual non-target object decreases as the distance increases, the virtual non-target object far from the pointer position is stored. The smaller the shape. Therefore, the user wearing the pointer 200 can more easily grasp the distance from the pointer 200 of each virtual non-target object.

<Modification 5>
As another example of the deformation of the virtual non-target object, there is an example in which the degree of deformation is further changed based on the shooting condition of the camera 300. Specifically, the degree of deformation of a virtual non-target object that is transparently displayed on the two-dimensional display 400 by being displayed on the display 400 so as to overlap the virtual target object is higher than that of the non-virtual target object. For example, the virtual non-target object may be deformed so as to increase.

  18 and 19 are diagrams illustrating specific examples of display according to the fifth modification. In the example shown in FIG. 18, when the pointer 200 is positioned within a predetermined range from one of the virtual objects (FIG. 18A), the virtual object is set as a virtual target object, and the other virtual objects Is identified as a virtual non-target object, and among the virtual non-target objects, a virtual target object and an object projected onto the two-dimensional display 400 are smaller than other virtual non-target objects. (FIG. 18B). In the example shown in FIG. 19, when the pointer 200 is located within a predetermined range from one of the virtual objects (FIG. 19A), the virtual object is set as a virtual target object. After the virtual object is identified as a virtual non-target object, a virtual non-target object that is projected onto the virtual target object and the two-dimensional display 400 is more than a virtual non-target object. The color is changed so that the transparency becomes higher (FIG. 19B).

  In this case, in step S305, the CPU further performs a process of determining whether each of the virtual non-target objects is projected on the virtual target object. As this processing, for example, the CPU projects an image of each virtual object on a plane whose normal is the shooting direction from the camera 300 based on the display position and shape information of each virtual object. It is determined whether or not these overlap. As the determination, for example, the sum of the radius of the virtual target object projected on the plane and the radius of the non-virtual target object is compared with the distance between the centers on the plane. As a result, when the sum of the radii is larger than the center-to-center distance, the CPU determines that the virtual non-target object is projected onto the virtual target object by displaying it at a position based on the captured image from the camera 300. If this is not the case, it is determined that the images are not projected together. Then, among the non-virtual objects to be deformed, those that are determined to be projected in an overlapping manner are applied by increasing the stored degree of deformation at a predetermined rate in the deformation.

  In this case, when it is detected that the position of the camera 300 has changed, the degree of deformation may be maintained for a predetermined time.

  By performing such processing, even when the virtual non-target object is displayed on the two-dimensional display 400 so as to overlap the virtual target object, the user wearing the pointer 200 has the pointer 200 of the virtual non-target object. It becomes easier to grasp the distance from the camera more visually.

<Display processing example 2>
FIG. 20 is a diagram illustrating a second example of display processing based on the relationship between the pointer position and the position of the virtual object. Referring to FIG. 20, in the second example, when the pointer position approaches the display position of the virtual object, the CPU of processing apparatus 100 functioning as the image processing control unit sets the virtual object as a virtual target object. A virtual object other than the virtual target object is set as a non-virtual target object, and the display position is moved so that the virtual non-target object moves away from the virtual target object according to the distance between the display position of the virtual target object and the pointer position. Let That is, as shown in FIG. 20 (A), when it is detected that the pointer 200 attached to the fingertip is approaching the display position of one displayed virtual object, it is shown in FIG. 20 (B). As described above, the virtual object is set as the virtual target object, and the display position of the non-virtual target object within the predetermined range from the virtual target object is moved away from the virtual target object.

  In order to perform this display process, the virtual object-pointer distance determination unit 106 includes a memory (not shown) for temporarily storing the calculated distance between the virtual object and the pointer position. In addition, a reference distance is stored as a threshold value for determining a virtual object to be processed. Then, the virtual object-pointer distance determining unit 106 identifies the virtual target object by comparing with the reference distance.

  Further, the virtual non-object display data generation unit 109 stores in advance the movement distance of the virtual non-target object to be processed for each range of the distance D between the position of the pointer 200 and the virtual target object. . As an example, a relationship is stored such that the moving distance of the non-virtual target object increases as the distance D between the position of the pointer 200 and the virtual target object decreases. Then, display data is generated by updating the display position defined for the virtual non-target object based on the correspondence.

  FIG. 21 is a flowchart illustrating a specific example of the operation for the second display process. Referring to FIG. 21, the CPU of processing apparatus 100 functioning as an image processing control unit in step S401 calculates a distance D between the pointer position and the display position of the virtual object. Details of the operation in step S401 have been described with reference to FIG.

  In step S403, the CPU extracts a virtual object having a display position within a predetermined range from the pointer position as a virtual object to be processed. In step S405, the CPU specifies a virtual target object from the extracted virtual objects to be processed. As an example of the process in step S405, the CPU determines, for one virtual object, the distance D between the pointer position calculated by the immediately preceding process and the display position of the virtual object, and the distance D calculated in step S401. If the difference is larger than the threshold value and the direction from the pointer position to the display position of the virtual object is the same as or substantially the same as the direction in the immediately preceding process, the virtual object Is identified as a virtual target object. In other words, when the position of the pointer 200 is advanced by a predetermined amount or more toward the same virtual object at a predetermined speed, it is determined that the pointer 200 is moving toward the virtual object, and the virtual object is determined as a virtual object. An object.

  The CPU identifies virtual objects other than the virtual target object identified by such a method among the virtual objects to be processed extracted in step S403 as non-virtual target objects, and in step S407, these virtual non-target objects. Execute the process of separating from the virtual target object. Specifically, based on the distance D between the pointer position calculated in step S401 and the display position of the virtual target object, the movement distance is determined based on the stored correspondence relationship for the virtual non-target object to be processed. The display position defined for each virtual non-target object is updated in consideration of the movement distance.

  The movement here is not limited to movement in a specific direction and distance. For example, as shown in FIG. 20, it may be moved upward by a predetermined distance, or as shown in FIG. 22, it may be moved downward by a predetermined distance (dropped). As another example, it may be moved by a predetermined distance in the direction of a straight line connecting the center of the non-virtual target object and the center of the virtual target object and away from the virtual target object.

  By displaying this image on the display 400 with transparency and projecting it to the real space visually recognized through the display 400, the display as shown in FIG. 20 or FIG. 22 is realized. As a result, the virtual object to which the pointer 200 is pointing is set as a virtual target object that is highly likely to be designated, and a virtual non-target object within a predetermined range from the virtual target object is displayed away from the virtual target object. Therefore, the user wearing the pointer 200 can easily visually grasp the distance from the pointer 200 of a plurality of displayed virtual objects, and can easily specify the virtual object to which the pointer 200 is directed.

<Modification 1>
In the second display example, similar to the first and fourth modified examples, there may be modified examples similar to the modified examples 4 and 5. That is, as in Modification 4 above, the virtual non-target object may be moved by a distance corresponding to the distance between the display position of the virtual non-target object and the pointer position. Further, as in the above-described modification example 5, the movement amount of a virtual non-target object displayed on the two-dimensional display 400 so as to overlap with the virtual target object may be larger than that of the non-virtual target object.

  As a specific example, as shown in FIG. 23, when moving a non-virtual target object upward by a predetermined distance so as to be separated from the virtual target object, a virtual target projected onto the virtual object on the two-dimensional display 400 is superimposed. For the outside object (the second and fourth virtual objects from the front in the example of FIG. 23A), the non-overlapping virtual non-target objects (the foremost and backmost in the example of FIG. 23A) The movement amount may be larger than that of the virtual object. As another example, as shown in FIG. 24, the virtual non-target object is moved downward by a predetermined distance so as to be separated from the virtual target object, and further projected onto the two-dimensional display 400 so as to overlap the virtual object. Virtual non-target objects (second and fourth virtual objects from the front in the example of FIG. 24A), non-overlapping virtual non-target objects (the frontmost in the example of FIG. 24A), and You may change a color so that transparency may become higher than a backmost virtual object.

  In this case, as a process for determining whether or not the images are projected in an overlapping manner, a process similar to the process described above can be adopted as an example. Further, the determination as to whether or not the images are overlapped may be performed based on the display position of the virtual non-target object after being moved according to the second processing example.

  By performing such processing, a virtual non-target object within a predetermined range from the virtual target object is displayed away from the virtual target object, and the virtual non-target object is displayed as a virtual target object on the two-dimensional display 400. Even in the case of overlapping projection, the user wearing the pointer 200 can more easily grasp the distance of the virtual non-target object from the pointer 200 more visually. Therefore, it becomes easier to specify the virtual object to which the pointer 200 is directed.

Furthermore, the first display example and the second display example may be combined.
Furthermore, a program for causing the processing device 100 to execute the above-described display processing can be provided. Such a program is stored in a computer-readable recording medium such as a flexible disk attached to the computer, a CD-ROM (Compact Disk-Read Only Memory), a ROM (Read Only Memory), a RAM (Random Access Memory), and a memory card. And can be provided as a program product. Alternatively, the program can be provided by being recorded on a recording medium such as a hard disk built in the computer. A program can also be provided by downloading via a network.

  The program according to the present invention is a program module that is provided as a part of a computer operating system (OS) and calls necessary modules in a predetermined arrangement at a predetermined timing to execute processing. Also good. In that case, the program itself does not include the module, and the process is executed in cooperation with the OS. A program that does not include such a module can also be included in the program according to the present invention.

  The program according to the present invention may be provided by being incorporated in a part of another program. Even in this case, the program itself does not include the module included in the other program, and the process is executed in cooperation with the other program. Such a program incorporated in another program can also be included in the program according to the present invention.

  The provided program product is installed in a program storage unit such as a hard disk and executed. The program product includes the program itself and a recording medium on which the program is recorded.

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

  100 processing apparatus, 101A camera image input unit, 101B pointer data input unit, 101C virtual object information input unit, 102 pointer object extraction unit, 103 image recognition processing unit, 104 pointer shape storage unit, 105 pointer display data generation unit, 106 virtual Object-pointer distance determination unit, 107 Virtual target object-pointer distance determination unit, 108 Virtual target object coordinate storage unit, 109 Virtual target object display data generation unit, 110 Virtual target object shape storage unit, 111 Display Processing unit, 200 pointer, 201 position detection sensor, 202 acceleration sensor, 203 direction detection sensor, 204 CPU, 205 communication I / F, 300 camera, 400, 401 display, 402 CPU, 4 03 communication I / F, 500 server, 600 portable terminal, 1021 pointer extraction unit, 1022 virtual object extraction unit, 1023 virtual non-target object extraction unit, 1031 pointer coordinate detection unit, 1032 virtual object display area detection unit, 1033 virtual non-target Object display area detector.

Claims (13)

A display system for displaying an image of a virtual object,
A camera,
Display,
A display processing device,
The display processing device includes:
As information about the virtual object, image information for displaying the virtual object in three dimensions, a display position of the virtual object represented in a three-dimensional coordinate system set for the captured space, An acquisition means for acquiring from the storage device information defining an action to be executed when the virtual object is selected;
A three-dimensional coordinate system is set for an image captured by the camera, a display image for displaying an image of the virtual object at the display position defined for the virtual object is generated, and the display Processing means for executing processing to be displayed;
Detecting means for detecting a position in the three-dimensional coordinate system designated by a user,
The processing means identifies one virtual object of the virtual objects as a virtual target object based on a relationship between the display position specified for the virtual object in the captured image and the designated position, and the virtual object A display system that executes a process of changing an image of a virtual non-target object by setting a virtual object other than the virtual target object among the objects as a virtual non-target object.   The changing process includes a process of reducing an image of the virtual non-target object, a process of changing a shape of the image of the virtual non-target object, a process of changing a color of the image of the virtual non-target object, and the virtual non-target object. The display system according to claim 1, wherein the display system is at least one of a process of changing a transparency of an image of an object and a process of changing a display position of an image of the virtual non-target object.   The processing means treats a virtual object whose distance between the display position specified for the virtual object in the captured image and the specified position is smaller than a first reference distance specified in advance as the virtual target. The display system according to claim 1, wherein the display system identifies the object.   The display system according to claim 3, wherein the processing unit reduces the image of the virtual non-target object according to a distance between the designated position and the virtual target object.   The processing means identifies, as the virtual target object, a virtual object whose display position corresponds to a direction in which the designated position is directed based on a change amount of the position designated by the user. Display system as described in.   The processing means changes the display position of the non-virtual target object in a direction away from the virtual target object according to a distance between the designated position and the virtual target object. Display system described.   The processing means, based on the positional relationship between the position of the camera and the virtual object, changes in the image of the virtual non-target object that is displayed overlapping the virtual target object. The display system according to claim 1, wherein the display system is changed more greatly than a change in the image of the object.   When the change in the position of the camera is detected, the processing unit is configured to display an image of the virtual non-target object specified to be displayed so as to overlap the virtual target object according to the position of the camera before the change. The display system of claim 7, wherein the change is maintained after the camera position is moved.   The processing means is a virtual object in which a distance between the display position specified for the virtual object in the captured image and the designated position is smaller than a second reference distance that is larger than the first reference distance. The display system according to claim 1, wherein a virtual object other than the virtual object identified as the virtual target object is identified as the non-virtual target object.   The said display permeate | transmits the light from the surface on the opposite side to a display surface, and displays the said display image produced | generated by the said display process means on the said display surface with transparency. A display system according to any one of the above. A display processing device for displaying an image of a virtual object on a display,
Input means for receiving input of the captured image;
As information about the virtual object, image information for displaying the virtual object in three dimensions, a display position of the virtual object represented in a three-dimensional coordinate system set for the captured space, An acquisition means for acquiring from the storage device information defining an action to be executed when the virtual object is selected;
A three-dimensional coordinate system is set for the photographed image, and a display image for displaying the image of the virtual object at the display position defined for the virtual object is generated and displayed on the display. Processing means for executing processing;
Detecting means for detecting a position in the three-dimensional coordinate system designated by a user,
The processing means identifies one virtual object of the virtual objects as a virtual target object based on a relationship between the display position specified for the virtual object in the captured image and the designated position, and the virtual object A display processing apparatus that executes a process of changing an image of a virtual non-target object using a virtual object other than the virtual target object as a virtual non-target object. In a display processing apparatus, a method for displaying an image of a virtual object on a display,
A step of receiving input of a photographed image from the camera;
Setting the three-dimensional coordinate system for the captured image;
Detecting a position in the three-dimensional coordinate system specified by a user;
As information about the virtual object, image information for displaying the virtual object in three dimensions, a display position of the virtual object represented in a three-dimensional coordinate system set for the captured space, Obtaining from the storage device information defining an action to be executed by selecting the virtual object;
One virtual object of the virtual objects is identified as a virtual target object based on the relationship between the display position specified for the virtual object in the captured image and the specified position, and the virtual object And a step of changing an image of the virtual non-target object using a virtual object other than the virtual target object as a virtual non-target object. A program for causing a display processing device to execute a process of displaying an image of a virtual object on a display,
A step of receiving input of a photographed image from the camera;
Setting the three-dimensional coordinate system for the captured image;
Detecting a position in the three-dimensional coordinate system specified by a user;
As information about the virtual object, image information for displaying the virtual object in three dimensions, a display position of the virtual object represented in a three-dimensional coordinate system set for the captured space, Obtaining from the storage device information defining an action to be executed by selecting the virtual object;
One virtual object of the virtual objects is identified as a virtual target object based on the relationship between the display position specified for the virtual object in the captured image and the specified position, and the virtual object A display program that causes the display processing device to execute a step of changing an image of the virtual non-target object using a virtual object other than the virtual target object as a virtual non-target object.

Images