JP2013105330A - Control processing program, image display device and image display method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a control processing program, an image display device, and an image display method for starting the operation of an extended actual object according to a position of an operator's finger.SOLUTION: A control processing program of the present invention has the steps of: obtaining a background video; obtaining display positional information of an image forming apparatus to be installed; generating a device image of the image forming apparatus, on the basis of the display positional information and size information of the image forming apparatus; obtaining positional information of a hand or a finger; generating a pointer image that illustrates a position of the hand or the finger, on the basis of the positional information of the hand or the finger; generating an output video by composing the background video, the device image, and the pointer image; executing a predetermined operation of a simulation model that simulates the operation of the image forming apparatus, on the basis of a position to install the image forming apparatus and the positional information of the hand or the finger; and displaying the output video.

Description

  The present invention relates to a control processing program, an image display device, and an image display method.

  In recent years, a virtual object image (hereinafter referred to as an augmented reality object) is synthesized on a real world image and displayed on a display, so that the augmented reality object exists as if it exists in the real world. A technology that uses augmented reality that viewers can see is drawing attention.

  For example, Patent Document 1 below discloses a technique in which a virtual furniture image is synthesized with an image of an actual room so that the furniture appears as if it exists in the actual room.

  In addition, a technique has been developed in which an augmented reality object starts some operation when an operator acts on the augmented reality object, instead of simply synthesizing and displaying the augmented reality object on a video in the real world. For example, Patent Document 2 below discloses a technique for moving a virtual character that is synthesized and displayed on a video image of the real world in accordance with a voice uttered by an operator.

  However, in the image display system of Patent Literature 2, although the operation of the augmented reality object can be started in accordance with the sound emitted by the operator, the augmented reality object is determined in accordance with the operation of the operator, for example, the position of the operator's finger. The operation cannot be started.

Special table 2010-541044 gazette Japanese Patent No. 4696184

  The present invention has been made to solve the above-described problems. Accordingly, an object of the present invention is to provide a control processing program, an image display device, and an image display method capable of starting an operation of an augmented reality object according to the position of an operator's finger.

  The above object of the present invention is achieved by the following.

  (1) Based on the procedure (a) for acquiring the background image, the procedure (b) for acquiring the display position information of the image forming apparatus to be installed, and the display position information and information on the size of the image forming apparatus, The position of the hand or finger is indicated based on the procedure (c) for generating the device image of the image forming apparatus, the procedure (d) for acquiring the position information of the hand or finger, and the position information of the hand or finger. A procedure (e) for generating a pointer image, a procedure (f) for synthesizing the background image, the device image, and the pointer image to generate an output image, a position where the image forming apparatus is installed, and the hand or A computer having a procedure (g) for executing a predetermined operation of a simulation model simulating the operation of the image forming apparatus based on the finger position information and a procedure (h) for displaying the output video Control processing program to be executed by the.

  (2) The control processing program according to (1), wherein the background video is acquired by an imaging unit imaging a place where the image forming apparatus is installed.

  (3) The control processing program according to (2), wherein in the step (b), a marker image is extracted from the background video to acquire the display position information.

  (4) A computer-readable recording medium on which the image processing program according to any one of (1) to (3) is recorded.

  (5) Based on background video acquisition means for acquiring a background video, display position information acquisition means for acquiring display position information of an image forming apparatus to be installed, and information on the display position information and the size of the image forming apparatus. A device image generation unit that generates a device image of the image forming device, a finger position information acquisition unit that acquires position information of a hand or a finger, and the position of the hand or finger based on the position information of the hand or the finger A pointer image generating means for generating a pointer image indicating image, an image synthesizing means for generating an output video by synthesizing the background video, the apparatus image, and the pointer image, a position where the image forming apparatus is installed, and the hand Alternatively, based on the finger position information, model execution means for executing a predetermined operation of a simulation model that simulates the operation of the image forming apparatus, and the output projection The image display apparatus characterized by having a display unit for displaying.

  (6) In the above (5), the background video acquisition unit includes an imaging unit, and the background video is acquired by imaging the location where the image forming apparatus is installed by the imaging unit. The image display device described.

  (7) The image display device according to (6), wherein the display position information acquisition unit acquires the display position information by extracting a marker image from the background video.

  (8) Based on the step of acquiring a background image, the step of acquiring display position information of the image forming apparatus to be installed, and the display position information and information relating to the size of the image forming apparatus, the apparatus of the image forming apparatus Generating an image; obtaining hand or finger position information; generating a pointer image indicating the hand or finger position based on the hand or finger position information; and the background video; The operation of the image forming apparatus is simulated based on the step of synthesizing the apparatus image and the pointer image to generate an output video, the position where the image forming apparatus is installed, and the position information of the hand or finger. An image display method comprising: performing a predetermined operation of a simulation model; and displaying the output video.

  According to the present invention, based on the position of the operator's hand or finger, a predetermined operation of the simulation model of the augmented reality object displayed on the display is executed, and the operation of the augmented reality object is started. Therefore, the operator can experience as if the image forming apparatus is actually operated, and thus can examine in advance in a state close to the case where the image forming apparatus is actually installed.

1 is a schematic perspective view showing an overview of an image display device according to a first embodiment of the present invention. 1 is a schematic block configuration diagram of an image display device according to a first embodiment of the present invention. It is a flowchart for demonstrating roughly the process sequence of the control processing program in the 1st Embodiment of this invention. FIG. 6 is a diagram illustrating a background image displayed on a display when “end request” is selected as an input operation in the first embodiment of the present invention. In the 1st Embodiment of this invention, it is a figure which illustrates the background image | video displayed on a display when the marker is not arrange | positioned. In the 1st Embodiment of this invention, it is a figure which illustrates the image | video displayed on a display when an operator's finger is not in the visual field of a camera. In the 1st Embodiment of this invention, it is a figure which illustrates the image | video displayed on a display when a background image | video, an apparatus image, and a pointer image are synthesize | combined. It is a flowchart explaining step S103 shown in FIG. It is a flowchart explaining step S105 shown in FIG. It is a flowchart explaining step S106 shown in FIG. It is a schematic diagram for demonstrating the method of calculating an operation point. It is a flowchart explaining step S108 shown in FIG. It is a flowchart explaining step S503 shown in FIG. It is a figure for illustrating a pointer image. It is a figure for illustrating a pointer image. It is a figure for illustrating a pointer image. It is a figure for illustrating a pointer image. It is a figure for illustrating a pointer image. It is a flowchart explaining step S110 shown in FIG. It is a flowchart explaining step S702 shown in FIG. It is a figure which illustrates the image | video displayed on a display when the simulation of "printing operation" is performed. It is a schematic block block diagram of the image display apparatus of the 2nd Embodiment of this invention.

(First embodiment)
Hereinafter, a first embodiment of the present invention will be described with reference to FIG. FIG. 1 is a schematic perspective view showing an overview of an image display apparatus according to a first embodiment of the present invention, and FIG. 2 is a schematic block diagram of the image display apparatus according to the first embodiment of the present invention. The image display device according to the present embodiment executes a predetermined operation of a simulation model of an augmented reality object displayed on a display based on the position of an operator's hand or finger, and starts the operation of the augmented reality object. And Note that the image display apparatus of this embodiment can be suitably used when considering the installation of an image forming apparatus such as a copying machine, a printer, a facsimile machine, or a multifunction machine.

  As shown in FIG. 1, the image display device 100 according to the present embodiment is a portable tablet terminal, and an operator of the image display device 100 (hereinafter simply referred to as an operator) can freely carry around the room. It is configured to be able to. In the present embodiment, the operator considers using the image display apparatus 100 to install the image forming apparatus and an optional apparatus for the image forming apparatus (hereinafter referred to as an image forming apparatus or the like) in a room with the window W. A case will be described.

  Note that the image display device of the present embodiment is not a tablet type as described above, but may be a goggle type (glasses type) that displays an augmented reality object superimposed on a background. In this case, since the image display device is attached to the face of the operator, it is not necessary to carry it.

  The image display device 100 includes a camera 110, a sensor 120, a display 130, a touch panel 140, a speaker 150, a control unit 160 (see FIG. 2), and a housing 170.

  The camera 110 captures an image of a place where an image forming apparatus or the like is installed as a background image acquisition unit and a display position information acquisition unit, and acquires a background image. The operator adjusts the orientation of the housing 170 so that the place where the image forming apparatus or the like is installed falls within the field of view of the camera 110.

  The camera 110 is attached to one surface of the housing 170, captures a video of a place where the image forming apparatus or the like is installed at a predetermined cycle, and transmits it to the control unit 160. At this time, the image of the marker 200 is also captured as part of the image of the place where the image forming apparatus or the like is installed, and is transmitted to the control unit 160. The marker 200 is a sheet-like medium having characters, symbols, graphics, etc. written on the surface thereof, and includes information for identifying the model of the image forming apparatus to be installed. The operator arranges the marker 200 in advance at the place where the image forming apparatus is installed. Note that the camera 110 can also be incorporated in the housing 170.

  The sensor 120 detects an operator's hand or finger (hereinafter referred to as a finger) as finger position information acquisition means. The sensor 120 includes, for example, an infrared sensor, and is attached to one surface of the housing 170 along with the camera 110. The sensor 120 detects infrared radiation from an object at a short distance from the sensor 120 with an infrared sensor. When the operator's finger enters the detection range of the infrared sensor, the sensor 120 detects the infrared radiation of the operator's finger and transmits it to the controller 160 as image data. The sensor 120 can also be built in the housing 170.

  As will be described later, in the image display apparatus 100 according to the present embodiment, a predetermined operation of the simulation model of the augmented reality object displayed on the display 130 is executed based on the position of the finger of the operator. The position of the operator's finger is acquired by the sensor 120 detecting the operator's finger and processing the detection result by the control unit 160.

  On the other hand, since the sensor 120 detects an object with an infrared sensor, the control unit 160 cannot determine whether the finger belongs to an operator or another person unless special determination processing is performed. . Below, for convenience of explanation, a case where the operator's fingers are used will be described as an example. However, the fingers may include human fingers other than the operator. In addition, an indicator stick or the like can be used instead of human fingers.

  The display 130 displays output video, various setting items, messages, and the like as display means. Specifically, the display 130 displays an output video in which the augmented reality object generated by the control unit 160 and the background video are combined. The display 130 displays a setting screen for setting setting items such as various modes and functions of the image display device 100, the model of the optional device, and the display method. Further, the display 130 displays a message, a warning message, or the like output from the control unit 160 when setting various setting items.

  The image display device 100 may further include an output terminal (not shown) that outputs an output video from the control unit 160 to the outside of the image display device 100. The front image display device 100 transmits an output video to a large screen display, projector, or the like through the output terminal, so that the same video as the video displayed on the display 130 can be simultaneously viewed on the large screen display, projector, or the like. .

  Touch panel 140 accepts an input operation from an operator. In the present embodiment, the operator can set setting items such as various modes, functions, optional device models, and display methods via the touch panel 140. The touch panel 140 is disposed on the display 130, and the setting item is selected when the operator's finger touches a position corresponding to the display on the display 130, and information on the selected setting item is displayed on the control unit 160. Is transmitted to. In the case of a goggle-type image display device, various modes, functions, optional device models, display methods, and the like can be set through pushbutton switches and the like.

  The speaker 150 outputs printing sound, sound effect, warning sound, voice guidance, and the like. The speaker 150 is generated when an operation simulation of the image forming apparatus or the like is performed by using, as a print sound, a sound recording or imitating a sound generated by an actual image forming apparatus or the like. The speaker 150 outputs a sound effect when selecting a setting item on the setting screen of the image display apparatus 100, for example. Further, when an operational trouble occurs, a warning sound or voice guidance is output.

  The control unit 160 controls the camera 110, the sensor 120, the display 130, the touch panel 140, and the speaker 150. As illustrated in FIG. 2, the control unit 160 includes a CPU (Central Processing Unit) unit 161, a ROM (Read Only Memory) unit 162, a RAM (Random Access Memory) unit 163, and a large capacity storage unit 164. The control unit 160 functions as display position information acquisition means, device image generation means, finger position information acquisition means, pointer image generation means, image composition means, and model execution means.

  The CPU unit 161 is connected to the ROM unit 162, the RAM unit 163, and the mass storage unit 164 through a bus. The CPU unit 161 is connected to the camera 110, the sensor 120, the display 130, the touch panel 140, and the speaker 150 through various dedicated interfaces (not shown).

  The ROM unit 162 stores a control processing program, 3D data, and content data. The control processing program is a software program for controlling the operation of the image display apparatus 100 and executing various arithmetic processes, and is sequentially executed by the CPU unit 161.

  The 3D data includes three-dimensional image data representing the outer shape of the image forming apparatus to be installed and the optional apparatus. Examples of the optional device include a saddle stitching staple device and a staple binding device. In this embodiment, 3D image data corresponding to the model of the image forming apparatus or the like is prepared as 3D data. The 3D data is stored in the ROM unit 162 in a general data format used in computer graphics, and the data format is not particularly limited.

  The content data includes substantial data regarding the image forming apparatus and the like. Specifically, the content data includes information regarding the configuration of the image forming apparatus, information regarding the size and position of each unit of the image forming apparatus, marker image information, a simulation model, and the like.

  The configuration of the image forming apparatus or the like differs depending on the model. The content data includes information such as the constituent elements of each model and the relationship between the constituent elements. In addition to the size of each housing of the image forming apparatus and the like, the content data includes the size of each constituent element such as an operation panel of the image forming apparatus, a rear door, and a paper discharge tray. And location.

  The marker image information is information included in the marker image. The marker image information is uniquely associated with the model of the image forming apparatus. Therefore, by extracting the image of the marker 200 from the background video and acquiring the marker image information, the model of the image forming apparatus to be installed can be specified.

  The simulation model is an operation simulation model for simulating the operation of the image forming apparatus or the like. After specifying the model of the image forming apparatus or the like to be installed, the CPU 161 runs the simulation model and starts an operation simulation of the image forming apparatus or the like.

  The RAM unit 163 stores a background video, an AR (Augmented Reality) object image, display state information, internal state information, and the like. The background video is a background video captured by the camera 110, acquired at a predetermined cycle, and stored in the RAM unit 163. In the present embodiment, the CPU 161 extracts the image of the marker 200 included in the background video, obtains display position information for installing the image forming apparatus, and stores it in the RAM 163.

  The AR object image includes an image of an image forming apparatus or the like as an augmented reality object (hereinafter referred to as a device image) and a pointer image of a finger as an augmented reality object.

  The apparatus image is obtained by converting 3D data into a two-dimensional image to be displayed on the display 130 in accordance with the size and orientation of the image forming apparatus to be installed. In the present embodiment, the CPU unit 161 forms an image from 3D data based on display position information of the image forming apparatus, information on the size of the image forming apparatus and the like stored in the ROM unit 162, and display state information. A device image of the device or the like is generated and stored in the RAM unit 163.

  The pointer image indicates the position of the operator's finger on the display 130. The pointer image is selected from predetermined images such as “arrow” and “finger”, for example, and is generated so as to be superimposed on the position of the image 10 of the operator's finger in the background video. It is also possible to set so that the pointer image is not displayed. The CPU unit 161 generates a pointer image based on the finger position information and stores the pointer image in the RAM unit 163. The CPU 161 calculates the finger position information based on the image data from the sensor 120. A specific method for acquiring finger position information and a specific method for generating a pointer image will be described later.

  The display state information is information on how the device image is displayed on the display 130. The display state information includes information regarding an option configuration that is currently displayed, information regarding a display method, and the like. As a display method, for example, there are superposition and transmission of device images. The display state information is updated when the option configuration or display method currently being displayed is changed. Information relating to the option configuration and information relating to the display method can be appropriately set by the operator via the touch panel 140.

  The internal state information is information related to the internal state of the simulation model. In the present embodiment, the internal state is roughly divided into a “normal state” and a “failure state”. When the operation simulation is started, the simulation model is in a “normal state”.

  When the internal state is “normal state”, the simulation model can execute an operation such as “printing operation”. On the other hand, in the “failure state”, the simulation model cannot execute an operation such as “printing operation”. On the other hand, there are operations that can be executed only when the internal state is a “failure state”. The internal state is “failed state”, for example, when the rear door is open.

  The initial value of the internal state information at the start of the operation simulation is stored in the content data of the ROM unit 162. As the operation simulation proceeds and the internal state changes, the internal state information is updated. When the internal state is “normal state” and the simulation model is printing, the internal state information includes information indicating “normal state” and “printing”.

  The CPU 161 refers to the internal state information of the simulation model, and executes a predetermined operation of the simulation model that simulates the operation of the image forming apparatus based on the position where the image forming apparatus is installed and the position information of the fingers. Specific operations of the simulation model will be described later.

  The large capacity storage unit 164 includes, for example, a hard disk drive, a flash memory, and the like. The large-capacity storage unit 164 includes moving image data or image data such as a moving image to be displayed when an operation simulation is executed, a message moving image to be displayed when setting items are set, printing sound, sound effect, warning sound, voice guidance, and the like. Audio data is stored.

  Hereinafter, the processing procedure of the control processing program in the present embodiment will be schematically described with reference to FIGS. FIG. 3 is a flowchart for schematically explaining the processing procedure of the control processing program in the first embodiment of the present invention. FIG. 4 is a diagram illustrating a background image displayed on the display when “end request” is selected as the input operation in the first embodiment of the present invention. FIG. 5 is a diagram illustrating a background image displayed on the display when no marker is arranged in the first embodiment of the present invention, and FIG. 6 is a diagram illustrating the background image displayed in the first embodiment of the present invention. It is a figure which illustrates the image | video displayed on a display when an operator's finger is not in the visual field of a camera. FIG. 7 is a diagram illustrating a video displayed on the display when the background video, the device image, and the pointer image are combined in the first embodiment of the present invention.

  First, a background video is acquired (step S101). The camera 110 captures a place where the image forming apparatus is installed indoors, and stores the captured background video in the RAM unit 163. When the marker 200 is arranged in advance at the place where the image forming apparatus is installed, the background video including the marker 200 is stored in the RAM unit 163. When the operator inserts a finger into the field of view of the camera 110, the background video includes the image 10 of the operator's finger.

  Next, an input operation is accepted (step S102). The control unit 160 receives an operator's input operation via the touch panel 140. The display 130 displays a selection screen of “end request”, “stop”, and “start”. When “end request” is selected as the input operation (step S102: end request), the process ends. When “stop” is selected as the input operation (step S102: stop), the process proceeds to step S111. As shown in FIG. 4, the control unit 160 outputs a background video as an output video, and the display 130 displays the output video. And it returns to step S101 again and continues a process.

  On the other hand, when “start” is selected as the input operation (step S102: start), display position information of the image forming apparatus to be installed is acquired (step S103). The control unit 160 extracts the image of the marker 200 from the background video, and acquires display position information of the image forming apparatus to be installed.

  Next, it is determined whether display position information has been acquired (step S104). When the display position information is not acquired (step S104: NO), the process proceeds to step S111. As shown in FIG. 5, the control unit 160 outputs a background video as an output video, and the display 130 displays the output video. And it returns to step S101 again and continues a process.

  On the other hand, when the display position information is acquired (step S104: YES), an apparatus image such as an image forming apparatus is generated (step S105). The control unit 160 generates a device image of the image forming apparatus or the like from the 3D data based on the display position information, the information regarding the size of the image forming apparatus or the like stored in the ROM unit 162, and the display state information. The generated device image of the image forming apparatus or the like is stored in the RAM unit 163.

  Next, the finger position information is acquired (step S106). The sensor 120 detects infrared radiation from a finger and transmits the detection result to the control unit 160 as image data. The controller 160 calculates finger position information based on the image data.

  Next, it is determined whether or not the finger position information has been acquired (step S107). When the finger position information is not acquired (step S107: NO), the process proceeds to step S111. As shown in FIG. 6, the control unit 160 outputs an output video obtained by synthesizing the device image 20 with the background video as an output video, and the display 130 displays the output video. And it returns to step S101 again and continues a process.

  On the other hand, if the finger position information has been acquired (step S107: YES), a pointer image is generated (step S108). The control unit 160 generates a pointer image having a predetermined shape based on the internal state information, the finger position information, and the like. The pointer image is generated based on the finger position information so as to be superimposed on the position of the finger image 10 of the operator of the background video. The generated pointer image is stored in the RAM unit 163.

  Next, the background video, the device image 20 and the pointer image are synthesized (step S109). The control unit 160 combines the background video, device image, and pointer image stored in the RAM unit 163.

  Next, a predetermined operation of the simulation model is executed (step S110). The control unit 160 calculates a separation distance between the position where the image forming apparatus is installed and the position of the operator's finger based on the position where the image forming apparatus is installed and the position information of the fingers. This operation is executed according to the size of the separation distance.

  Next, the output video is displayed (step S111). As shown in FIG. 7, the control unit 160 outputs an output video obtained by combining the background video, the device image 20, and the pointer image 30, and the display 130 displays the output video. And it returns to step S101 again and continues a process.

  The processing procedure of the control processing program in the present embodiment has been schematically described above. Below, with reference to FIGS. 8-17, the process sequence of the control processing program of this embodiment shown in FIG. 3 is demonstrated in detail.

<Acquisition of Display Position Information of Image Forming Apparatus to be Installed (Step S103)>
FIG. 8 is a flowchart for explaining step S103 shown in FIG. As shown in FIG. 8, first, an image of the marker 200 is extracted from the background video (step S201). The CPU unit 161 extracts an image of the marker 200 from the background video stored in the RAM unit 163.

  Next, it is determined whether an image of the marker 200 has been extracted (step S202). When the image of the marker 200 is extracted (step S202: YES), the display position information of the image forming apparatus to be installed is acquired (step S203). Specifically, the CPU 161 calculates the display position coordinates and orientation of the image forming apparatus to be installed as display position information from the size and orientation of the marker 200 image extracted from the background video. Here, the orientation of the image forming apparatus is represented by the angle of the image forming apparatus with respect to the direction of a straight line connecting the image display apparatus 100 and the marker 200.

  Note that a technique for calculating the display position coordinates and orientation of the augmented reality object from the size and orientation of the marker image extracted from the background video is a known technique, and thus description thereof is omitted here.

  In the present embodiment, the position of the marker 200 corresponds to the position of the central portion of the bottom surface of the image forming apparatus to be installed. However, the position of the marker 200 may correspond to the position of the end of the bottom surface of the image forming apparatus to be installed.

  Further, the CPU 161 acquires marker image information and the distance between the camera 110 and the marker 200 from the extracted image of the marker 200.

  On the other hand, when the image of the marker 200 is not detected (step S202: NO), the process proceeds to step S104.

<Generation of Device Image of Image Forming Apparatus (Step S105)>
FIG. 9 is a flowchart for explaining step S105 shown in FIG. As shown in FIG. 9, first, display state information is acquired (step S301). As described above, the display state information is information on how the device image 20 is displayed on the display 130.

  Next, 3D data of the image forming apparatus or the like is acquired (step S302). Specifically, the 3D data of the ROM unit 162 is acquired based on the model of the image forming apparatus and the display state information.

  Next, an apparatus image such as an image forming apparatus is generated (step S303). Specifically, the apparatus image 20 such as an image forming apparatus is generated based on the 3D data acquired in step S302, the display position information, and information regarding the size of the image forming apparatus or the like. For example, a model X of a copying machine is generated as an image forming apparatus, and a model Y of a saddle stitching staple apparatus is generated as an optional device in a transparent display.

<Acquisition of Finger Position Information (Step S106)>
FIG. 10 is a flowchart for explaining step S106 shown in FIG. 3, and FIG. 11 is a schematic diagram for explaining a method for calculating an operation point.

  First, as shown in FIG. 10, an object at a short distance from the sensor is detected (step S401). When the operator inserts a finger or the like into the detection range of the sensor 120, the sensor 120 detects an object at a short distance from the sensor 120 and transmits it to the control unit 160 as image data.

  Next, the finger position information is acquired (step S402). The position information of the operator's fingers is acquired from the image data from the sensor 120.

  Specifically, the control unit 160 measures a separation distance between the camera 110 and the operator's finger, and extracts a finger region from the image data based on the separation distance. Note that a technique for extracting an image area whose distance from the sensor 120 is within a certain range and distinguishing it from other areas is publicly known, and thus description thereof is omitted here.

As shown in FIG. 11, in the present embodiment, an operation point is calculated from the extracted finger region 40 and used as finger position information. The finger region 40 is extracted in such a manner as to extend from any of the upper, lower, left, and right sides of a background image region (hereinafter referred to as a viewing region 50) captured by the camera 110. The operation point (point B) is determined according to which of the sections P1 to P4 shown in FIG. 11 the position (point A) where the finger region 40 and the side of the visual recognition area 50 intersect is present. Specifically, it is as follows.
When point A is present on the side of section P1, point B is at the bottom and rightmost of the finger area.
When point A is present on the side of section P2, point B is at the bottom and leftmost of the finger area.
When point A exists on the side of section P3, point B is at the top and rightmost of the finger area.
When point A is present on the side of section P4, point B is on the top and leftmost of the finger area.

<Generation of Pointer Image (Step S108)>
12 is a flowchart for explaining step S108 shown in FIG. 3, and FIG. 13 is a flowchart for explaining step S503 shown in FIG. 14A to 14E are diagrams for illustrating pointer images.

  Hereinafter, the image forming apparatus installed at the installation position is assumed in real space, and the image forming apparatus is referred to as a virtual image forming apparatus. The virtual image forming apparatus visualized on the display 130 is the apparatus image 20 of the image forming apparatus described above. In this embodiment, the shape of the pointer image is selected according to the distance between the virtual image forming apparatus and the operation point.

  The separation distance between the virtual image forming apparatus and the operation point can be calculated based on the position where the image forming apparatus is installed and the position of the operation point. In addition, the position of each component of the virtual image forming apparatus, for example, the position of the operation panel, is calculated based on the position where the image forming apparatus is installed and information on the size and position of each part of the image forming apparatus and the like. Can do.

  On the other hand, as described above, the position of the operation point is based on the relative positional relationship between the camera 110 and the operator's finger, and thus needs to be converted to a position based on the marker 200. The separation distance between the camera 110 and the marker 200 can be calculated from the image of the marker 200 extracted from the background video. Therefore, the position of the operation point in the real space with reference to the marker 200 can be calculated. Hereinafter, the position of the operation point with reference to the marker 200 is simply referred to as an operation point.

  In the present embodiment, the shape, color, and transmittance of the pointer image can be changed according to the internal state of the simulation model and the separation distance between the virtual image forming apparatus and the operation point. Hereinafter, a procedure for generating a pointer image will be described.

  First, as shown in FIG. 12, the internal state information of the simulation model is acquired (step S501). As described above, the internal state information is information related to the internal state of the simulation model.

  Next, a separation distance between the virtual image forming apparatus and the operation point is calculated (step S502). Specifically, the control unit 160 calculates a separation distance between the virtual image forming apparatus and the operation point based on the position of the virtual image forming apparatus and the position information of the fingers.

  Next, a pointer image is selected (step S503). The control unit 160 selects a pointer image according to the internal state of the simulation model and the separation distance between the virtual image forming apparatus and the operation point. Specifically, this is as shown in FIG. Hereinafter, “arrow (Obj1)”, “finger (Obj2)”, “translucent arrow (Obj3)”, and “tool (Obj4)” will be described as examples of pointer images. It is not limited. Further, the correspondence relationship between the distance and the internal state and the shape, color, and transmittance of the pointer image is not limited to that described below.

  First, it is determined whether or not the distance between the virtual image forming apparatus and the operation point is equal to or greater than a first predetermined distance (step S601). When the distance between the virtual image forming apparatus and the operation point is equal to or greater than the first predetermined distance (step S601: equal to or greater than the first predetermined distance), as shown in FIG. 14A, a “translucent arrow (Obj3) is displayed as the pointer image 31. ) ”Is selected (step S602).

  On the other hand, when the distance between the virtual image forming apparatus and the operation point is less than the first predetermined distance (step S601: less than the first predetermined distance), the internal state of the simulation model is determined (step S603). When the internal state is “failure state” (step S603: failure state), “tool (Obj4)” is selected as the pointer image 32 as shown in FIG. 14B (step S604). The first predetermined distance is, for example, 50 cm to 2 m, and preferably 1 m.

  On the other hand, when the internal state is the “normal state” (step S603: normal state), it is determined whether the surface closest to the operation point is the back surface among the surfaces of the virtual image forming apparatus (step S605). . When the surface closest to the operation point is the back surface among the surfaces of the virtual image forming apparatus (step S605: YES), “tool (Obj4)” is selected as the pointer image 32 as shown in FIG. 14C (step S606). ).

  On the other hand, when the surface closest to the operation point is not the back surface among the surfaces of the virtual image forming apparatus (step S605: NO), the operation point is within the second predetermined distance from the operation panel of the virtual image forming apparatus. It is determined whether or not it exists (step S607). When the operation point exists within the second predetermined distance from the operation panel of the virtual image forming apparatus (step S607: YES), “finger (Obj2)” is selected as the pointer image 33 as shown in FIG. 14D ( Step S608). Note that the second predetermined distance is, for example, 10 to 30 cm, and preferably 20 cm.

  On the other hand, if the operation point does not exist within the second predetermined distance from the operation panel of the virtual image forming apparatus (step S607: NO), “arrow (Obj1)” is selected as the pointer image 30 as shown in FIG. 14E. (Step S609).

  Next, returning to FIG. 12 again, a pointer image is generated (step S504). As shown in FIGS. 4A to 4E, the pointer images 30 to 33 are generated so that their tips coincide with the operation points and are superimposed on the image 10 of the operator's finger in the background video. The generated pointer images 30 to 33 are stored in the RAM unit 163.

<Execution of predetermined operation of simulation model (step S110)>
FIG. 15 is a flowchart for explaining step S110 shown in FIG. 3, and FIG. 16 is a flowchart for explaining step S702 shown in FIG. FIG. 17 is a diagram illustrating an example of an image displayed on the display when a “printing operation” simulation is executed.

  First, internal state information of the simulation model is acquired (step S701). The control unit 160 acquires the internal state information of the simulation model, and refers to the current state of the simulation model.

  Next, the operation of the simulation model is selected (step S702). The control unit 160 selects the operation of the simulation model according to the positional relationship between the virtual image forming apparatus and the operator's fingers and the internal state of the simulation model. Specifically, it is as shown in FIG. In the following description, the case of selecting “printing operation” and “rear door opening operation” as the operation of the simulation model will be described as an example, but the operation of the simulation model will be described as “printing operation” and “rear door opening operation”. It is not limited.

  First, the internal state of the simulation model is determined (step S801). When the internal state is “normal state” (step S801: normal state), the process proceeds to step S802. On the other hand, when the internal state is “failure state” (step S801: failure state), the process proceeds to step S805.

  When the process proceeds to step S802, it is determined whether or not the operation point exists within a third predetermined distance from the operation panel of the virtual image forming apparatus (step S802). If the operation point does not exist within the third predetermined distance from the operation panel of the image forming apparatus (step S802: NO), the process proceeds to step S703 in FIG. 15 without selecting the operation of the simulation model. Note that the third predetermined distance is, for example, 1 to 10 cm, and preferably 5 cm.

  On the other hand, when the operation point exists within the third predetermined distance from the operation panel of the virtual image forming apparatus (step S802: YES), it is determined whether or not the simulation model is operating (step S803). If the simulation model is operating (step S803: YES), the process proceeds to step S703 in FIG. 15 without selecting the operation of the simulation model.

  On the other hand, if the simulation model is not in operation (step S803: NO), “printing operation” is selected as the operation of the simulation model, and the process proceeds to step S703 in FIG.

  Next, when the process proceeds to step S805, it is determined whether or not the operation point is within a fourth predetermined distance from the opening / closing part of the rear door of the virtual image forming apparatus (step S805). If the operation point does not exist within the fourth predetermined distance from the opening / closing part of the rear door of the virtual image forming apparatus (step S805: NO), the process proceeds to step S703 in FIG. 15 without selecting the operation of the simulation model. . The fourth predetermined distance is, for example, 1 to 10 cm, and preferably 5 cm.

  On the other hand, if the operation point exists within the fourth predetermined distance from the opening / closing part of the rear door of the virtual image forming apparatus (step S805: YES), it is determined whether or not the simulation model is open at the rear door (step S805). S806). If the simulation model is open at the rear door (step S806: YES), the process proceeds to step S703 in FIG. 15 without selecting the operation of the simulation model.

  On the other hand, if the simulation model is not open at the back door (step S806: NO), “back door open operation” is selected as the operation of the simulation model, and the process proceeds to step S703 in FIG.

  Next, returning to FIG. 15 again, the internal state of the simulation model is updated (step S703). Based on the operation of the simulation model selected in step S702, the internal state of the simulation model is updated.

  Next, it is determined whether or not the operation point exists in a specified area (step S704). As the operator's finger moves, the display position of the pointer image also moves. The control unit 160 determines whether or not the operation point exists in a predetermined region (for example, a start button on the operation panel) of the virtual image forming apparatus. If the operation point does not exist in the specified area (step S704: NO), the process proceeds to step S111 in FIG. 3 without executing the predetermined operation of the simulation model.

  The prescribed area does not necessarily have to be an area in contact with the operation point. For example, it can be set to include a predetermined range above the start button on the operation panel.

  Further, instead of determining whether or not the operation point exists in a predetermined region, for example, a predetermined operation of the simulation model is performed according to the distance that the position of the operation point moves per unit time, that is, the movement speed of the operation point. It may be configured to execute.

  On the other hand, when the operation point exists in the specified region (step S704: YES), a predetermined operation of the simulation model is executed (step S705). Based on the operation of the simulation model selected in step S702, a predetermined operation of the simulation model is executed.

  For example, when the operation of the selected simulation model is “printing operation”, the control unit 160 executes “printing operation” with the simulation model. As shown in FIG. 17, when the “printing operation” is executed, the display 130 displays a moving image of the discharge of the paper S printed by the device image 20. Further, the speaker 150 generates a printing sound in accordance with the printing operation of the device image 20. Instead of displaying the printing of the device image 20 as a moving image, a moving image that records the printing performed by the actual image forming apparatus may be displayed on the display 130.

  As described above, the image display apparatus 100 according to the present embodiment extracts the image of the marker 200 from the background video of the place where the image forming apparatus is installed, acquires the display position information, and generates the apparatus image 20. In addition, the image display apparatus 100 acquires finger position information and generates pointer images 30 to 33. Then, based on the position where the image forming apparatus is installed and the position information of the fingers, a predetermined operation of the simulation model is executed, and the apparatus image 20 and the pointer images 30 to 33 are combined with the background video and displayed.

  As described above, the described embodiment has the following effects.

  (A) The apparatus image 20 is synthesized and displayed on the background image of the place where the image forming apparatus is installed, and a predetermined operation of the simulation model of the apparatus image 20 displayed on the display 130 is performed based on the position of the finger of the operator. Execute and start the operation of the device image 20. Therefore, the operator can experience as if the image forming apparatus or the like is actually operated at the installation location, so that it is possible to examine in advance in a state close to the case where the image forming apparatus or the like is actually installed. it can.

  (B) Since the image of the marker 200 arranged at the position where the image forming apparatus is installed is extracted and the display position information of the image forming apparatus is acquired, the position of the operator holding the image display apparatus 100 changes. In addition, the device image 20 can be displayed on the background image with an accurate position and orientation. As a result, the operator can reliably grasp the positional relationship between the image forming apparatus to be installed and other objects in the room, for example, the window W.

  (C) Without using the actual image forming apparatus, it is possible to reduce the cost of securing and installing the actual object for confirmation that can confirm the operation of the image forming apparatus, toner, paper, and the like.

(Second Embodiment)
In the first embodiment, it is described that a marker image is extracted from a background video of a place where an image forming apparatus is installed to obtain display position information, and the background video, the device image, and the pointer image are combined and displayed. did. In the second embodiment, the marker position image is extracted from the background image and the display position information is acquired by using another method without acquiring the display position information, and the specified background image, device image, and pointer image are obtained. Explain that combining and displaying. In the following description, the description of the same configuration as that of the first embodiment is omitted to avoid duplication of description.

  FIG. 18 is a schematic block diagram of an image display apparatus according to the second embodiment of the present invention. As illustrated in FIG. 18, the image display apparatus 100 according to the present embodiment includes a background video generation unit 180 instead of the camera 110 according to the first embodiment.

  The background video generation unit 180 may be built in the housing 170 of the image display device 100 or may be disposed outside the housing 170. When the background video generation unit 180 is disposed outside the housing 170, the video data is transmitted to the control unit 160 by wired communication or wireless communication. Hereinafter, a case where the background video generation unit 180 is built in the housing 170 will be described.

<Acquisition of background video>
The background video generation unit 180 stores a plurality of prescribed background videos in the large-capacity storage unit 164 as background video acquisition means. The background image includes, for example, an image of an office where an image forming apparatus or the like is installed, an image of an exhibition hall that introduces the image forming apparatus or the like, an image of computer graphics, and the like. The control unit 160 generates an output video by combining the device image 20 and the pointer image with the prescribed background video generated by the background video generation unit 180, and the display 130 displays the combined output video. Therefore, the operator can appropriately switch the background video according to the place or situation where the image forming apparatus or the like is assumed to be installed.

<Acquisition of display position information of image forming apparatus to be installed>
In the present embodiment, for example, it is assumed that the position of the image display apparatus 100 is determined in advance and the position of the image forming apparatus to be installed and the relative position of the image display apparatus 100 are known. Or you may use the well-known technique which can acquire the position of the image display apparatus 100 correctly. The control unit 160 can acquire display position information of the image forming apparatus based on the relative positional relationship between the position of the image forming apparatus to be installed and the image display apparatus 100.

  As described above, the image display apparatus 100 according to the present embodiment acquires display position information based on the relative positional relationship between the position of the image forming apparatus to be installed and the operator, and is generated by the background video generation unit 180. A device image and a pointer image are synthesized and displayed on a prescribed background video.

  As described above, the described embodiment has the following effects in addition to the effects of the first embodiment.

  (D) The background video generation unit 180 holds a plurality of prescribed background videos in the large capacity storage unit 164. Therefore, the operator can appropriately switch the background video according to the place where the image forming apparatus or the like is assumed to be installed.

  As described above, the image display device of the present invention has been described in the embodiment. However, it goes without saying that the present invention can be appropriately added, modified, and omitted by those skilled in the art within the scope of the technical idea.

  For example, in the first embodiment, the case where one marker is arranged and one image forming apparatus is installed has been described. However, the number of markers is not limited to one, and a plurality of markers may be arranged. For example, when two markers are arranged, device images of two image forming apparatuses can be displayed.

  Further, in the first embodiment, it has been described that display position information is acquired using a marker arranged at a place where the image forming apparatus is installed. However, the method for acquiring the display position information is not limited to the method using a marker.

  In the first embodiment, a finger region is extracted from the image data of the object detected by the sensor, and the position where the finger region and the side of the camera viewing region intersect each other exists in any predetermined section. It has been explained that the operation point is determined depending on whether or not to do so. However, the operation point determination method is not limited to such a method, and other methods may be used.

10 Images of the fingers of the operator,
20 Device image,
30-33 pointer image,
40 finger area,
100 image display device,
110 cameras,
120 sensor 130 display,
140 touch panel,
150 speakers,
160 control unit,
161 CPU section,
162 RAM section,
163 ROM part,
164 Mass storage unit,
170 housing,
180 background image generator,
200 markers.

Claims (8)

A procedure (a) for acquiring a background image;
A procedure (b) of acquiring display position information of the image forming apparatus to be installed;
A procedure (c) for generating a device image of the image forming device based on the display position information and information on the size of the image forming device;
A procedure (d) of acquiring position information of a hand or a finger;
A step (e) for generating a pointer image indicating the position of the hand or finger based on the position information of the hand or finger;
A procedure (f) of generating an output video by combining the background video, the device image, and the pointer image;
A procedure (g) for executing a predetermined operation of a simulation model for simulating the operation of the image forming apparatus based on the position where the image forming apparatus is installed and the position information of the hand or finger;
A procedure (h) for displaying the output video;
A control processing program for causing a computer to execute the program.   The control processing program according to claim 1, wherein the background video is acquired by an imaging unit imaging a place where the image forming apparatus is installed.   The control processing program according to claim 2, wherein in the step (b), a marker image is extracted from the background video to acquire the display position information.   The computer-readable recording medium which recorded the image processing program of any one of Claims 1-3. Background image acquisition means for acquiring a background image;
Display position information acquisition means for acquiring display position information of the image forming apparatus to be installed;
An apparatus image generation unit configured to generate an apparatus image of the image forming apparatus based on the display position information and information on the size of the image forming apparatus;
Finger position information acquisition means for acquiring position information of the hand or finger;
Pointer image generation means for generating a pointer image indicating the position of the hand or finger based on the position information of the hand or finger;
Image combining means for combining the background image, the device image, and the pointer image to generate an output image;
Model execution means for executing a predetermined operation of a simulation model for simulating the operation of the image forming apparatus based on the position where the image forming apparatus is installed and the position information of the hand or finger;
Display means for displaying the output video;
An image display device comprising: The background video acquisition means has imaging means,
The image display apparatus according to claim 5, wherein the background video is acquired by the imaging unit imaging a place where the image forming apparatus is installed.   The image display apparatus according to claim 6, wherein the display position information acquisition unit acquires the display position information by extracting a marker image from the background video. Acquiring a background image;
Acquiring display position information of the image forming apparatus to be installed;
Generating a device image of the image forming apparatus based on the display position information and information on a size of the image forming apparatus;
Obtaining hand or finger position information;
Generating a pointer image indicating the position of the hand or finger based on the position information of the hand or finger;
Synthesizing the background video, the device image, and the pointer image to generate an output video;
Executing a predetermined operation of a simulation model for simulating the operation of the image forming apparatus based on the position where the image forming apparatus is installed and the position information of the hand or finger;
Displaying the output video;
An image display method comprising: