JP2013105329A - Simulation method, simulation device and control program for simulation device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a simulation method, a simulation device and a control program for the simulation device, which can reduce trouble in an object device after its purchase.SOLUTION: The invention includes a step (S05) for acquiring an image of installation site in a real space where a device to be an object of AR simulation is arranged, a step (S06) for detecting a designated installation site of the object device, a step (S08) for generating an AR object of the object device fitted with an optional device, at the detected installation site, according to the fitting condition of the optional device fitted to the object device, and a step (S09) for generating a composite image including the image and AR object, and displaying the composite image.

Description

  The present invention relates to a simulation method, a simulation apparatus, and a control program for the simulation apparatus.

  When purchasing the target device, which is a product, when considering in advance whether there is sufficient space at the installation location, etc., the only option is to use the dimensions described in the device catalog and the mockup of the target device. It was difficult to prevent troubles due to mistakes and omissions.

  On the other hand, a technique has been proposed in which augmented reality (AR) is used to synthesize and present furniture AR objects in the real environment (see, for example, Patent Document 1).

Special table 2010-541044 gazette

  However, even if the technique of Patent Document 1 is applied and the AR object of the target device is synthesized and presented in augmented reality, it does not have effective information for examining in advance whether or not the installation status of the target device is appropriate. Therefore, it was difficult to reduce trouble after purchase.

  The present invention has been made in order to solve the problems associated with the above-described prior art, and has an object to provide a simulation method, a simulation apparatus, and a control program for the simulation apparatus that can reduce trouble after the purchase of the target apparatus. To do.

  The above object of the present invention is achieved by the following means.

(1) A simulation device control program,
A background image acquisition procedure for acquiring an image of the installation location in the real space where the target device of the AR simulation is arranged;
An installation position detection procedure for detecting an installation position of the specified target device;
An AR object generation procedure for generating an AR object of the target device to which the option device is mounted at the detected installation position in accordance with the mounting status of the option device to be mounted on the target device;
An image composition procedure for generating a composite image including the video and the AR object;
A display procedure for displaying the composite image;
A control program causing the simulation apparatus to execute a process having

  (2) The control program according to (1), wherein, in the AR object generation procedure, the AR object is generated in a state where a plurality of the optional devices are overlapped or transmitted.

  (3) The control program according to (1) or (2), wherein the processing further includes a data reproduction procedure for reproducing audio data, moving image data, or device rotation data prepared in advance. .

(4) The process further includes an additional information generation procedure for generating additional information for determining whether or not the installation status of the target device is appropriate.
The control program according to any one of (1) to (3), wherein in the image composition procedure, a composite image to which the additional information is added is generated.

(5) The installation status considers the casing size of the target device,
The control program according to (4), wherein the case size includes a size of the option device when the option device is mounted on the target device.

(6) The installation situation considers the installation size of the target device,
The installation size is calculated by adding a work area required for actual operation to the housing size of the target device,
The control program according to (4), wherein the case size includes a size of the option device when the option device is mounted on the target device.

  (7) The control program according to (6), wherein the work area is set in consideration of a body shape of an operator of the target device.

  (8) The control program according to (6), wherein the work area is set in consideration of a connector for connecting to an external peripheral device.

  (9) In the above (8), the additional information further includes an insertion direction of the connector, a standard of the connector, a name of a peripheral device connected to the connector, or a combination thereof. The control program described in 1.

(10) The processing further includes a detection procedure for detecting interference between the target device and an obstacle located at the installation location of the real space,
The control program according to any one of (1) to (9), wherein the additional information includes warning information that alerts the interference.

  (11) The control program according to (10), wherein the warning information includes a message indicating the interference.

  (12) In the installation position detection procedure, the installation position of the target device is detected based on a marker arranged at an installation location in the real space. The control program according to any one of the above.

(13) A background video acquisition step of acquiring a video of an installation location in a real space where a target device for AR simulation is arranged;
An installation position detection step for detecting an installation position of the specified target device;
An AR object generation step of generating an AR object of the target device to which the option device is mounted at the detected installation position according to a mounting state of the option device to be mounted on the target device;
An image compositing step for generating a composite image including the video and the AR object;
A display step for displaying the composite image;
A simulation method characterized by comprising:

  (14) The simulation method according to (13), further including a data reproduction step of reproducing audio data, moving image data, or device rotation data prepared in advance.

(15) The method further includes an additional information generation step of generating additional information for determining whether or not the installation status of the target device is appropriate,
The simulation method according to (13) or (14) above, wherein in the image composition step, a composite image to which the additional information is added is generated.

(16) An interference detection step of detecting interference between the target device and an obstacle located at the installation location in the real space,
The simulation method according to any one of (13) to (15), wherein the additional information includes warning information that alerts the interference.

(17) a background image acquisition unit for acquiring an image of an installation location in a real space where an AR simulation target device is arranged;
Installation position detection means for detecting the installation position of the specified target device;
An AR object generating means for generating an AR object of the target device to which the optional device is mounted at a detected installation position according to a mounting state of the optional device mounted on the target device;
Image synthesizing means for generating a synthesized image including the video and the AR object;
A display unit for displaying the composite image;
A simulation apparatus comprising:

  (18) The simulation apparatus according to (17), further including data reproduction means for reproducing audio data, moving image data, or apparatus rotation data prepared in advance.

(19) It further includes additional information generating means for generating additional information for determining whether or not the installation status of the target device is appropriate,
The simulation apparatus according to (17) or (18), wherein the image synthesis unit generates a synthesized image to which the additional information is added.

(20) It further includes interference detection means for detecting interference between the target device and an obstacle located at the installation location of the real space,
The simulation apparatus according to any one of (17) to (19), wherein the additional information includes warning information that alerts the interference.

  According to the simulation method, the simulation apparatus, and the control program for the simulation apparatus according to the present invention, the composite image displayed to the operator includes the image of the target device that is arranged at the designated installation position and that corresponds to the installation status of the optional device. (Effective information for examining whether the installation status of the target device is appropriate or not) is included, and it is possible to visually grasp the state close to when the target device was actually installed. When examining in advance whether or not the installation status of the apparatus is appropriate, it is easy to prevent troubles due to calculation mistakes or omissions. Therefore, it is possible to provide a simulation method, a simulation apparatus, and a control program for the simulation apparatus that can reduce trouble after purchase of the target apparatus.

It is the schematic for demonstrating the simulation apparatus which concerns on embodiment of this invention. It is the schematic for demonstrating the function of the simulation apparatus shown by FIG. It is a block diagram for demonstrating the structure of the simulation apparatus shown by FIG. It is a side view for demonstrating an example of the target product which concerns on 3D data shown by FIG. It is a side view for demonstrating another example of the target product which concerns on 3D data shown by FIG. It is a side view for demonstrating an example of the option apparatus which concerns on 3D data shown by FIG. It is a side view for demonstrating another example of the option apparatus which concerns on 3D data shown by FIG. It is a conceptual diagram for demonstrating an example of the operator body shape pattern which concerns on 3D data shown by FIG. It is a conceptual diagram for demonstrating another example of the operator figure pattern which concerns on 3D data shown by FIG. 4 is a chart for explaining a control processing program of a ROM section shown in FIG. 3. 4 is a chart for explaining display state information of a RAM section shown in FIG. 3. It is a conceptual diagram for demonstrating the menu screen of the display shown by FIG. It is a conceptual diagram for demonstrating the sensor shown by FIG. It is a conceptual diagram for demonstrating an example of designation | designated of the option apparatus of the option apparatus selection mode shown by FIG. It is a conceptual diagram for demonstrating another example of designation | designated of the option apparatus of the option apparatus selection mode shown by FIG. It is a conceptual diagram for demonstrating the main body color of the option apparatus selection mode shown by FIG. It is a conceptual diagram for demonstrating the superimposition of the superimposition mode shown by FIG. It is a conceptual diagram for demonstrating transmission of the superposition mode shown by FIG. It is a conceptual diagram for demonstrating the child of the operator designation | designated mode shown by FIG. It is a conceptual diagram for demonstrating the man of the operator designation | designated mode shown by FIG. It is a conceptual diagram for demonstrating the housing | casing size of the size display mode shown by FIG. It is a conceptual diagram for demonstrating the installation size of the size display mode shown by FIG. It is a conceptual diagram for demonstrating an example of the obstruction which concerns on the obstruction mode shown by FIG. It is a conceptual diagram for demonstrating the detection of the obstruction which concerns on the obstruction mode shown by FIG. It is a conceptual diagram for demonstrating the obstruction mode (interference warning) shown by FIG. It is a conceptual diagram for demonstrating the audio | voice mode, moving image mode, and rotation mode which are shown by FIG. It is a conceptual diagram for demonstrating the weight display mode shown by FIG. It is a conceptual diagram for demonstrating the connector position display mode shown by FIG. It is a flowchart for demonstrating the algorithm of the control processing program shown by FIG. It is a flowchart for demonstrating the algorithm of the touchscreen input process of step S03 shown by FIG. It is a flowchart for demonstrating the algorithm of the marker detection process of step S06 shown by FIG. It is a flowchart for demonstrating the algorithm of AR object image creation processing of step S08 shown by FIG. It is a flowchart for demonstrating the algorithm of AR object image composition display processing of step S09 shown by FIG.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a schematic diagram for explaining a simulation apparatus according to an embodiment of the present invention, and FIG. 2 is a schematic diagram for explaining functions of the simulation apparatus shown in FIG.

  The simulation apparatus 100 according to the embodiment of the present invention uses augmented reality (AR) technology to synthesize (overlay) the background image of the installation location 90 in the real space and the AR object image of the target device 10 that is a product. For example, an operator who purchases the target device 10 confirms the operation state of the target device 10 in advance (determines whether the installation status of the target device 10 is appropriate) Used to reduce trouble after purchase. For example, the installation location 90 is an office where an operator works, and the target apparatus 10 is a printer dedicated machine or an MFP (Multi-Function Peripheral) having a copy function, a printer function, and a scan function. A user of the target device 10.

  A symbol M is a marker composed of a sheet on which figures, symbols, characters, and the like are written, and simulates the installation position, the distance to the simulation device 100, and device identification information uniquely associated with the target device 10 to be installed. Used to teach the device 100. The information held by the marker M is described by, for example, a two-dimensional matrix code. The marker M is not limited to a sheet shape. Further, like the RFID tag, a memory and a transmitter can be attached to the marker M, and the installation position information and information stored in the memory (information that uniquely associates the target device 10) can be transmitted to the simulation device 100. is there.

  Next, the configuration of the simulation apparatus 100 will be described in detail.

  FIG. 3 is a block diagram for explaining the configuration of the simulation apparatus shown in FIG. 1, and FIGS. 4 and 5 are examples for explaining one example and another example of the target product related to the 3D data shown in FIG. 6 is a side view, FIG. 8 is a side view for explaining another example of the optional device related to the 3D data shown in FIG. 3, and FIG. 8 is a diagram showing the 3D data shown in FIG. FIG. 10 is a conceptual diagram for explaining an example of the operator body pattern and another example, FIG. 10 is a chart for explaining the control processing program of the ROM unit shown in FIG. 3, and FIG. 11 is shown in FIG. FIG. 12 is a conceptual diagram for explaining the menu screen of the display shown in FIG. 3, and FIG. 13 is a diagram for explaining the sensor shown in FIG. Conceptual diagram That.

  The simulation apparatus 100 includes a camera-equipped mobile phone and a smartphone, and includes a CPU 110, a ROM unit 120, a RAM unit 130, a touch panel 140, a display 150, a camera 160, a sensor 170, and a speaker 180. Are connected to each other.

  The CPU 110 is a control circuit composed of a microprocessor or the like that executes control of the above-described units and various arithmetic processes according to a program. Each function of the simulation apparatus 100 is exhibited by the CPU 110 executing a corresponding program. Is done.

  The ROM unit 120 has, for example, a nonvolatile semiconductor memory and a large-capacity magnetic disk device, and stores 3D (dimensional) data, content data, and a control processing program.

  The 3D data includes external data of an apparatus main body (see FIGS. 4 and 5) and an optional device (see FIGS. 6 and 7) of the target device 10 and external data for schematically displaying an operator (see FIGS. 8 and 8). 9) and outline data for schematically displaying an obstacle located at the installation location 90 in the real space.

  The content data includes specification information, reproduction data, and marker image information related to the device main body and the optional device.

  The specification information includes the case size, color and weight of the device main unit and optional device, suitability of the combination of the device main unit and optional device, connector information for connecting to external peripheral devices, operator classification data (operation Human body information expressing the body pattern of the person). The connector information includes the installation position and insertion direction of the connector, the connector standard, and the name of the peripheral device connected to the connector.

  The reproduction data includes audio data, moving image data, and rotation data related to the operation of the apparatus. The rotation data is device rotation data for rotating the target device and grasping the appearance of the entire device. The marker image information is identification information for separating (cutting out) the image of the marker M from the background video of the installation place 90 in the real space. Note that the housing size can also be calculated from 3D data.

  The control processing program acquires an image of the installation location in the real space where the target device of the AR simulation is arranged, detects the installation position of the designated target device, and according to the mounting status of the optional device mounted on the target device At the detected installation position, an AR object of the target device on which the optional device is mounted is generated, a composite image including the video and the AR object is generated, and used to display the composite image. As shown in the figure, the program includes a program for executing touch panel input processing, marker detection processing, AR object image creation processing, AR object image composition display processing, and the like, which are read by the CPU 110 as necessary and executed on the RAM unit 130. Is done.

  Therefore, the composite image displayed to the operator includes an image of the target device that is placed at the specified installation position and according to the mounting status of the optional device (for examining whether the installation status of the target device is appropriate). Effective information), it is possible to visually grasp the state near the time when the target device was actually installed, and when calculating in advance whether or not the target device is installed properly It is easy to prevent troubles due to mistakes and omissions. That is, it is possible to reduce troubles after purchasing the target device.

  The touch panel input process is executed to update the display state information. The marker detection process is executed to calculate the installation position (display position) and direction (direction) of the target device 10. The AR object image creation process is executed to convert 3D data into an AR object image. The AR object image composition display process is executed to present an AR simulation result to the operator.

  The RAM unit 130 includes, for example, a volatile semiconductor memory and a large-capacity magnetic disk device, and temporarily stores programs and data as a work area, background video, AR object image, display state information Used to remember.

  The background video is image data obtained by photographing the installation location 90 in the real space with the camera 160. The AR object image is image data converted from 3D data of the device main body, the optional device, and the obstacle. The display state information is used to additionally display obstacles and additional information and reproduce designated data when presenting the AR simulation result to the operator, as shown in FIG. In addition, an optional device selection mode, an overlay mode, an operator designation mode, a size display mode, an obstacle mode, an audio mode, a moving image mode, a rotation mode, a weight display mode, and a connector position display mode are included.

  The touch panel 140 is an input unit used for an operator to perform various instructions such as character input, various settings, and a start instruction. For example, the touch panel 140 is used to input (update) display state information. It is also possible to input identification data for specifying the position of the target device 10 to be installed or specifying the target device 10 to be installed using the touch panel 140.

  The display 150 is integrated with the touch panel 140 in the present embodiment, and displays an image of the camera 160, an AR simulation result, an error occurrence state, a currently changeable setting, a composite image (annotation), and the like. Is an output means used for presenting.

  The display 150 has a screen having a hierarchical structure for inputting display state information. For example, in the configuration shown in FIG. 12, when the operator presses the Menu button, the screen is updated, a selection menu related to the change of the display state information is displayed, and the display state can be changed by the operator. It becomes. For example, toggles and radio buttons can be used to change the display state information.

  In changing the display state information, it is possible to switch to the main mode by touching the screen without using the menu display. For example, each time a place other than the Menu button is clicked, the size display mode may be switched in the order of “none” → “case size” → “installation size” → “none”.

  The installation size includes the housing size of the apparatus main body and the work area required for actual operation. The housing size of the target device includes the size of the optional device when the optional device is mounted on the device main body.

  The camera 160 is composed of, for example, an RGB camera, and is used to capture a background image of the installation space 90 in the real space. The background video is used for image extraction of the marker M and synthesis with the AR object image. The image of the marker M is used to calculate the distance between the marker M and the simulation apparatus 100 (camera 160) and the direction (direction) of the marker M from the size and inclination of the marker M. . Note that the size and orientation of the target device 10 on the AR are directly calculated from the size and inclination of the image of the marker M obtained from the background video without calculating the distance between the marker M and the simulation device 100 in real space. It is also possible to do.

  The sensor 170 is a depth sensor that calculates the distance to the object based on the time until the reflected light from the object returns, and is used to measure the distance to the obstacle as shown in FIG. Is done. The measurement light is, for example, infrared. The obstacle is, for example, a wall 92. In the present embodiment, the camera 160 and the sensor 170 are integrated, and are composed of a kinect sensor. As the sensor 170, a three-dimensional sensor such as a magnetic sensor or an ultrasonic sensor can be applied as necessary.

Note that the reference point P ₁ representing the position of the marker M calculated from the size and inclination of the image of the marker M is combined with the measurement points P _{2 to} P ₄ of the obstacle and the background image of the installation place 90 in the real space. It is used when 3D data of the target device 10 or the like is subjected to coordinate conversion. The obstacle measurement points P _{2 to} P ₄ are used to detect the presence or absence of interference between the target device 10 arranged at the measurement point P ₁ of the marker M and the obstacle.

In the interference of the obstacle, the obstacle is an image that occupies a part of the background image, and it cannot be directly determined whether or not the object device 100 interferes (there is a three-dimensional collision). Therefore, when the distance is arranged target device 10 on the reference point P ₁ of the calculated and are (location is specified) marker M, the target device 10 that would be placed in the farthest distance from the sensor 170 The position of the part is the background reference distance. Next, when the sensor 170 scans the installation place 90 in the real space, the distances between the plurality of measurement points P _{2 to} P ₄ in the background image and the sensor 170 (camera 160) are detected, and these distances are detected. The 3D data of the obstacle is acquired by integrating the data.

The 3D data of the obstacle can be compared three-dimensionally with the 3D data of the target device 10 arranged at the marker M. That, AR object obstacle located in the interior of the installation size region (in the case of measuring point P ₃₎ In case of overlap with the installation size of the target device 10, determines that interferes with the obstacle, installation size located outside the region (in the case of measuring point P ₂ and the measurement point P ₄₎ when not overlap the installation size of the target device 10, it is possible to determine not to interfere with the obstacle.

  Note that the number of measurement points is not particularly limited, and is appropriately set depending on the performance of the sensor 170, for example. The presence or absence of interference can also be based on the size of the housing. Further, the 3D data of the installation location 90 in the real space can be obtained by using a stereo camera that can record information in the depth direction by simultaneously photographing the object from a plurality of different directions.

  The speaker 180 is used to reproduce the sound specified in the sound mode of the display state information and the sound included in the moving image specified in the moving image mode.

  Next, each mode of the display state information will be described in detail.

  14 and 15 are conceptual diagrams for explaining an example of designation of an option device in the option device selection mode shown in FIG. 11 and another example, and FIG. 16 is a main body of the option device selection mode shown in FIG. It is a conceptual diagram for demonstrating a color.

  The optional device selection mode has a list of optional devices that can be mounted and a list of switchable device main body colors (main body colors). The optional device can be specified (with or without mounting), or the device main body color ( Used to switch the body color. For example, the optional device is a finisher or ADF (Auto Document Feeder), and the main body colors are white, black, and gray.

  For example, when an optional device is designated, as shown in FIGS. 14 and 15, 3D data of a target device having the optional device attached to the device main body is generated. When the main body color is switched, for example, when black is selected, the white printer shown in FIG. 4 is changed to the black printer shown in FIG.

  When the optional device is selected, the position data of the outer shape of the target device (relative distance data from the marker M) is automatically recalculated based on the position data of the marker M. That is, the relative distance between the reference point representing the position of the target device and the marker M is changed according to the selection (mounting) state of the optional device. The selection of the reference point that represents the position of the target device is not particularly limited, and for example, an internal center point or a center of gravity point of the target device, a corner portion on the bottom surface of the target device, or the like can be used. Further, the position data of the optional device can be specified by the difference data based on the position data of the apparatus main body without changing the position data of the apparatus main body.

  17 and 18 are conceptual diagrams for explaining “superposition” and “transmission” in the superposition mode shown in FIG. 11.

  In the superposition mode, “single”, “superposition”, and “transmission” can be selected. “Single” is designated when only one optional device that cannot be installed at the same time (ie, alternative or exclusive) is to be displayed. “Overlapping” and “Transmission” are used when it is desired to simultaneously display a plurality of optional devices that are exclusively installed. In “overlay”, as shown in FIG. 17, option devices having a large display size are displayed in order from the background layer. In “transparent”, as shown in FIG. The device is displayed with an arbitrary transmittance. Note that the overlaid optional devices can be displayed in different colors so that they can be easily identified.

  FIG. 19 is a conceptual diagram for explaining “child” and “male” in the operator designation mode shown in FIG.

  The operator designation mode is used to designate an operator type and additional features in the operator's AR object image. The operator types are “none”, “children”, “female” and “male”, and an additional feature is “body type”. “None” is designated when it is not desired to display the AR object image of the operator. The object sizes of “children”, “female”, and “male” are 130 cm tall, 160 cm tall, and 170 cm tall, respectively. “Body shape” is classified into three types (thin, medium, and thick), and is designated when fine adjustment of the size of the AR object is desired.

  For example, when “child” is designated, the AR object image shown in FIG. 19 is synthesized, and when “male” is designated, the AR object image shown in FIG. 20 is synthesized. The AR object related to the “body shape” can be generated by preparing independently or by transforming the designated AR object.

  21 and 22 are conceptual diagrams for explaining the housing size and the installation size in the size display mode shown in FIG.

  As the size display mode, “None”, “Case size”, and “Installation size” can be selected. “None” is designated when it is not desired to display the frame and size data. As shown in FIG. 21, “casing size” is designated when it is desired to display a frame and size data in accordance with each size of the option configuration currently displayed. As shown in FIG. 22, the “installation size” is designated when it is desired to display a frame and size data in accordance with the installation size of the option configuration currently displayed.

  The work area required for the actual operation included in the installation size includes the operator type and additional features, the location of the connector to connect to external peripheral devices and the location of the connector to connect to the power supply. It can be calculated dynamically based on the presence or absence of surrounding obstacles, paper discharge status during operation simulation, etc., or by adding a fixed size. is there.

  FIG. 23 is a conceptual diagram for explaining an example of an obstacle related to the obstacle mode shown in FIG. 11, and FIG. 24 is a concept for explaining detection of an obstacle related to the obstacle mode shown in FIG. 25 is a conceptual diagram for explaining the obstacle mode (interference alarm) shown in FIG.

  The obstacle mode is used to consider an obstacle present at the installation location 90 in the real space, and “None”, “Display only”, and “Interference alarm” can be selected. The obstacle is, for example, a wall 92 or a bookshelf 94 shown in FIG. “None” is designated when it is not desired to execute detection and display of surrounding obstacles. “Display only” is specified when it is desired to execute only detection of surrounding obstacles. For example, as shown in FIG. 24, an obstacle object is displayed. “Interference warning” is designated when it is desired to execute detection of obstacles in the vicinity and detection of interference, and as shown in FIG. 25, a video and a warning message indicating a collision state with an obstacle are displayed. The The warning message is interference information that calls attention to an obstacle.

  The occurrence of interference is determined by detecting contact between the target apparatus and an obstacle (overlap between an area occupied by the installation size and an area occupied by surrounding obstacles). In addition, the warning is not limited to a form constituted by characters, and may be constituted by highlighting a portion where the target device and the obstacle are in contact with each other, or may be appropriately combined. is there.

  FIG. 26 is a conceptual diagram for explaining the audio mode, moving image mode, and rotation mode shown in FIG. 11.

  The audio mode, the moving image mode, and the rotation mode are used for operation display using reproduction data including content data, for example, as shown in FIG.

  The voice mode is used for designating voice data (voice Nos. 1 to 3) prepared in advance. The sound data is, for example, an operation sound, and a designated sound is reproduced. The moving image mode is used to specify moving image data (moving image Nos. 1 to 3) prepared in advance. The moving image data is, for example, a moving image in which paper is discharged as printing starts, and a specified moving image is reproduced. The rotation mode is used to rotate the target device to grasp the overall appearance of the device, and the device rotation data is reproduced.

  FIG. 27 is a conceptual diagram for explaining the weight display mode shown in FIG. 11.

  As the weight display mode, “present” and “absent” can be selected. “None” is designated when the weight is not desired to be displayed as shown in FIG. As shown in FIG. 27, “present” is designated when it is desired to display the total weight of the currently displayed apparatus main body and the optional apparatus. Note that the weight of the apparatus main body and the weight of the optional device can be displayed independently.

  FIG. 28 is a conceptual diagram for explaining the connector position display mode shown in FIG.

  “Yes” and “No” can be selected as the connector position display mode. “Present” is designated when it is desired to display the connector installation position data and insertion direction for connecting to an external peripheral device, the connector standard, and the name of the peripheral device, as shown in FIG. For example, the connector standard is USB (Universal Serial Bus), and the name of the peripheral device is a power supply. The insertion direction of the connector, the standard of the connector, and the name of the peripheral device can be displayed in a balloon, for example.

  Next, the control processing program will be described in detail.

  FIG. 29 is a flowchart for explaining an algorithm of the control processing program shown in FIG.

  First, in order to detect the presence or absence of a touch panel operation (operation instruction using the touch panel), an input scan is executed (step S01), and the presence or absence of the touch panel operation is determined (step S02). When the touch panel operation is not detected (step S02: No), the execution sequence proceeds to step S05. When a touch panel operation is detected (step S02: Yes), touch panel input processing is executed (step S03). Then, it is determined whether or not the control process has ended (step S04).

  When it is determined that the control process is finished (step S04: Yes), the execution of the control process program is finished. When it is determined that the control process does not end (step S04: No), the execution sequence proceeds to step S05.

  In step S05, the background video from the camera is captured (stored) in the background video portion of the RAM, and then the marker detection process is executed (step S06). Next, it is determined whether or not a marker is detected in the marker detection process (step S07).

  When it is determined that the marker has not been detected (step S07: No), the execution sequence returns to step S01. If it is determined that a marker has been detected (step S07: Yes), an AR object image creation process and an AR object image composition process are executed (steps S08 and S09), and the execution sequence returns to step S01.

  Next, the touch panel input process will be described in detail.

  FIG. 30 is a flowchart for explaining the algorithm of the touch panel input process of step S03 shown in FIG.

  First, the presence or absence of pressing of the Menu button displayed on the display 150 is detected using the touch panel 140 (step S11). When pressing of the Menu button is not detected (step S11: No), the execution sequence proceeds to step S15.

  When pressing of the Menu button is detected (step S11: Yes), a screen (see FIG. 12) for inputting display state information is displayed on the display 150 (step S12).

  Whether or not the mode designation item (see FIG. 11) displayed on the display 150 is pressed (setting change) is detected using the touch panel 140 (step S13). When the pressing of the mode designation item is not detected (step S13: No), the execution sequence proceeds to step S15.

  When the mode designation item is pressed and a change in setting is detected (step S13: Yes), the display state information is updated according to the change in setting (step S14), and the execution sequence proceeds to step S15.

  In step S15, it is determined whether or not the touch panel input process has ended. When it is determined that the touch panel input process is not finished (step S15: No), the execution sequence returns to step S11. When it is determined that the touch panel input process is finished (step S15: Yes), a series of processes for ending the process is executed (step S16), and the touch panel input process is finished.

  Next, the marker detection process will be described in detail.

  FIG. 31 is a flowchart for explaining the marker detection processing algorithm of step S06 shown in FIG.

  First, a process of extracting marker image information included in the content data of the ROM unit 120 from the background video captured in the background video unit of the RAM unit 130 is executed (step S21), and the marker image information is extracted (marker is detected). It is determined whether or not it has been completed (step S22).

  If it is determined that the marker image information could not be extracted (step S22: No), the marker detection process ends. When the marker image information can be extracted (step S22: Yes), the target device is specified by the device identification information included in the marker image information, and the marker and sensor (from the angle and size of the marker included in the marker image information) The distance and direction from the camera are calculated, and based on this, the display position and orientation of the target device are obtained (step S23), and the marker detection process ends.

  Next, the AR object image creation process will be described in detail.

  FIG. 32 is a flowchart for explaining the algorithm of the AR object image creation processing in step S08 shown in FIG.

  First, the latest display state information is acquired from the RAM unit (step S31), and in addition to the 3D data corresponding to the outer shape of the apparatus main body unit, setting of the optional device selection mode and the operator designation mode included in the display state information is performed. 3D data corresponding to is acquired from the ROM unit (step S32).

  Next, it is determined whether or not “None” is designated in the obstacle mode (step S33). If it is determined that “none” is designated (step S33: Yes), the execution sequence proceeds to step S36.

  When it is determined that “None” is not designated (step S33: No), the position information of the obstacle is acquired based on the scan result of the sensor (step S34), and the 3D data of the obstacle is obtained from the position information. After being generated (step S35), the execution sequence proceeds to step S36.

In step S36, the 3D data of the 3D data and obstacle obtained from the ROM unit, based on the distance or the like to the position data and the reference point P ₁ of the reference point P ₁ of the marker M, AR object consisting of contour image data The AR object image creation process is completed after being appropriately converted into an image.

  For example, if “option device” is set in the option device selection mode, the AR object image shown in FIG. 14 or FIG. 15 is created, and if “superposition” is set in the overlay mode, FIG. 18 is created, and if “transparent” is set, the AR object image shown in FIG. 18 is created. If “child” is set in the operator designation mode, FIG. If the AR object image shown is created and “male” is set, the AR object image shown in FIG. 20 is created, and “display only” or “interference alarm” is set in the obstacle mode. The AR object image shown in FIG. 24 is created.

  Next, the AR object image composition display process will be described in detail.

  FIG. 33 is a flowchart for explaining the algorithm of the AR object image composition display processing in step S09 shown in FIG.

  First, the AR object image and the background video stored in the background video portion of the RAM portion are combined (overlaid) (step S41). At this time, when the obstacle mode included in the display state information is “none”, the AR object image generated from the 3D data of the obstacle is not synthesized.

  Next, the presence / absence of additional information is determined (step S42). The additional information is, for example, image data according to the settings of the size display mode, weight display mode, connector position display mode, and obstacle mode included in the display state information.

  If it is determined that there is no additional information based on the latest display state information (step S42: No), the execution sequence proceeds to step S44. When it is determined that there is additional information (step S42: Yes), additional information is further combined with the combined image, the combined image is displayed on the display 150 (step S43), and the execution sequence proceeds to step S44. .

  For example, if “chassis size” is set in the size display mode, the additional information shown in FIG. 21 is additionally combined. If “installation size” is set, the additional information shown in FIG. 21 is added. If “Yes” is set in the weight display mode, the additional information shown in FIG. 27 is additionally combined. If “Yes” is set in the connector position display mode, the additional information shown in FIG. 28 is added. If information is additionally combined and “interference warning” is set in the obstacle mode, the additional information (warning message) shown in FIG. 25 is additionally combined.

  In step S44, it is determined whether or not there is reproduction data (step S44). When it is determined that there is no reproduction data based on the latest display state information (audio mode, moving image mode, and rotation mode) (step S44: No), the AR object image composition display processing ends.

  When it is determined that there is reproduction data (step S44: Yes), the reproduction of the sound, the start of the moving image, and the rotation of the target device are executed according to the settings of the audio mode, the moving image mode, and the rotation mode (step S45) ) (See FIG. 26), the AR object image composition display processing ends.

  Note that the additional information can be combined (simultaneously) with the AR object image and the background video. If the display supports 3D display, a composite image can be generated in an appropriate 3D format and configured to be stereoscopically viewable. Furthermore, the composite image can be printed and presented to the operator.

  As described above, in the present embodiment, the composite image displayed to the operator includes the image of the target device (the installation status of the target device according to the installation status of the optional device) that is placed at the designated installation position. It is possible to visually grasp the state when the target device is actually installed, and whether the target device is installed properly. When examining whether or not in advance, it is easy to prevent troubles due to calculation mistakes or omissions. Therefore, it is possible to reduce trouble after purchase of the target device.

  The present invention is not limited to the above-described embodiments, and various modifications can be made within the scope of the claims. For example, the target device is not limited to a dedicated printer or MFP. In addition, the simulation device can be configured by a camera-equipped tablet or a goggle type (glasses type) display device (for example, a head-mounted display) that synthesizes and displays an augmented reality and real space (external) scenery. is there. In the case of a goggle type display device, it is necessary to have input means such as an external button, for example, instead of an operation from the touch panel.

10 Target device (product),
90 Installation location,
92 walls,
94 Bookshelf 100 Simulation device,
110 CPU,
120 ROM part,
130 RAM section,
140 touch panel,
150 display,
160 cameras,
170 sensors,
180 speakers,
190 bus,
M marker,
P1 reference point,
P2-4 Measurement points.

Claims (20)

A control program for a simulation device,
A background image acquisition procedure for acquiring an image of the installation location in the real space where the target device of the AR simulation is arranged;
An installation position detection procedure for detecting an installation position of the specified target device;
An AR object generation procedure for generating an AR object of the target device to which the option device is mounted at the detected installation position in accordance with the mounting status of the option device to be mounted on the target device;
An image composition procedure for generating a composite image including the video and the AR object;
A display procedure for displaying the composite image;
A control program causing the simulation apparatus to execute a process having   2. The control program according to claim 1, wherein the AR object is generated in a state where a plurality of the optional devices are overlapped or transmitted in the AR object generation procedure.   The control program according to claim 1, wherein the processing further includes a data reproduction procedure for reproducing audio data, moving image data, or device rotation data prepared in advance. The processing further includes an additional information generation procedure for generating additional information for determining whether or not the installation status of the target device is appropriate,
The control program according to any one of claims 1 to 3, wherein in the image composition procedure, a composite image to which the additional information is added is generated. The installation situation considers the casing size of the target device,
The control program according to claim 4, wherein the case size includes a size of the option device when the option device is mounted on the target device. The installation status takes into account the installation size of the target device,
The installation size is calculated by adding a work area required for actual operation to the housing size of the target device,
The control program according to claim 4, wherein the case size includes a size of the option device when the option device is mounted on the target device.   The control program according to claim 6, wherein the work area is set in consideration of a body shape of an operator of the target device.   The control program according to claim 6, wherein the work area is set in consideration of a connector for connecting to an external peripheral device.   The control according to claim 8, wherein the additional information further includes an insertion direction of the connector, a standard of the connector, a name of a peripheral device connected to the connector, or a combination thereof. program. The processing further includes a detection procedure for detecting interference between the target device and an obstacle located at an installation location of the real space,
The control program according to claim 1, wherein the additional information includes warning information that alerts the interference.   The control program according to claim 10, wherein the warning information includes a message indicating the interference.   In the installation position detection procedure, the installation position of the target device is detected based on a marker arranged at an installation location in the real space. The described control program. A background image acquisition step of acquiring an image of an installation location in a real space where the target device of the AR simulation is arranged;
An installation position detection step for detecting an installation position of the specified target device;
An AR object generation step of generating an AR object of the target device to which the option device is mounted at the detected installation position according to a mounting state of the option device to be mounted on the target device;
An image compositing step for generating a composite image including the video and the AR object;
A display step for displaying the composite image;
A simulation method characterized by comprising:   The simulation method according to claim 13, further comprising a data reproduction step of reproducing audio data, moving image data, or device rotation data prepared in advance. An additional information generation step of generating additional information for determining whether or not the installation status of the target device is appropriate,
The simulation method according to claim 13 or 14, wherein, in the image composition step, a composite image to which the additional information is added is generated. An interference detection step of detecting interference between the target device and an obstacle located at the installation location in the real space,
The simulation method according to claim 13, wherein the additional information includes warning information that alerts the interference. A background image acquisition unit for acquiring an image of an installation location in a real space where the target device of the AR simulation is arranged;
Installation position detection means for detecting the installation position of the specified target device;
An AR object generating means for generating an AR object of the target device to which the optional device is mounted at a detected installation position according to a mounting state of the optional device mounted on the target device;
Image synthesizing means for generating a synthesized image including the video and the AR object;
A display unit for displaying the composite image;
A simulation apparatus comprising:   18. The simulation apparatus according to claim 17, further comprising data reproduction means for reproducing audio data, moving image data, or apparatus rotation data prepared in advance. Additional information generating means for generating additional information for determining whether or not the installation status of the target device is appropriate,
The simulation apparatus according to claim 17 or 18, wherein the image composition unit generates a composite image to which the additional information is added. Interference detection means for detecting interference between the target device and an obstacle located at the installation location of the real space,
The simulation apparatus according to claim 17, wherein the additional information includes warning information that alerts the interference.