JP2017187962A - Device explanatory program using augmented reality technology - Google Patents

Abstract

PROBLEM TO BE SOLVED: To simply and plainly display instruction manuals of a variety of devices such as analyzing device to a user.SOLUTION: A program comprises an explanatory information storage unit 23 that stores explanatory information 30 being information to explain an operation of a target device 10 to correspond to each another code, a marker extraction unit 24 that extracts an AR marker 12 displayed on an external surface of the target device 10 from a photographic image of a portable terminal 20, a code reading unit 25 that reads a code from the extracted AR marker 12, and an AR display unit 26 that displays explanatory information 30 corresponding to the read code by the code reading unit 25 on a screen of a portable terminal 20 by superimposing on a photographic image. The explanatory information 30 may be displayed based on not the AR marker but the geometry of the device.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a program for explaining various devices such as an analysis device using augmented reality (AR) technology.

  When various devices such as an analysis device are newly manufactured, an instruction manual for correctly using the device is created at the same time. When the apparatus is complicated or has many functions, or when an unskilled person is assumed as the user who is the operation target, the content of the instruction manual increases. Some paper (booklet) instruction manuals are several centimeters thick or are divided into several volumes, but the user can read through them as well as read the necessary parts. Even it is becoming difficult.

  Therefore, in recent years, there are an increasing number of electronic manuals such as pdf file formats that can be viewed on a computer. If it is an electronic version instruction manual, it is easy to search for a necessary part, and it is also easy to move immediately from the current reading position to a related part. Furthermore, since moving images and sounds can be inserted, it becomes easier to understand.

  On the other hand, in recent years, a technology called AR (augmented reality) has been widely used in mobile terminal devices such as smartphones and tablets. This is to display various types of information such as other images, moving images, characters, etc., superimposed on the captured image when photographing a scene in front of the user with a portable terminal device (Non-patent Document 1). reference).

@IT, “Basic Knowledge of AR (Augmented Reality) That Can't Be Heared Now”, [online], September 26, 2012, [Search May 8, 2015], Internet <URL: http: // www. atmarkit.co.jp/ait/articles/1109/26/news136.html>

  When providing a new device to the user, by providing the electronic version of the instruction manual at the same time, the user can easily operate the device even if the device is complex and has abundant functions as described above. Can understand and grasp. However, since the electronic version of the instruction manual can only be viewed on a computer, it can be conveniently used in the case of a device connected to a computer and controlled and operated on the computer. In the case of a stand-alone device that is not connected, a separate computer must be prepared. For example, a stand-alone analyzer is excellent in portability and quick start-up, so it is suitable for on-site analysis. In such a case, the electronic version instruction manual is not a suitable explanation medium. .

  The problem to be solved by the present invention is to provide a device explanation program that can easily explain the operation of a stand-alone device not connected to a computer to a user.

A first aspect of the present invention made to solve the above problems is a program that operates on a portable terminal having an image capturing function,
a) an explanation information storage unit that stores explanation information, which is information explaining various operations of the target device, in association with a code for each operation;
b) a marker extraction unit that extracts an AR marker displayed on the outer surface of the target device from a captured image of the mobile terminal;
c) a code reading unit for reading a code from the extracted AR marker;
d) An AR display unit that reads out the description information corresponding to the code read by the code reading unit from the description information storage unit, and displays it on the screen of the portable terminal superimposed on the captured image.

  The first aspect of the present invention corresponds to a so-called marker-type vision base AR, and reads a specific marker called an AR marker from an image taken by a mobile terminal, and various information such as an image, a moving image, and characters corresponding thereto. Is superimposed on the captured image of the mobile terminal. In the first aspect of the present invention, it is assumed that such an AR marker is displayed on the outer surface of the target device. This “AR marker displayed on the outer surface of the target device” is set for each target device. It may be fixed, or may be variable according to the internal state of the target device. In the former case, operation information corresponding to the target device is displayed as an AR on the screen of the mobile terminal. In the latter case, in addition, the AR marker changes depending on the internal state of the target device (for example, when the target device is an analysis device, preparation for analysis / during analysis / end of analysis, etc.) Information to be operated by the user at that time is displayed on the screen.

A second aspect of the present invention is a program that operates on a portable terminal that also has an image capturing function,
a) Appearance pattern storage unit that stores the appearances of a plurality of configurations that can be taken by the target device by patterning them using a predetermined method,
b) a configuration information storage unit that stores, in association with each pattern of the appearance, configuration information that is information related to the configuration of the target device corresponding to the pattern;
c) An appearance patterning unit that extracts an appearance of the target device from a captured image of the mobile terminal and patterns the target device by the predetermined method;
d) An AR display unit that reads configuration information corresponding to the pattern patterned by the appearance patterning unit from the configuration information storage unit and displays the configuration information on the screen of the mobile terminal superimposed on a captured image. To do.

  The second aspect of the present invention corresponds to a so-called markerless vision-based AR, and the AR marker may not be displayed on the target device. In the second aspect of the present invention, the appearance patterning unit extracts the appearance of the target device from the captured image and patterns it by a predetermined method. Known image processing techniques such as color difference extraction and brightness difference extraction can be used to extract the target device from the captured image, and image processing techniques such as binarization and contour extraction are used to pattern the extracted image. Can be used. The AR display unit compares the pattern of the appearance of the target device thus patterned with each pattern stored in the configuration information storage unit, detects a matching pattern, and transmits the configuration information corresponding to the matched pattern to the portable terminal. Is superimposed on the captured image. Since the configuration information can include operation information corresponding to the configuration, the user can easily obtain operation information regarding the target device having the configuration.

  According to the first aspect of the program of the present invention, when the target device is photographed using the image photographing function of the portable terminal, the explanation information according to the AR marker code displayed on the outer surface of the subject device is Are appropriately superimposed on the image of the target device displayed on the screen. Therefore, the user can easily obtain appropriate instructions and information corresponding to the current status of the target device without the trouble of searching for and referring to related description locations from a paper or electronic version of the manual.

  According to the second aspect of the program of the present invention, when the target device is photographed using the image photographing function of the portable terminal, configuration information corresponding to the appearance of the target device is displayed on the screen of the portable terminal. The image is appropriately superimposed on the image of the target device. Accordingly, the user can easily obtain the current configuration of the target device and appropriate instructions and information corresponding to the configuration without having to search for and refer to related descriptions from paper or electronic manuals. it can.

The block diagram and operation | movement schematic diagram of the apparatus description program which concern on 1st Embodiment. The figure which shows the other form of AR marker. An example in which explanatory information is displayed on a target device in a captured image, (a) an example in which characters and images are combined, and (b) an example in which a moving image is displayed. The block diagram and operation | movement schematic diagram of the apparatus description program which concern on 2nd Embodiment. An example of a patterned appearance.

Hereinafter, a specific example of the operation of the apparatus explanation program according to the present invention will be described in detail as an embodiment.
In any embodiment, the program according to the present invention is installed in the mobile terminal 20 in advance.
Various controls of the analysis apparatus 10 are performed by a user inputting various commands from an input unit (keyboard, touch panel, etc.) on the main body of the analysis apparatus 10. It is also possible to operate from the terminal 20 wirelessly. In that case, a program (application software) for that purpose is installed in the mobile terminal 20 in advance.

FIG. 1 is a configuration diagram and operation schematic diagram of a device explanation program according to the first embodiment. In the present embodiment, the analysis device 10 that is the target device has an AR marker 12 that indicates the state of the analysis device 10 on the front surface of the device. The AR marker 12 is provided with a black frame of a predetermined thickness, and displays a pattern, characters, etc. (code) corresponding to the internal state of the apparatus inside the black frame. From an LCD (Liquid Crystal Display) or the like Can be configured.
As another form of the AR marker 12, for example, as shown in FIG. 2, a combination of lighting and extinguishing of a plurality of lamps made of LEDs or the like can be used. When there are 16 lamps as shown in FIG. 2, the device state of the 2 16th power case can be expressed. Not only lighting / extinguishing, but also device status can be expressed using color.

  The mobile terminal 20 includes a central processing unit (CPU) that is a central processing unit, a memory, a mobile terminal screen (display unit) 21 including a touch panel LCD, and a mass storage device such as an SSD (Solid State Drive). It consists of a storage unit and the like. In addition, the mobile terminal 20 includes a camera that can capture a still image and a moving image, and an image capturing function that controls the operation of the camera.

  In addition to an OS (Operating System) that controls the overall operation of the mobile terminal 20, the storage unit of the mobile terminal 20 stores an apparatus explanation program 22 according to the present invention and other various programs (application software). .

  As shown in FIG. 1, the apparatus explanation program 22 includes an explanation information storage unit 23, a marker extraction unit 24, a code reading unit 25, and an AR display unit 26. These are basically functional means realized by software by the CPU of the portable terminal 20 executing the device explanation program 22.

  In the explanation information storage unit 23, explanation information 30, which is information explaining an operation according to the operation state of the target apparatus (analysis apparatus 10) (analysis preparation, analysis, and analysis end) is associated with the AR marker type. Are recorded. The explanation information 30 may additionally record related information such as a display of a warning / warning of the analyzer 10 according to the operating state of the apparatus and the performance of the apparatus. In addition, the classification of the operation state is not limited to the above, but another classification such as when the apparatus is activated may be added, or the above classification may be further subdivided, for example, setting an item being prepared for analysis as a sample The analysis time may be set.

  The operation when the user actually executes the device explanation program 22 will be described below.

  The analysis apparatus 10 sequentially changes the display of the AR marker 12 according to the internal state of the apparatus. The user executes the apparatus explanation program 22 on the portable terminal 20 and then photographs the analysis apparatus 10 using the image photographing function. This may be a moving image or a still image.

In response to this photographing operation, the marker extraction unit 24, from the photographed image (the frame in the still image or the moving image), the AR marker 12 displayed on the outer surface of the analysis device 10, that is, the black frame and the pattern included therein. To extract.
Next, the code reading unit 25 reads a code that is a pattern included in the extracted AR marker 12.

  Thereafter, the AR display unit 26 compares the code read by the code reading unit 25 with the code stored in the description information storage unit 23, and reads the description information 30 corresponding to the read code from the description information storage unit 23. And superimposed on the captured image and displayed on the mobile terminal screen 21.

  The explanation information 30 may be character information as shown in FIG. 1, or as shown in FIG. 3, for example, a graphic such as an arrow or a circle is used as the explanation information image 31, and the predetermined information of the analysis apparatus 10 in the captured image. You may display in the part of. As shown in FIG. 3A, a combination of both is naturally possible. Further, an image such as an arrow may be displayed on the screen as a moving image. Further, as shown in FIG. 3B, an operation explanation moving image previously captured may be superimposed and displayed as a description information moving image 32 on part or all of the terminal screen.

  In addition, in order to display the description information 30 at a predetermined location of the analysis apparatus 10 in the captured image, the display location may be specified by executing image recognition processing, or a predetermined relative position from the AR marker 12 The description information 30 may be displayed on the screen.

In the above-described embodiment, the processing from the extraction of the AR marker 12 to the display of the description information 30 is performed on a still image when the captured image is a still image. The processing may be performed every time or every certain frame interval.
When the above processing is performed on a moving image captured in real time, even when the analysis apparatus 10 is operated by the user and the apparatus state changes, the AR marker 12 that has changed according to the state of the apparatus is sequentially read. Therefore, it is possible to always display the description information corresponding to the device state on the mobile terminal screen. Therefore, the user can obtain updated explanation information without operating the mobile terminal.

  Next, an apparatus explanation program according to the second embodiment will be described with reference to FIG. 4 which is a configuration diagram and an operation schematic diagram thereof.

  In the present embodiment, the analysis device 10 that is the target device can be replaced with accessories, and the appearance changes depending on the type of the accessory.

As in the case of the first embodiment described above, the mobile terminal 20 includes a CPU, which is a central processing unit, a mobile terminal screen (display unit) 21 including a memory, a touch panel LCD, and a large-capacity storage device such as an SSD. It is comprised from the memory | storage part etc. which consist of. In addition, the mobile terminal 20 includes a camera that can capture a still image and a moving image and an image capturing function that controls the operation of the camera.
In addition to the OS, the storage unit of the portable terminal 20 stores a device explanation program 42 according to the present invention and other various programs.

  As shown in FIG. 4, the apparatus explanation program 42 according to the present embodiment includes an appearance pattern storage unit 43, a configuration information storage unit 44, an appearance patterning unit 45, and an AR display unit 46. These are basically functional means realized by software when the CPU of the portable terminal 20 executes the device explanation program 42.

  In the appearance pattern storage unit 43, a plurality of appearances that can be configured by the analyzer 10 are recorded in a pattern by a predetermined method. FIG. 5 shows three examples of such patterned appearances. These external appearances are created by applying a combination of processing such as contour extraction and binarization techniques used in normal image recognition preprocessing to an image of the upper and side surfaces of the apparatus. Further, although all the different parts are displayed one by one in FIG. 5, a plurality of external appearances having different shooting angles may be stored for one component. Alternatively, the 3D model of the device may be stored in the appearance pattern storage unit 43, and the patterning process may be performed every time the user takes a picture.

  In the configuration information storage unit 44, configuration information 50, which is information related to the configuration of the analyzer 10 corresponding to each pattern of the appearance, is recorded in association with each other. This configuration information 50 is information relating to the configuration of accessories attached to the apparatus. In addition, information related to operations in the configuration may be included.

  Hereinafter, an operation when the user actually executes the apparatus explanation program 42 will be described.

  The user executes the device explanation program 42 in the portable terminal 20 and then photographs the analysis device 10 using the image photographing function. This may be a moving image or a still image.

  In response to this photographing operation, the appearance patterning unit 45 extracts the appearance of the target device from the photographed image (a frame in the still image or moving image) of the mobile terminal 20 and patterns it by the predetermined method.

Next, the AR display unit 46 compares the appearance data patterned by the appearance patterning unit 45 with the patterned appearance data stored in the appearance pattern storage unit 43, and the appearances match or are most similar. The pattern is determined as the configuration of the analyzer 10. Then, the configuration information 50 corresponding to the pattern corresponding to the analysis device 10 is read from the configuration information storage unit 44 and superimposed on the captured image and displayed on the mobile terminal screen 21 (FIG. 4).
The display position of the configuration information 50 displayed by the AR display unit 46 may be a place unrelated to the outer shape of the apparatus. For example, an arrow or a figure such as a circle is displayed at a predetermined position of the analysis apparatus 10 in the captured image. You may do it. Of course, a combination of both is also possible. In addition, as in the first embodiment, the apparatus configuration and operation information related thereto may be displayed as a moving image on part or all of the screen.
Also in the present embodiment, as in the first embodiment, when the captured image is a still image, the above processing is performed every time the analyzer 10 is captured, and when the moving image is a captured image, every frame or constant The above processing may be performed at every frame interval.

  In the first and second embodiments, the description information 30 and the configuration information 50 are displayed so as to be superimposed on the captured image of the mobile terminal 20, but the description information 30 or the configuration information 50 is not superimposed on the captured image. It may be displayed alone.

  In the above, the analysis apparatus is taken as an example. However, the system according to the present invention is not limited thereto, and can be applied to various apparatuses such as a personal computer, an automobile, and a truck.

DESCRIPTION OF SYMBOLS 10 ... Analytical apparatus 12 ... AR marker 20 ... Portable terminal 21 ... Portable terminal screen 22 ... Device explanation program 23 ... Explanation information storage part 24 ... Marker extraction part 25 ... Code reading part 26 ... AR display part 30 ... Explanation information 31 ... Explanation information image 32 ... Explanation information moving picture 42 ... Device explanation program 43 ... Appearance pattern storage section 44 ... Configuration information storage section 45 ... Appearance patterning section 46 ... AR display section 50 ... Configuration information

Claims (4)

A program that operates on a portable terminal having an image capturing function,
a) an explanation information storage unit that stores explanation information, which is information explaining various operations of the target device, in association with a code for each operation;
b) a marker extraction unit that extracts an AR marker displayed on the outer surface of the target device from a captured image of the mobile terminal;
c) a code reading unit for reading a code from the extracted AR marker;
d) An apparatus description including an AR display unit that reads out the description information corresponding to the code read by the code reading unit from the description information storage unit and displays it on the screen of the portable terminal in a superimposed manner on the captured image. Program.   The apparatus explanation program according to claim 1, wherein the AR display unit superimposes and displays the explanation information at a predetermined position related to the target device in the captured image. A program that operates on a portable terminal having an image capturing function,
a) Appearance pattern storage unit that stores a plurality of appearances that can be configured by the target device by patterning them with a predetermined method;
b) a configuration information storage unit that stores, in association with each pattern of the appearance, configuration information that is information related to the configuration of the target device corresponding to the pattern;
c) An appearance patterning unit that extracts an appearance of the target device from a captured image of the mobile terminal and patterns the target device by the predetermined method;
d) An AR display unit that reads configuration information corresponding to the pattern patterned by the appearance patterning unit from the configuration information storage unit and displays the configuration information on the screen of the mobile terminal superimposed on a captured image. To explain the device.   The apparatus explanation program according to claim 3, wherein the AR display unit superimposes and displays the configuration information at a predetermined position related to the target device in a captured image.

Images