JP2019133280A - Information processing system, information processing method and program - Google Patents

Abstract

To improve security more by performing personal identification with authentication information including fingerprint authentication on a user and then superimposing superimposition data, matched with user's authority, on a recording medium during photography, for display.SOLUTION: An information processing system comprises: a host PC 10 which generates print data to which a marker image indicative of display conditions of an image having been printed is imparted; an image formation apparatus 20 which prints the print data on a recording medium; a smart terminal 40 which has an imaging part 50 and an input part for inputting user authentication information including fingerprint authentication information; and a server 30 which determines, when the print image printed on the recording medium is photographed by the smart terminal 40, display authority based upon display conditions linked to the marker image and user authentication information on the smart terminal 40, and then changes an image to be displayed at a display part 46 that the smart terminal 40 has according to the determination.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an information processing system, an information processing method, and a program.

  In recent years, AR (Augmented Reality) technology has been provided in which auxiliary information is presented when a user recognizes the real world, thereby presenting information beyond the reality in front of the user. In conventional information processing systems, the AR technology is used to recognize the state pattern printed by the image forming device using a smart terminal, and to perform FAQ (Frequently Asked Questions) and error cancellation. Regarding the functions to be supported and the installation of the image forming apparatus, a technology for making a sales promotion tool that can be imaged by visual observation through a smart terminal is already known.

  Patent Document 1 discloses an image forming system for the purpose of providing a user who provides a document such as a printed material to a person who has viewed the document easily with additional information about the document. Specifically, the image forming system includes an image forming apparatus, a server, and a mobile terminal. The image forming apparatus identifies the augmented reality additional information from the basic document and the document obtained by adding the augmented reality additional information to the basic document, generates a tag corresponding to the augmented reality additional information, and associates the tag with the tag. The augmented reality additional information is registered in the server, and an image of the basic document with the tag added is output. The portable terminal captures the image of the basic document with the tag output from the image forming apparatus, extracts the tag from the captured image, and adds the augmented reality additional information associated with the tag from the server. The augmented reality additional information is acquired and superimposed on the captured image.

  However, in the AR technology in the conventional information processing system, the state of the image forming apparatus is detected and an image obtained by combining information representing the state is displayed on the screen of the smart terminal. Since the additional information for augmented reality can be browsed uniformly by anyone, there is a problem that security is low. Patent Document 1 discloses that augmented reality additional information is added when a document is visually recognized through a mobile terminal (smart terminal). However, since anyone can browse the augmented reality additional information uniformly, the security is low. In this case, it is conceivable to improve the security with an account of a specific authorized person. However, in this case, since there is a vulnerability due to impersonation of the person, the identification of the user himself / herself is required.

  The present invention has been made in view of the above, and after performing identity verification using authentication information including user's fingerprint authentication, superimposition data according to the user's authority is displayed superimposed on the recording medium being shot. The purpose is to further improve security.

  In order to solve the above problems, the present invention provides an information processing terminal for creating print data to which a marker image indicating a display condition of an image after printing is added, an image forming apparatus for printing the print data on a recording medium, A portable terminal having an imaging unit and an input unit for inputting user authentication information including fingerprint authentication information, and when the printed image printed on the recording medium is photographed by the portable terminal, the marker image is associated with the marker image. A display authority is determined based on display conditions and the user authentication information of the mobile terminal, and a server that changes an image displayed on the display unit of the mobile terminal according to the determination is provided.

  According to the present invention, after verifying the identity with authentication information including the user's fingerprint authentication, it is possible to superimpose data that matches the user's authority on the recording medium being photographed, and to display it more securely. The effect that can be improved.

FIG. 1 is a diagram illustrating an overall configuration of an information processing system. FIG. 2 is a functional block diagram of the information processing system. FIG. 3 is a diagram illustrating an example of the AR marker. FIG. 4 is a flowchart illustrating an example of the operation of the smart terminal. FIG. 5 is a diagram illustrating coordinate representation in the smart terminal. FIG. 6 is a diagram illustrating perspective projection onto the camera virtual screen by the smart terminal. FIGS. 7A to 7D are diagrams illustrating images in which virtual object images are superimposed by the processing of the smart terminal. 8A and 8B are image diagrams of AR markers included in the print data according to the embodiment of the present invention. FIG. 9 is a diagram in which AR marker images related to examples of corresponding words and classification thereof according to the embodiment are classified. FIGS. 10A to 10E are display image diagrams of the smart terminal according to the embodiment. FIG. 11 is a flowchart illustrating processing of the host PC according to the embodiment. FIG. 12 is a diagram illustrating an AR information creation screen of the host PC according to the embodiment. FIG. 13 is a diagram showing an AR marker image selection screen of the host PC according to the embodiment. FIG. 14A is a diagram showing a duplicate warning message when the AR marker of the host PC according to the embodiment is not explicitly shown, and FIG. 14B explicitly shows the AR marker of the host PC according to the embodiment. FIG. 14C is a diagram showing an AR marker registration screen of the host PC according to the embodiment. FIG. 15 is a diagram illustrating a message for prompting reselection of the AR marker of the host PC according to the embodiment. FIG. 16 is a diagram illustrating an AR marker rearrangement image of the host PC according to the embodiment. FIG. 17 is a flowchart illustrating processing of the image forming apparatus according to the embodiment. FIG. 18 is a flowchart showing processing of the smart terminal according to the present embodiment. FIG. 19 is a diagram illustrating a message when the AR information is not received from the server in the smart terminal according to the embodiment. FIG. 20 is a flowchart illustrating processing of the server (first server) according to the embodiment. FIG. 21 is a diagram illustrating a message when there is no user ID transmitted from the server (first server) according to the embodiment. FIG. 22 is a flowchart illustrating processing of the server (first server) according to the embodiment. FIG. 23 is a flowchart illustrating processing of the server (second server) according to the embodiment. FIG. 24 is a flowchart illustrating mode processing of the server (second server) according to the embodiment. FIG. 25 is a diagram illustrating a message for prompting fingerprint authentication according to the embodiment.

  Hereinafter, the present invention will be described in detail with reference to embodiments shown in the drawings. The present invention has the following configuration in order to improve the security by displaying superimposed data in accordance with the authority of the user on the recording medium being shot.

  That is, an information processing system of the present invention includes an information processing terminal that creates print data to which a marker image indicating a display condition of an image after printing is attached, an image forming apparatus that prints the print data on a recording medium, and an imaging unit And a portable terminal having an input unit for inputting user authentication information including fingerprint authentication information, and a display condition associated with a marker image when a print image printed on a recording medium is photographed with the portable terminal, and the portable A display authority is determined based on the user authentication information of the terminal, and a server that changes an image displayed on the display unit of the portable terminal according to the determination is provided.

  By providing the above configuration, after verifying the identity of the user by fingerprint authentication, the superimposed data that matches the user's authority can be displayed superimposed on the recording medium being shot, further improving security. Can be made. Hereinafter, the features of the present invention will be described in detail with reference to the drawings.

  First, the AR technology will be described. Augmented reality is a variant of virtual reality (VR), which means that information is added, deleted, emphasized, or attenuated in the real environment surrounding the environment, literally extending the real world as seen by humans. While virtual reality replaces reality with artificially constructed reality, augmented reality is a technology that alters part of reality. Virtual reality, for example, presents the five senses to a person as if they were in a virtual room and looking at a virtual teapot placed on a virtual table, whereas in augmented reality, the reality that a person is actually present Information is presented as if a virtual teapot is placed on the table in the room.

  It is sometimes called augmented reality or augmented reality because it adds computer information to provide human information with real information that is strengthened or augmented. Since the real environment is used as an information source, real-time processing is necessarily required in many cases. As a technical problem, in addition to the virtual reality, in particular, it is a problem to solve the interaction between the virtual object image and the real environment / object, such as lighting, shadow, contact, concealment, etc. in the visual element.

  As a means of extending the reality of computers, information presentation for all human sense organs and somatosensory sensations such as sight, hearing and touch has been attempted. In virtual reality, importance is attached to the reality of a virtual object image to be presented to a person, whereas in augmented reality, relevance to contexts such as positions and objects in the real world is also emphasized. In a system that places emphasis on the latter, a simple information presentation method may be used as a means of extending reality, for example, explaining a place or an object by text or voice.

  In many systems, devices or technologies widely used in virtual reality are used to present and acquire information. For example, a head mounted display is mainly used for visual information presentation. However, presentation using a screen of a small information terminal such as a mobile phone is also being studied with the intention of supporting work in a real environment. Since the subject of the presented environment is the real environment, work support in the real environment is expected as an application field. For example, applied research such as providing route guidance information, providing technical information to engineers who perform maintenance of aircraft and copiers, and presenting information for surgical support in the medical field has been conducted.

  A common method for creating services for general consumers is to use an image recognition routine such as ARTToolKit to create 2D code patterns and still images (called "AR markers") with digital cameras. When a picture is taken, there is a case where a 3DCG animation using the AR marker as an AR marker is combined and displayed in real time, or a content suitable for a still image is displayed.

  Next, an AR implementation method will be described. When the AR technique is roughly divided, it can be divided into two due to technical characteristics. One is called “location-based AR”, which presents information using position information that can be acquired from GPS, etc. The other is directly in front of the eyes by applying techniques such as image recognition and space recognition. It is called “vision-based AR” that presents information by recognizing and analyzing the environment in

<Information processing system>
The overall configuration of the information processing system 1 will be described with reference to FIG. FIG. 1 is a diagram showing the overall configuration of the information processing system 1. The information processing system 1 is configured on a network environment in which a host PC 10, an image forming apparatus 20, servers 30, 70, and a smart terminal 40 are connected via a network N (Internet or LAN (Local Area Network)). Note that each device connected by the network N is not limited to one device, but may be configured by a plurality of devices. The server 30 as the first server is illustrated as the server 1, and the server 70 as the second server is illustrated as the server 2.

  Next, a functional block diagram of the information processing system will be described with reference to FIG. FIG. 2 is a functional block diagram of the information processing system.

<Host PC>
The host PC 10 shown in FIG. 2 includes a print data creation unit 12, an AR information creation unit 14, and a communication unit 16. The host PC receives print data processing (deletion of confidential data, etc.), AR display condition information, designation of an AR marker to be used, print condition designation, print instruction, and the like by the user. The print data creation unit 12 assigns an AR marker to the print data based on the print condition and the result of analyzing the print data. The AR information creation unit 14 creates AR information for associating original print data before processing with AR display condition information indicating a security level. The communication unit 16 transmits print data to the image forming apparatus 20 via the network N. In addition, the communication unit 16 transmits original print data and AR information to the server 30 via the network N. Here, a plurality of AR markers to be used may be designated, and a default AR marker may be selected when not designated.

<Image forming apparatus>
The image forming apparatus 20 illustrated in FIG. 2 includes a communication unit 22 and a print data output unit 24. The communication unit 22 receives print data from the host PC 10 via the network N. The print data output unit 24 prints print data instructed by the host PC 10 on a recording medium.

<Server 1 (first server)>
A server 30 as a first server shown in FIG. 2 includes a display data creation unit 32, a communication unit 34, an original print data storage unit 36, and an AR information storage unit 38. The display data creation unit 32 processes the original print data based on the AR marker information and the AR information to generate processed data. The communication unit 34 transmits the processed data to the smart terminal 40 via the network N. The original print data storage unit 36 stores original data, AR markers, and security level information. The AR information storage unit 38 includes an identifier table, a security table, and a comparison table, and stores AR information corresponding to each identifier and each user ID. The user ID stored in the AR information storage unit 38 can be rewritten.

<Server 2 (second server)>
A server 70 as a second server illustrated in FIG. 2 includes a communication unit 71, a user ID conversion unit 72, and group data 73. The communication unit 71 notifies the smart terminal 40 of the group information to which the user converted by the user ID conversion unit 72 belongs. The user ID conversion unit 72 converts the user ID into group information to which the user belongs. The group data 73 holds user group information (see Tables 5 and 6 to be described later) and fingerprint authentication information for each user (see Table 7 to be described later).

<Smart terminal>
A smart terminal 40 as a portable terminal includes a communication unit 42, an operation unit 44, a display unit 46, a storage unit 48, a fingerprint authentication unit 49, an imaging unit 50, and a control unit 52. The communication unit 42 communicates information with an external device such as a host PC 10 (Personal Computer) via the network N, and can be realized by a communication device such as a NIC (Network Interface Card).

  The operation unit 44 as an input unit inputs various types of information required by application software in accordance with user operations, and can be realized by an input device such as a touch panel or a key switch. The display unit 46 displays various screens and can be realized by a display device such as a liquid crystal display and a touch panel display. The storage unit 48 stores various application software programs executed on the smart terminal 40 and data used for various processes executed on the smart terminal 40. The storage unit 48 can store information magnetically, optically, and electrically, such as a hard disk drive (HDD), a solid state drive (SSD), a memory card, an optical disk, and a random access memory (RAM). This can be realized by any of the storage devices.

  The fingerprint authentication unit 49 acquires the presence or absence of the user's fingerprint authentication from the fingerprint authentication device (part of the input unit) mounted on the smart terminal 40 and holds it. The imaging unit 50 generates an image by optically reading a document in accordance with an instruction from the control unit 52. For example, the imaging unit 50 can be realized by a digital camera as an example of a reading device.

  The control unit 52 controls each unit of the smart terminal 40 and can be realized by a CPU (Central Processing Unit), an LSI (Large Scale Integration), or the like. The control unit 52 includes, for example, a ROM (read only memory), a RAM, and a CPU. The control unit 52 reads the operating system OS from the ROM, expands the program on the RAM, starts the OS, and manages the program from the ROM under OS management. (Processing module) is read and various processes are executed.

  Further, the control unit 52 includes an image acquisition unit 54, an information acquisition unit 56, a calculation unit 58, and a replacement unit 60. The image acquisition unit 54 acquires an image including an AR marker used for augmented reality processing. The image acquisition unit 54 optically reads a document on which an AR marker is printed on a recording medium, and acquires an image including the AR marker. However, the image acquisition unit 54 may acquire an image including the AR marker from the outside (for example, a host PC 10 (an example of an information processing apparatus) connected via the network N).

  The information acquisition unit 56 acquires, from the AR marker in the image acquired by the image acquisition unit 54, first information regarding the size of the AR marker in the real world embedded in the AR marker before printing the image. . Specifically, the information acquisition unit 56 detects an AR marker from the image acquired by the image acquisition unit 54, and acquires first information embedded in the detected AR marker.

  The calculation unit 58 calculates (estimates) the relative position and orientation of the camera using the coordinates of the four vertices. The calculation unit 58 obtains the relative position / orientation of the camera by obtaining the transformation from the coordinates of the four vertices arranged in a square in the three-dimensional marker coordinate system to the coordinate system of the two-dimensional camera virtual screen S ′. calculate. The replacement unit 60 arranges in the three-dimensional space of the marker coordinate system based on the relative position / posture of the camera and the second information (representing the size in the real world).

<AR marker>
Next, the AR marker will be described with reference to FIG. FIG. 3 is a diagram illustrating an example of the AR marker. The AR marker 200 includes an external area 201 and an internal area 202. The external area 201 is used to detect the AR marker 200, and the internal area 202 is used to embed first information. In the example illustrated in FIG. 3, the AR marker 200 has a square shape, but is not limited thereto, and may be a rectangular shape or a circular shape as long as the shape is known and can be detected. Good. The color of the AR marker 200 is preferably monochrome from the viewpoint of image processing, but is not limited thereto, and may be a color.

  Here, processing by the information acquisition unit 56 of the smart terminal 40 will be described. For example, the information acquisition unit 56 binarizes the image acquired by the image acquisition unit 54, performs a labeling process (a process in which adjacent pixels of the same color are combined into one block), and blocks of white pixels and black pixels. To separate. And the information acquisition part 56 performs the process which detects four vertices on the outline with respect to each lump of the separated black pixels.

  As a result, the cluster of black pixels from which the four vertices are detected becomes the external area of the AR marker 200, so the information acquisition unit 56 can detect the AR marker 200 from the image acquired by the image acquisition unit 54, and the AR marker The first information can be acquired from the 200 internal areas 202. When the shape of the AR marker 200 is a circle, the information acquisition unit 56 may perform hub conversion or the like to detect a black pixel block constituting the circle.

  The first information represents the size of the AR marker in the real world before printing (copying) the image. More specifically, the first information is a combination of the colors of the constituent elements constituting the predetermined area in the AR marker. Represents the size of the AR marker in the real world. For example, in the case of the AR marker 200 shown in FIG. 3, the size of the AR marker before printing the image in the real world is represented by a combination of the colors of the pixels constituting the internal region 202.

  In the AR marker 200 shown in FIG. 3, the internal region 202 is composed of, for example, 6 × 6 (36) pixels, and is assumed to be 0 when the pixel color is white and 1 when the pixel color is black. Thus, the size of the AR marker in the real world before printing the image is represented by a 36-bit binary number. However, among the 36 bits (pixels), the values of the pixels at the four corners of the internal region 202 are the values of the image (AR marker) so that the AR marker 200 can be detected even if the image rotates in any direction, up, down, left, or right. 200).

  Of the remaining 32 bits, the first 8 bits are used as an identifier for identifying a 3D virtual object image arranged in a three-dimensional space based on the AR marker 200 used in augmented reality processing. Therefore, in the example illustrated in FIG. 3, the length of one side of the AR marker 200 can be expressed by 24 bits (pixels), and when the length of one side of the AR marker 200 is described in units of 1 mm, 1 mm It can be described in a range of ˜16777216 mm. However, when adding a checksum or the like to improve robustness, the description range of the length of one side of the AR marker 200 becomes smaller than this.

<Operation explanation of smart terminal>
Next, the operation of the smart terminal 40 will be described with reference to FIG. FIG. 4 is a flowchart showing an example of the operation of the smart terminal 40. It is assumed that an image including the AR marker 200 is printed on a document (printed material) printed on a recording medium by the image forming apparatus 20.

  In FIG. 4, first, in step S <b> 201, the imaging unit 50 captures a document on which an image including the AR marker 200 is printed, and the image acquisition unit 54 acquires the image. Subsequently, in step S203, the information acquisition unit 56 extracts the AR marker 200 from the acquired image. Subsequently, in step S <b> 205, the information acquisition unit 56 acquires the second information embedded in the AR marker 200 and the identifier of the 3D virtual object image embedded in the AR marker 200 from the extracted AR marker 200. .

  Subsequently, in step S207, the calculation unit 58 calculates the relative position and orientation of the camera using the coordinates of the four vertices detected in step S203 (for details, refer to the method described in the information acquisition unit 56) ( presume. Specifically, the calculation unit 58 obtains coordinates converted from the coordinates of four vertices arranged in a square in the three-dimensional marker coordinate system to the coordinate system of the two-dimensional camera virtual screen S ′, Calculate the relative position and orientation of the camera.

  As a method for detecting the relative position / orientation between the AR marker 200 and the camera, “Development of Augmented Reality System Construction Tool ARToolKit” IEICE Technical Committee, pp 79-86, 2002-02 ”is known. ing. In addition, the marker coordinate system is often a global coordinate system when a virtual object image is finally arranged. Here, since it is known that the four vertices are on the same plane, it is assumed that the center of the AR marker 200 is (x, y, z) = (0, 0, 0) in the three-dimensional marker coordinate system. The coordinates of the four vertices M0 to M3 are M0 = (− a, −a, 0), M1 = (a, −a, 0), M2 = (− a, a, 0), M3 = (a, a, 0) (see FIG. 5).

  Here, the coordinate expression in the smart terminal 40 will be described with reference to FIG. FIG. 5 is a diagram illustrating coordinate representation in the smart terminal 40. On the other hand, when the replacement unit 60 performs three-dimensional coordinate transformation by performing rotation and parallel movement to be obtained, the coordinates of the four vertices M0 to M3 of the smart terminal 40 are the coordinates of the three-dimensional camera coordinate system. Is converted to Then, by performing perspective projection from the three-dimensional camera coordinate system to the camera virtual screen S ′, two-dimensional coordinate values M0 ′ to M3 ′ are obtained (see FIG. 6). At this time, the parameters of rotation and translation correspond to the relative position and orientation of the camera.

  Here, with reference to FIG. 6, the perspective projection on the camera virtual screen S ′ by the smart terminal 40 will be described. FIG. 6 is a diagram showing perspective projection onto the camera virtual screen S ′ by the smart terminal 40.

  Subsequently, in step S <b> 209, the communication unit 42 transmits the identifier of the 3D virtual object image embedded in the AR marker 200 to the server 30 via the network N. Then, the information acquisition unit 56 acquires a 3D virtual object image (superimposition data) from the server 30 via the communication unit 42. Then, the replacement unit 60 arranges in the three-dimensional space of the marker coordinate system based on the relative position / posture of the camera and the second information (representing the size in the real world). Here, since the replacement unit 60 uses the second information (representing the size in the real world) for the arrangement of the 3D virtual object image, the replacement unit 60 arranges the 3D virtual object image in a size according to the real world. be able to.

  Subsequently, in step S211, the display unit 46 draws an image when the 3D virtual object image arranged in the three-dimensional space is perspective-projected on the screen S, and superimposes the image acquired in step S201. For example, the smart terminal 40 draws a 3D virtual object image arranged in a three-dimensional space using a 3D programming API such as OpenGL or Direct3D on the image acquired in step S201.

  Subsequently, in step S213, the display unit 46 displays an image (see FIG. 7) on which the virtual object image is superimposed on a display or the like.

  The first information is information associated with the first size information indicating the size of the AR marker 200 in the real world before the image is printed, and the second information is the reality of the AR marker 200 after the image is printed. It is good also as information linked | related with the 2nd magnitude | size information which shows the magnitude | size in the world.

The control unit 52 uses image editing processing or image composition processing on the image captured by the imaging unit 50 (camera), and adds some information to the image. At this time, the configuration of the image captured by the imaging unit 50 or the AR marker 200 is detected to determine information to be added to the image, and the information to be added is the current position information. Switching display can be performed.

  The control unit 52 repeatedly performs the above processing (steps S207 to S213) about 60 times per second and displays the superimposed image in real time by displaying the screen. The number of repetitions is changed by detecting the CPU performance according to the table shown in Table 1. Thereby, it is possible to switch the number of repetitions related to the processing from extracting the identifier to displaying the AR information according to the performance of the CPU.

  The parameters of the table shown in Table 1 can be changed by the user. Further, the number of repetitions per second may be directly input by the user via the operation unit 44. Thereby, it can switch to the repetition frequency which a user desires.

<Display image on smart terminal>
Next, an image displayed on the smart terminal 40 will be described with reference to FIGS. FIGS. 7A to 7D are views showing images in which virtual object images are superimposed by the processing of the smart terminal 40. FIG.

  By applying the replacement process shown in FIG. 6 to the “original photographed image” image 300a shown in FIG. 7A, the superimposed characters of the “AR superimposed image (superimposed portion)” image 300b shown in FIG. As in (".confidential information A1" to ".confidential information D2") (superimposed data), it is possible to synthesize a superimposed image with a captured image.

  The “AR superimposed image: AR marker ratio reflection (superimposed portion)” image 300c shown in FIG. 7C is taken from the smart terminal 40 positioned in front of the “original photographed image” image 300a shown in FIG. 7A. This is an example. In this example, the superimposition image is also scaled and superimposed at the same ratio according to the marker coordinates and inclination based on the relative position / posture of the camera and the second information by performing the replacement process shown in FIG. It is.

  For example, when the ratio of the “original photographed image” image 300a of the AR marker 200 to the captured image is 10: 5 in the vertical method and 10: 8 in the horizontal direction, the ratio in the vertical direction is 5/10 = 1/2. Since the horizontal direction has a ratio of 8/10 = 4/5, the superimposed image is also displayed on the smart terminal 40 with the vertical direction being halved and the horizontal direction being 4/5.

  Note that, when the entire AR marker 200 is not included in the imaging range as in the “AR marker non-detectable” image 300d illustrated in FIG. 7D, the information included in the AR marker 200 cannot be recognized. Since it is determined as “none”, the image cannot be superimposed. In that case, the display unit 46 may display on the smart terminal 40 a message stating “Please make the AR marker fit within the screen”.

  As a result, the user who visually confirms the message prompting the marker image to be within the predetermined imaging range can notice that the angle of view should be moved so that the marker image is within the predetermined imaging range.

  Next, the image of the AR marker 200 included in the print data will be described with reference to FIGS. 8A and 8B are image diagrams of AR markers included in the print data according to the embodiment of the present invention. For the host PC 10, it is possible to specify an AR marker addition type / AR markerless type according to a user specification.

In the AR marker giving type 310 shown in FIG. 8A, the AR marker 200 is explicitly arranged in the print data, and the AR marker arrangement condition is as follows.
AR marker size (vertical × horizontal) <Empty size (vertical × horizontal)
It is.

  Here, a method of arranging the AR marker 200 will be described. The first method is a method of searching for a free image area that matches the size (vertical × horizontal) of the AR marker 200 with respect to the print data and arranging the AR marker 200. An empty image area less than the area represented by the vertical and horizontal directions of the designated AR marker 200 is searched and arranged. If the conditions do not match, the AR marker 200 side can be reduced and arranged.

  The second method is a method of searching for an AR marker 200 that matches a pre-arranged image empty area with respect to print data and arranging the AR marker 200. The AR marker 200 that is less than the vertical x horizontal area of the designated image empty area is searched, and the AR marker 200 that matches the condition is arranged. If the conditions do not match, the specified AR marker 200 can be reduced and arranged.

  Further, when a plurality of AR markers 200 are specified, and when the AR marker 200 is not specified, the word of the print data is extracted by OCR processing, and the related AR marker 200 is switched depending on the word content. it can. Here, the AR marker 200 with the largest number of search words used in the OCR process is selected.

  The AR markerless type shown in FIG. 8B is a method of handling the image content and image arrangement of print data as the image bit arrangement of the AR marker 200.

<AR marker image>
Here, the AR marker image will be described with reference to FIG. FIG. 9 is a diagram in which AR marker images related to examples of corresponding words and classification thereof according to the embodiment are classified. As shown in FIG. 9, the AR marker image can be selected by the host PC 10 in accordance with the corresponding word example and its classification content.

<Smart terminal display image>
Next, a display image displayed on the smart terminal 40 will be described with reference to FIGS. FIGS. 10A to 10E are display image diagrams on the smart terminal 40 according to the embodiment.

  FIG. 10A shows a document printed by the image forming apparatus 20. On the other hand, in FIGS. 10B to 10E, the image shown in FIG. 10A is captured on the smart terminal 40 and displayed on the smart terminal 40 with information corresponding to the security level added. It is an example. It can be seen that data that can be acquired from the server 30 is switched according to the security levels A to D.

<Operation description of host PC>
Next, the operation of the host PC 10 will be described with reference to FIG. FIG. 11 is a flowchart illustrating processing of the host PC 10 according to the embodiment. In FIG. 11, in step S <b> 401, the AR information creation unit 14 selects the presence / absence of the AR marker and the arrangement position. Specifically, the AR information creating unit 14 inputs the format of the AR marker 200 input by the user, the AR marker placement position, and the disclosure information for each security level.

<AR information creation screen of host PC>
Here, the AR information creation screen of the host PC 10 will be described with reference to FIG. FIG. 12 is a diagram showing an AR information creation screen of the host PC 10 according to the embodiment. As shown in FIG. 12, in the AR information creation screen 400 displayed on the host PC 10, in addition to the “original photographed image” image 300a, a setting area for “marker selection”, “non-disclosure information for each security level”. The setting area is displayed.

  Here, in the setting area of “marker selection”, “with explicit” is a case where the AR marker 200 is put in the document so that a person can confirm, and “without explicit” indicates that the AR marker 200 is not included in the document so that the person cannot check. This is the case.

  In “marker arrangement”, it is possible to specify coordinates to be arranged, vertical x horizontal size, and whether or not automatic reduction is performed. In the setting area of “non-disclosure information for each security level”, it is possible to specify the coordinates of the data to be cut for each of the security levels A to D, the vertical × horizontal size, and the rotation angle.

  When the “add AR marker” button 410 is pressed, the AR marker 200 is displayed in the output image. In order to confirm the setting contents, an “OK” button 420 may be pressed. On the other hand, in order to cancel the setting contents, a “cancel” button 430 may be pressed.

  In step S402, the AR information creation unit 14 checks whether the AR marker 200 shown in FIG. If the AR marker 200 is “existent”, the AR information creation unit 14 sends an AR marker list request to the server 30 via the communication unit 42 to make an inquiry in step S403. The AR information creation unit 14 displays the AR marker list received from the server 30 via the communication unit 42 to the user, and prompts the user to select the AR marker type from the AR marker list. The AR information creation unit 14 creates AR information based on the result of selection by the user. A plurality of AR markers selected by the user can be selected. On the other hand, if the AR marker 200 is “no explicit”, in step S <b> 404, the AR information creation unit 14 creates AR information so that the document itself is handled as the AR marker 200.

<AR marker image selection screen of host PC>
Here, the AR marker image selection screen of the host PC 10 will be described with reference to FIG. FIG. 13 is a diagram showing an AR marker image selection screen 450 of the host PC 10 according to the embodiment.

  In the “marker image selection” setting area, “automatic” indicates that the AR marker is automatically created by the AR information creation unit 14. The AR marker 460 automatically created by the AR information creation unit 14 is shown in the upper right part of the screen. In order to confirm the setting contents, an “OK” button 470 may be pressed. On the other hand, in order to cancel the selection, the “cancel” button 480 may be pressed.

  In step S405, the communication unit 16 sends an AR information request to the server 30 via the network N to make an inquiry. The communication unit 16 receives a return of AR information from the server 30. In step S406, the AR information creation unit 14 checks whether the AR marker 200 is duplicated. The AR information creation unit 14 displays a warning message when the AR marker 200 is duplicated (S406: Duplicate). When the cancel button is pressed by a user operation, the process returns to step S401 to prompt the user to input again.

  As a result, a message prompting re-input of the AR marker 200 is displayed, so that the creator who visually confirms the message can recognize that re-input of the AR marker 200 is necessary.

<Duplicate warning message of host PC>
Here, the duplicate warning message of the host PC 10 will be described with reference to FIGS. FIG. 14A illustrates a duplicate warning message 500 when the AR marker 200 of the host PC 10 according to the embodiment is not explicitly shown. FIG. 14B is a diagram showing a duplication warning message 530 when the AR marker 200 of the host PC 10 according to the embodiment is clearly shown. FIG. 14C is a diagram showing an AR marker registration screen 570 of the host PC 10 according to the embodiment.

  When the AR marker 200 of the host PC 10 is not clearly indicated, the AR information creation unit 14 displays a duplication warning message 500 as shown in FIG. When the warning content is confirmed, an “OK” button 510 is pressed. On the other hand, when canceling, the “cancel” button 520 is pressed. Further, the AR information creation unit 14 displays a warning message shown in FIG. 14B when the AR marker 200 is clearly indicated and the AR marker 200 is duplicated. When the cancel button 540 is pressed, the process returns to step S403 to prompt the user to input the AR marker 200 again. On the other hand, in order to cancel, the “cancel” button 540 is pressed.

  As a result, a message prompting re-input of the AR marker 200 is displayed, so that the creator who visually confirms the message can recognize that re-input of the AR marker 200 is necessary.

  When the AR marker 200 is clearly indicated and the AR marker 200 is duplicated, the AR information creation unit 14 returns to step S403. In step S403, an image preview 590 is displayed by inputting a file path in the input area 580 on the AR marker registration screen 570 shown in FIG. 14C, and the user's host is displayed by pressing an “OK” button 600. Information on the AR marker 200 held by the PC 10 can be uploaded to the server 30. The AR marker 200 held by the server 30 is added, and the AR marker list can be updated. On the other hand, in order to cancel in FIG. 14C, a “cancel” button 610 is pressed.

  Thereby, by making the AR marker 200 and the list of the AR marker 200 updatable, the AR marker 200 and the list of the AR marker 200 can be used as the latest.

  In addition, the AR information creation unit 14 can also overwrite the AR information that is a duplication target with the current information. As a result, the AR marker 200 used this time is overwritten on the already used AR marker 200, so that the latest AR marker 200 can be used.

In step S407, the AR information creation unit 14 searches for the arrangement of the AR marker 200 in the print data. Here, the arrangement search process is performed only when the AR marker 200 is specified in step S401. Search conditions are as follows.
AR marker area selected by the user (vertical x horizontal) <User specified layout area (vertical x horizontal)

  The AR information creation unit 14 repeats the user-specified arrangement area (vertical x horizontal) larger than the AR marker area (vertical x horizontal) selected by the user for the AR marker selected by the user. .

  In step S408, the AR information creation unit 14 displays a warning message when all the AR markers 200 cannot be arranged. When the cancel button is pressed by a user operation, the AR information creation unit 14 returns to step S403 and prompts the AR marker 200 to be selected again.

  As a result, a message prompting the rearrangement of the AR marker 200 is displayed, and the creator who visually confirms the message may notice that the arrangement position on the original image to which the AR marker 200 should be added should be changed. it can.

<Reselection message>
Here, with reference to FIG. 15, a message for prompting reselection of the AR marker 200 will be described. FIG. 15 is a diagram showing a message 650 that prompts reselection of the AR marker 200 of the host PC 10 according to the embodiment. FIG. 5 shows an example of a message prompting the user to re-select the AR marker 200. After confirmation, the “OK” button 660 is pressed, and when canceling, the “Cancel” button 670 is pressed.

  When the AR information creation unit 14 cannot arrange the AR marker 200, the AR information creating unit 14 searches for a free image area in the document, searches for an area in which the size of the AR marker 200 fits, and determines the arrangement position of the AR marker 200. .

  Thereby, even when a part of the original image occupies the arrangement position on the original image to which the AR marker 200 should be added, the AR marker 200 can be arranged in the empty area of the searched original image.

<AR marker relocation image>
Here, the rearrangement image of the AR marker will be described with reference to FIGS. FIGS. 16A and 16B are diagrams showing an AR marker rearrangement image of the host PC 10 according to the embodiment.

  As shown in FIG. 16 (a), the AR information creation unit 14 uses the empty space (characters and images not arranged) in the document 680 for the marker size (vertical x horizontal area) in the automatic rearrangement process. If the condition of marker size <free space in the document is satisfied, the search is rearranged.

  FIG. 16A shows the AR marker planned location 690, but the AR marker cannot be placed because there is a character at the planned location. On the other hand, FIG. 16B shows a case where a space having an AR marker size is searched in the document 680 and arranged in a space 700 in which the AR marker is accommodated. If this condition is not satisfied, the AR information creation unit 14 returns to step S403 and prompts the AR marker 200 to be selected again.

  In step S409, the communication unit 16 adds the identifier to the AR information, the association information of the AR marker 200 and the print data, the disclosure information for each security level, and the network N by adding an identifier to the original print data. To the server 30.

  In step S410, the print data creation unit 12 creates print data (original image) in which the image in the range indicated by the disclosure information for each security level is erased from the original print data, and the print data is transmitted from the communication unit 16 to the network. N is transmitted to the image forming apparatus 20 via N.

  Thereby, an original image to which the AR marker 200 is attached can be created, and the original image can be transmitted to the image forming apparatus 20.

<Description of Operation of Image Forming Apparatus>
Next, the operation of the image forming apparatus 20 will be described with reference to FIG. FIG. 17 is a flowchart illustrating processing of the image forming apparatus 20 according to the embodiment.

  In FIG. 17, in step S <b> 501, the communication unit 22 receives print data from the host PC 10 via the network N. Next, in step S502, the print data output unit 24 prints this print data on a recording medium and outputs it. As a result, the image forming apparatus 20 prints on the recording medium the contents in which the image in the range indicated by the disclosure information for each security level is erased. Thereby, the received print data (original image) can be printed on the recording medium.

<Operation explanation of smart terminal>
Next, the operation of the smart terminal 40 will be described with reference to FIG. FIG. 18 is a flowchart showing processing of the smart terminal 40 according to the embodiment.

  In FIG. 18, in step S <b> 601, the imaging unit 50 captures a document printed on a recording medium with a camera, and then the information acquisition unit 56 detects an identifier from the AR marker 200. In step S <b> 602, the communication unit 42 transmits an inquiry request for the identifier and the user ID to the server 30 using the identifier represented by the detected AR marker 200 and the user ID read from the smart terminal 40. That is, the communication unit 42 transmits the user ID and the identifier to the server 30 via the network N. The smart terminal 40 can also specify the user ID when making an inquiry to the server 30 by using the operation unit 44. Thereby, AR information corresponding to the specified user ID can be acquired.

  Thereafter, in step S603, when the communication unit 42 receives the AR information from the server 30 via the network N, the process proceeds to step S604, and the display unit 46 combines the captured information with the AR information that is the superimposed data. And displayed on the screen of the smart terminal 40. In step S603, when the AR information cannot be received from the server 30, the display unit 46 does not particularly process the display image.

  Further, when the AR information is not received, the display unit 46 can also display an error message to that effect.

  Thereby, when AR information cannot be acquired from the server 30, an error message is displayed, so that the user can notice that the AR information cannot be acquired from the server 30.

<Smart terminal display message>
Here, a message displayed on the smart terminal 40 will be described with reference to FIG. FIG. 19 is a diagram illustrating a message 710 when AR information cannot be received from the server 30 in the smart terminal 40 according to the embodiment. In FIG. 19, the “OK” button 720 is pressed after confirming the message, and the “Cancel” button 730 is pressed when canceling.

<Explanation of server operation>
Next, the operation of the server 30 will be described with reference to FIG. FIG. 20 is a flowchart illustrating processing of the server 30 according to the embodiment.

  In FIG. 20, in step S <b> 701, the communication unit 34 receives an inquiry request for the identifier of the AR marker 200 and the user ID from the smart terminal 40 via the network N. In step S702, the display data creation unit 32 searches for AR information corresponding to the identifier stored in the identifier table of the AR information storage unit 38, using the received identifier as a key. Table 2 is an identifier table in which AR information corresponding to the identifier is described.

  In step S703, the display data creation unit 32 checks whether or not the identifier of the AR marker 200 requested by the smart terminal 40 matches the identifier of the AR information obtained as a result of retrieval from the AR information storage unit 38. If they do not match, the process ends.

  On the other hand, if the two match, the process advances to step S704, and the display data creation unit 32 searches the security table (Table 3) for the security level using the received user ID as a search key. Table 3 is a security table showing the relationship between the security level and the user ID.

  In step S705, the display data creation unit 32 checks whether there is a security level, and if not, ends the processing. On the other hand, when there is no user ID itself in the security table, the user can be notified of an error message indicating that the user ID itself is not in the security table, and prompted to re-enter the user ID.

  Thereby, the user who confirmed the error message can notice that the user ID is not registered. Also, based on the received identifier and user ID, AR information matched to the user's authority can be searched and transmitted to the transmission source of the user ID, and security can be improved.

<Message when there is no user ID>
FIG. 21 is a diagram illustrating a message 750 when there is no user ID transmitted from the server 30 according to the embodiment. In FIG. 21, the “OK” button 770 is pressed after confirming the message, and the “Cancel” button 760 is pressed when canceling.

  On the other hand, if the user ID is in the security table, the display data creation unit 32 transmits the AR information to the smart terminal 40 via the communication unit 34 in step S706. In addition, the security table used in step S704 indicates that the display data creation unit 32 changes the user ID when a change request for the number of security levels and the user ID corresponding to each security level is received from the host PC 10. Also possible. Thereby, the user ID on the security table can be changed.

  Further, when a change request for the AR information displayed at the security level is received from the host PC 10, the display data creation unit 32 can also change the AR information. Thereby, AR information can be changed and the convenience can be improved.

  The display data creation unit 32 retrieves the security level (for example, A to D) for the user ID from the security table, and then refers to the AR image / security level comparison table (Table 4) to refer to the AR information and the security level. The image corresponding to is selected and read. The display data creation unit 32 transmits the selected image data to the smart terminal 40 via the communication unit 34 as superimposed data. Table 4 is a comparison table showing the relationship between the smart terminal transmission image corresponding to the AR information and the security level (display condition information).

  In response to the inquiry of the AR marker information from the host PC 10, the server 30 checks whether the same AR marker 200 is used in the AR information stored in the AR information storage unit 38 and checks the result. It is possible to respond to the PC 10.

  In addition, when an update request for the AR marker 200 and the AR marker list stored in the original print data storage unit 36 of the server 30 is received from the host PC 10, the information in the server 30 can be updated according to the request content. And

  As described above, the imaging unit 50 captures an image. When a recording medium on which an AR marker 200 (marker image) indicating data display conditions is photographed, an identifier for identifying the recording medium from the AR marker 200 (marker image) on which the information acquisition unit 56 has been photographed. Extract. The communication unit 42 transmits an identifier and a user ID for specifying a user to the server 30. The communication unit 42 receives superimposition data that matches the user's authority from the server 30. The display unit 46 displays the superimposed data superimposed on the recording medium being photographed.

Thereby, it is possible to display the superimposed data according to the authority of the user so as to be superimposed on the recording medium being photographed, thereby improving the security.

  Next, an example in which the security level is managed by a user group will be described. Here, just determining the security level based on the user ID requires management information for each individual, which is not convenient for management. Therefore, a mechanism for enabling management by user groups is provided.

  Until now, the user ID was directly sent to the server 30 and converted to a security level associated with the user ID on the server 30 side. On the other hand, in this example, using the server 70 for converting the user ID into the user group ID, the server 30 notifies the user ID of the user ID, and the server 70 converts the user ID into the user group ID. After that, AR information is acquired using the user group ID instead of the user ID.

  Next, the operation of the server 30 will be described with reference to FIG. FIG. 22 is a flowchart illustrating processing of the server 30 according to the embodiment. In FIG. 22, in step S801, first, an inquiry about the AR marker 200 and the user group ID from the smart terminal 40 is received. Note that the user ID is used for the inquiry of the user group ID. In step S802, the server 70 is inquired about the AR marker 200 and the user group ID in order to obtain the user group ID. That is, in step S <b> 802, the server 30 requests the user group ID from the server 70.

  In step S <b> 803, the server 30 determines whether a user group ID has been received from the server 70. In step S803, when the user group ID is received from the server 70 (determination, yes), the same operation as in step S702 of FIG. 20 is executed. On the other hand, if the user group ID is not received from the server 70 in step S803 (judgment, no), an error with no user group ID is transmitted to the smart terminal 40 in step S804, and this operation ends.

  In step S804, a display image as shown in FIG. 21, for example, is displayed on the display unit 46 of the smart terminal 40, for example.

  FIG. 23 is a flowchart illustrating processing of the server 70 according to the embodiment. 23, first, in step S901, an inquiry about the AR marker 200 and the user group ID from the server 30 is received. Subsequently, in step S902, based on the information received from the server 30, a group corresponding to the AR marker 200 held by the server 70 is searched, and a user group ID is searched from the searched group data 73.

  In step S903, it is determined whether the user ID matches. If they do not match (determination, none), an error is transmitted to the server 30 in step S905, and the process is terminated. On the other hand, if they match in step S903 (determination, yes), the user group ID that matches the user ID received in step S904 is transmitted to the server 30.

  The group search described above selects a group that matches the type of the AR marker 200 with reference to the group table assigned to the type of the AR marker 200. The group table is registered in the group data 73 of the server 70 in advance. The search for the user group ID refers to the user group ID table corresponding to the selected group and selects a user group ID that matches the user ID. A group table structure example is shown in Table 5 below, and a user group ID table example is shown in Table 6 below.

  Next, in the image forming system using the AR technology, the security can be improved by allowing only a specific authorized person to display information displayed when the printed document is viewed through the screen of the smart terminal 30. In addition, an example will be described in which the user is identified by adding fingerprint authentication of the smart terminal 40 to improve security.

  FIG. 24 is a flowchart illustrating mode processing of the server 70 according to the embodiment. In FIG. 24, in step S1001, the server 70 sets one of a fingerprint authentication mode, a non-fingerprint authentication mode, and an automatic fingerprint authentication mode.

  When the fingerprint authentication mode in step S1002 is set, first, in step S1005, it is determined whether fingerprint authentication information is given to the user ID. If it is determined that the fingerprint authentication information is present, the user is identified in step S1006 and AR information corresponding to the user ID authority is returned. On the other hand, if it is determined in step S1005 that there is no fingerprint authentication information in the user ID, since the person cannot be determined in step S1007, AR information with the lowest security level is returned.

  When the non-fingerprint authentication mode in step S1003 is set, it is determined in step S1008 that the user is the user, and AR information corresponding to the user ID authority is returned.

  When the automatic fingerprint authentication mode in step S1004 is set, it is further determined whether or not fingerprint authentication is required for each user ID from the fingerprint authentication necessity information table shown in Table 7 stored in the server 70 in step S1009. If the fingerprint authentication necessity information is “necessary” in step S1009, the process proceeds to step S1005 to identify the user himself / herself and return AR information corresponding to the user ID authority. On the other hand, if the fingerprint authentication necessity information is “No” in step S1009, the process proceeds to step S1008 to determine that the user is the user himself / herself, and AR information corresponding to the user ID authority is returned.

  On the server 70 side, it is possible to set three modes of fingerprint authentication / not / automatic. After the processing for each set mode, the responding security information is managed by the method shown in FIG.

Thus, on the smart terminal 40 side, the fingerprint authentication information is given to the user ID so far. The fingerprint authentication on the smart terminal 40 side uses either one authenticated at the time of login or one authenticated by displaying an authentication screen before inquiring to the server 70. The conditions for granting the presence or absence of fingerprint authentication to the user ID are as follows.
With fingerprint authentication: User ID + fingerprint authentication information Without fingerprint authentication: User ID only

  When AR information with the lowest security level is received from the server 70 side as a result of the determination that there is no fingerprint authentication information, a screen for prompting fingerprint authentication is displayed, for example, the display unit of the smart terminal 40 prompting fingerprint authentication shown in FIG. 46. FIG. 25 shows a screen 800 for prompting fingerprint authentication. After confirming the message, the “OK” button 820 is pressed, and when canceling, the “Cancel” button 810 is pressed.

<Configuration, operation, and effect of exemplary embodiment>
<First aspect>
The information processing system 1 according to this aspect includes a host PC 10 that creates print data to which an AR marker 200 (marker image) indicating a display condition of an image after printing is provided, and an image forming apparatus 20 that prints the print data on a recording medium. And a smart terminal 40 having an imaging unit 50 and an input unit (operation unit 44, fingerprint authentication unit 49) for inputting user authentication information including fingerprint authentication information, and a print image printed on a recording medium by the smart terminal 40. When shooting, the display authority is determined based on the display condition associated with the AR marker 200 (marker image) and the user authentication information of the smart terminal 40, and displayed on the display unit 46 of the smart terminal 40 according to this determination. And a server 30 for changing an image to be displayed.

  According to the first aspect, after verifying the identity with the authentication information including the user's fingerprint authentication, it is possible to display the superimposed data according to the authority of the user in an overlapping manner on the recording medium being photographed. Can be improved.

<Second aspect>
The information processing system 1 according to this aspect includes a host PC 10 that creates print data to which an AR marker 200 (marker image) indicating a display condition of an image after printing is provided, and an image forming apparatus 20 that prints the print data on a recording medium. A smart terminal 40 having an imaging unit 50 and an input unit (operation unit 44, fingerprint authentication unit 49) for inputting user authentication information including fingerprint authentication information, a first server (server 30), and a second The first server (server 30) receives user group authentication information from the smart terminal 40, and the second server (server 70) is the first server (server 30). The user group of the smart terminal 40 is authenticated from the user group authentication information sent from the smart terminal 40, and the first server (server 30) is printed on the recording medium. When a print image is taken by the smart terminal 40, the display authority is determined based on the display condition associated with the AR marker 200 (marker image) and the user group authentication information of the smart terminal 40, and the smart terminal is determined according to this determination. The image displayed on the display part 46 which 40 has is changed, It is characterized by the above-mentioned.

  According to the second aspect, after verifying the identity with the authentication information including the user's fingerprint authentication, it is possible to display the superimposed data according to the user's authority in an overlapping manner on the recording medium being photographed. Can be improved.

<Third aspect>
In the information processing system 1 according to this aspect, when creating an original image to which the AR marker 200 (marker image) is assigned, if the original image to which the AR marker 200 is assigned is already used, the creator is notified of the AR. The host PC 10 includes an AR information creation unit 14 that displays a message prompting the user to input the marker 200 again.

  According to the third aspect, when the AR information creation unit 14 provided in the host PC 10 creates an original image to which the AR marker 200 is attached, the original image to which the AR marker 200 is attached is already used. In the case, a message prompting the creator to re-input the AR marker 200 is displayed. As a result, a message prompting re-input of the AR marker 200 is displayed, so that the creator who visually confirms the message can recognize that re-input of the AR marker 200 is necessary.

<4th aspect>
When the AR marker 200 is already used when creating the original image to which the AR marker 200 (marker image) is added, the information processing system 1 of this aspect prompts the creator to input the AR marker 200 again. The host PC 10 includes an AR information creation unit 14 that displays a prompting message.

  According to the fourth aspect, when the AR information creation unit 14 provided in the host PC 10 creates the original image to which the AR marker 200 is attached, if the AR marker 200 is already used, the creator is notified. A message prompting re-input of the marker image is displayed. As a result, a message prompting re-input of the AR marker 200 is displayed, so that the creator who visually confirms the message can recognize that re-input of the AR marker 200 is necessary.

<5th aspect>
The information processing system 1 of this aspect hosts the AR marker 200 and the AR information creation unit 14 that can update the list of the AR markers 200 when creating an original image to which the AR marker 200 (marker image) is attached. It is provided in the PC 10.

  According to the fifth aspect, when the AR information creation unit 14 provided in the host PC 10 creates an original image to which the AR marker 200 is attached, the AR marker 200 and the list of the AR marker 200 can be updated. . Thereby, by making the AR marker 200 and the list of the AR marker 200 updatable, the AR marker 200 and the list of the AR marker 200 can be used as the latest.

<Sixth aspect>
When the AR marker 200 is already used when creating the original image to which the AR marker 200 (marker image) is added, the information processing system 1 according to this aspect is used as the AR marker 200 already used. The host PC 10 includes an AR information creation unit 14 that overwrites the AR marker 200 to be used.

  According to the sixth aspect, when the AR information creation unit 14 included in the host PC 10 creates an original image to which the AR marker 200 is attached, if the AR marker 200 is already used, it is already used. The AR marker 200 to be used this time is overwritten on the AR marker 200 currently being used. As a result, the AR marker 200 used this time is overwritten on the already used AR marker 200, so that the latest AR marker 200 can be used.

<Seventh aspect>
In the information processing system 1 of this aspect, when creating an original image to which the AR marker 200 (marker image) is added, a part of the original image occupies the arrangement position on the original image to which the AR marker 200 is to be added. The host PC 10 includes an AR information creation unit 14 for displaying a message prompting the creator to rearrange the AR marker 200.

  According to the seventh aspect, when the AR information creation unit 14 provided in the host PC 10 creates the original image to which the AR marker 200 is attached, the original information is placed at the arrangement position on the original image to which the AR marker 200 is to be attached. When a part of the image is occupied, a message prompting the creator to rearrange the AR marker 200 is displayed. As a result, a message prompting the rearrangement of the AR marker 200 is displayed, and the creator who visually confirms the message may notice that the arrangement position on the original image to which the AR marker 200 should be added should be changed. it can.

<Eighth aspect>
In the information processing system 1 of this aspect, when creating an original image to which the AR marker 200 (marker image) is added, a part of the original image occupies the arrangement position on the original image to which the AR marker 200 is to be added. The host PC 10 includes an AR information creation unit 14 that searches for an empty area of the original image and places the AR marker 200 in the empty area.

  According to the eighth aspect, when the AR information creation unit 14 provided in the host PC 10 creates the original image to which the AR marker 200 is attached, the original information is placed at the arrangement position on the original image to which the AR marker 200 is to be attached. When a part of the image is occupied, the empty area of the original image is searched and the AR marker 200 is arranged in the empty area. Thereby, even when a part of the original image occupies the arrangement position on the original image to which the AR marker 200 should be added, the AR marker 200 can be arranged in the empty area of the searched original image.

<Ninth aspect>
The information processing system 1 of this aspect is characterized in that the smart terminal 40 includes a display unit 46 that displays an error message when AR information (superimposed data) cannot be acquired from the server 30.

  According to the ninth aspect, the display unit 46 provided in the smart terminal 40 displays an error message when the AR information cannot be acquired from the server 30. Thereby, when AR information cannot be acquired from the server 30, an error message is displayed, so that the user can notice that the AR information cannot be acquired from the server 30.

<10th aspect>
The information processing system 1 of this aspect is characterized in that the smart terminal 40 includes an operation unit 44 that specifies a user ID when acquiring AR information (superimposed data) from the server 30.

  According to the tenth aspect, when the operation unit 44 provided in the smart terminal 40 acquires AR information from the server 30, the AR information corresponding to the specified user ID is acquired by specifying the user ID. can do. Thereby, AR information corresponding to the specified user ID can be acquired.

<Eleventh aspect>
The information processing system 1 according to this aspect is characterized in that when the superimposition data is requested, if the user ID is not registered, the server 30 includes a display data creation unit 32 that responds with an error message.

  According to the eleventh aspect, when the superimposition data is requested, the display data creation unit 32 provided in the server 30 responds with an error message when the user ID is not registered. Thereby, the user who confirmed the error message can notice that the user ID is not registered.

<Twelfth aspect>
The information processing system 1 of this aspect is characterized in that the server 30 includes a display data creation unit 32 that can change the security table corresponding to the user ID when superimposition data is requested.

  According to the twelfth aspect, when the display data creation unit 32 included in the server 30 is requested to superimpose data, the security table corresponding to the user ID can be changed. Thereby, the user ID on the security table can be changed.

<13th aspect>
The information processing system 1 according to this aspect is characterized in that the server 30 includes a display data creation unit 32 that can change AR information (superimposition data).

  According to the thirteenth aspect, the display data creation unit 32 provided in the server 30 enables the AR information to be changed. Thereby, AR information can be changed and the convenience can be improved.

<14th aspect>
The information processing system 1 according to the present aspect smartly controls the control unit 52 that switches the number of repetitions related to the process from extracting the identifier to displaying the AR information (superimposed data) according to the performance of the CPU that executes the process. The terminal 40 is provided.

  According to the fourteenth aspect, the control unit 52 included in the smart terminal 40 extracts the identifier and displays the number of repetitions related to the process until the AR information is displayed according to the performance of the CPU that executes the process. Switch. Thereby, it is possible to switch the number of repetitions related to the processing from extracting the identifier to displaying the AR information according to the performance of the CPU.

<15th aspect>
The information processing system 1 according to this aspect is characterized in that the smart terminal 40 includes a control unit 52 that switches the number of repetitions according to a user's designation.

  According to this 15th aspect, the control part 52 with which the smart terminal 40 was equipped switches the repetition frequency according to a user's designation | designated. Thereby, it can switch to the repetition frequency which a user desires.

<16th aspect>
When the information processing system 1 according to this aspect captures a recording medium and the marker image displayed on the recording medium is not within the predetermined imaging range, a message prompting the marker image to be within the predetermined imaging range is displayed. The display unit 46 for displaying is provided in the smart terminal 40.

  According to the sixteenth aspect, when the display unit 46 provided in the smart terminal 40 captures the recording medium, if the marker image displayed on the recording medium is not within the predetermined imaging range, the marker image is A message prompting to fit within the imaging range is displayed. As a result, the user who visually confirms the message prompting the marker image to be within the predetermined imaging range can notice that the angle of view should be moved so that the marker image is within the predetermined imaging range.

<17th aspect>
The information processing method of this aspect includes a step of creating print data to which an AR marker 200 (marker image) indicating a display condition of an image after printing is added by the host PC 10, and the print data is recorded on a recording medium by the image forming apparatus 20. The step of printing, the step of inputting user authentication information including fingerprint authentication information with the smart terminal 40, and the AR image 200 (marker image) when the print image printed on the recording medium is photographed with the smart terminal 40 Determining the display authority by the server 30 based on the attached display condition and the user authentication information of the smart terminal 40, and changing the image displayed on the display unit 46 of the smart terminal 40 according to this determination. It is characterized by.

  Since the operation and effect of the seventeenth aspect are the same as those of the first aspect, description thereof will be omitted.

<18th aspect>
The program according to the present aspect includes a step of creating print data to which an AR marker 200 (marker image) indicating a display condition of an image after printing is added by the host PC 10 and printing the print data on a recording medium by the image forming apparatus 20. A step of inputting user authentication information including fingerprint authentication information with the smart terminal 40; and when the smart terminal 40 takes a print image printed on a recording medium, the AR marker 200 (marker image) is associated. The server 30 determines display authority based on the display conditions and the user authentication information of the smart terminal 40, and causes the computer to execute a step of changing an image displayed on the display unit 46 of the smart terminal 40 according to the determination. .

  According to this aspect, each step can be executed by the processor.

  A program executed by the information processing system 1 according to the present embodiment is provided by being incorporated in advance in a ROM or the like included in each module. The above program is recorded in a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, a DVD (Digital Versatile Disk), etc. in a file that can be installed or executed. May be provided.

  Furthermore, the program executed in the present embodiment may be configured to be provided by being stored on a computer connected to a network such as the Internet and downloaded via the network. In addition, the program executed in the present embodiment may be configured to be provided or distributed via a network such as the Internet.

  The program executed in the present embodiment has a module configuration including the above-described units. As actual hardware, the CPU reads out the program from the ROM and executes the program, so that the respective units are loaded onto the main storage device, and the respective units are generated on the main storage device.

  The embodiment described above is presented as an example for realizing the present invention, and is not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the spirit of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are also included in the invention described in the claims and the equivalents thereof.

1 Information processing system 10 Host PC
DESCRIPTION OF SYMBOLS 20 Image forming apparatus 30,70 Server 40 Smart terminal 12 Print data creation part 14 AR information creation part 16 Communication part 24 Print data output part 32 Display data creation part 36 Original print data storage part 38 AR information storage part 44 Operation part 46 Display Unit 48 storage unit 49 fingerprint authentication unit 50 imaging unit 52 control unit 54 image acquisition unit 56 information acquisition unit 58 calculation unit 60 replacement unit 200 AR marker 201 external area 202 internal area

JP 2013-026922 A

Claims (10)

An information processing terminal for creating print data to which a marker image indicating a display condition of an image after printing is attached;
An image forming apparatus for printing the print data on a recording medium;
A portable terminal having an imaging unit and an input unit for inputting user authentication information including fingerprint authentication information;
When a print image printed on the recording medium is taken by the mobile terminal, display authority is determined based on the display condition associated with the marker image and the user authentication information of the mobile terminal, Therefore, a server that changes the image displayed on the display unit of the portable terminal,
An information processing system comprising: An information processing terminal for creating print data to which a marker image indicating a display condition of an image after printing is attached;
An image forming apparatus for printing the print data on a recording medium;
A portable terminal having an imaging unit and an input unit for inputting user authentication information including fingerprint authentication information;
A first server;
A second server;
With
The first server receives the user group authentication information from the mobile terminal,
The second server authenticates the user group of the mobile terminal from the user group authentication information from the mobile terminal sent from the first server,
The first server displays a print image printed on the recording medium based on the display condition associated with the marker image and the user group authentication information of the mobile terminal when the mobile terminal photographs the print image. An information processing system characterized by determining an authority and changing an image to be displayed on a display unit of the portable terminal according to the determination.   The information processing system according to claim 2, wherein a group table in which the marker image and the user group are associated with each other is registered in the second server in advance.   The information processing system according to claim 2, wherein the second server performs authentication permission for a plurality of user groups.   When the second server cannot acquire the user group authentication information from the portable terminal, the second server transmits an error occurrence to the first server, and the first server displays an error message of the portable terminal. The information processing system according to claim 2.   3. The information according to claim 2, wherein the first server changes the data displayed on the portable terminal to information having the lowest security level when there is no fingerprint authentication information in the user group authentication information. 4. Processing system. The first server is set with a fingerprint authentication mode, a non-fingerprint authentication mode, and an automatic fingerprint authentication mode,
In the case of the non-fingerprint authentication mode, based on user authentication information not including the fingerprint authentication information, the data displayed on the mobile terminal is changed,
In the case of the fingerprint authentication mode, based on user authentication information including the fingerprint authentication information, the data displayed on the mobile terminal is changed,
In the case of the automatic fingerprint authentication mode, it is determined whether or not authentication associated with the user authentication information is necessary based on fingerprint authentication necessity information of the second server, and in the case of authentication failure, user authentication information not including fingerprint authentication information 3. The information processing system according to claim 2, wherein when only authentication is required, data displayed on the portable terminal is changed based on user authentication information including the fingerprint authentication information.   3. The information processing system according to claim 2, wherein when there is no fingerprint authentication information in the user group authentication information, the first server performs a display prompting to perform fingerprint authentication on the mobile terminal side. A step of creating print data to which a marker image indicating a display condition of an image after printing is attached by an information processing terminal;
Printing the print data on a recording medium with an image forming apparatus;
Inputting user authentication information including fingerprint authentication information on a mobile terminal;
When a print image printed on the recording medium is taken by the mobile terminal, the display authority is determined by the server based on the display condition associated with the marker image and the user authentication information of the mobile terminal, Changing the image displayed on the display unit of the portable terminal according to the determination;
An information processing method comprising: Creating print data to which a marker image indicating a display condition of an image after printing is attached at an information processing terminal;
Printing the print data on a recording medium with an image forming apparatus;
Inputting user authentication information including fingerprint authentication information on a mobile terminal;
When a print image printed on the recording medium is taken by the mobile terminal, the display authority is determined by the server based on the display condition associated with the marker image and the user authentication information of the mobile terminal, Changing the image displayed on the display unit of the portable terminal according to the determination;
A program that causes a computer to execute.

Images