JP6724464B2 - Printing assistance device, printing device, printing system, printing assistance method and program - Google Patents

Description

The present invention relates to a print assisting device, a printing device, a printing system, a print assisting method, and a program.

Patent Document 1 discloses a manual scanning type printing apparatus that prints contents such as characters, symbols, logos, characters, and marks on a print medium by gripping a housing and manually scanning.

JP, 10-35028, A

In the manual scanning type printing apparatus disclosed in Patent Document 1, the user himself/herself predicts the total length of the content to be printed based on the number of characters of the content, the size of the character, etc., and the print start position, the scanning distance, etc. Need to be adjusted. Therefore, although the user himself/herself manually scans the printing apparatus, there is a possibility that the printing intended by the user cannot be performed, such as protruding from the printing medium.

The present invention has been made to solve the above problems, and a print assisting device, a printing device, which can assist printing by a user when scanning and printing a manual scanning type printing device, An object is to provide a printing system, a printing assistance method, and a program.

In order to achieve the object of the present invention, one aspect of the print assisting device according to the present invention is
Marker detection means for detecting a reference marker arranged in a manual scanning type printing device from a captured image of the printing device,
Generating an auxiliary image for assisting the scanning of the printing device, based on a first arrangement position that is the position of the printing device in the captured image specified by the reference marker detected by the marker detection means, Image generating and displaying means for displaying the auxiliary image and the image showing the position of the printer head of the printing device in an overlapping manner on the captured image;
It is characterized by including.

Further, in order to achieve the object of the present invention, one aspect of the print assisting device according to the present invention is
Marker detection means for detecting a reference marker arranged in a manual scanning type printing device from a captured image of the printing device,
On the basis of the outer shape of the printing device in the captured image specified by the reference marker detected by the marker detection unit and the pre-printed image that has been printed in advance on the print medium on which the content to be printed is printed, As an auxiliary image for assisting scanning, an image generation and display unit that generates the already printed image of a portion hidden by the printing device, and displays the auxiliary image by superimposing it on the captured image,
It is characterized by including.

Further, in order to achieve the object of the present invention, one aspect of the printing apparatus according to the present invention is
A communication unit that receives the content to be printed from the print auxiliary apparatus;
Printing means for printing based on the content;
It is characterized by including.

Further, in order to achieve the object of the present invention, one aspect of the printing apparatus according to the present invention is
A communication unit that receives the content to be printed from the print auxiliary apparatus;
A printing unit that performs printing based on the content,
The reference marker is arranged,
It is characterized by

Further, in order to achieve the object of the present invention, one aspect of the printing apparatus according to the present invention is
A manual scanning type printing device,
From the print assisting device including an image capturing unit that captures a physical space including a reference marker arranged in the printing device and a first communication unit that transmits information about the captured image captured by the image capturing unit, Second communication means for receiving information,
Marker detection means for detecting the reference marker from the received captured image,
An image showing the arrangement position and the scanning direction of the printing device and the image showing the position of the printer head of the printing device in the captured image specified by the reference marker detected by the marker detecting means are the images of the printing device. Image generation and display means for displaying the captured image in a superposed manner,
It is characterized by including.

Further, in order to achieve the object of the present invention, one aspect of the printing system according to the present invention is
The printing auxiliary device,
The printing device in which the reference marker is arranged,
It is characterized by including.

Further, in order to achieve the object of the present invention, one aspect of the printing assistance method according to the present invention is:
A marker detection step of detecting a reference marker arranged in the manual scanning type printing device from a captured image of the printing device,
Generating an auxiliary image for assisting scanning of the printing device, based on a first arrangement position that is the position of the printing device in the captured image specified from the reference marker detected in the marker detection step, An image generation and display step of displaying the auxiliary image and an image showing the position of the printer head of the printing device in an overlapping manner on the captured image;
It is characterized by including.

Further, in order to achieve the object of the present invention, one aspect of the printing assistance method according to the present invention is:
A marker detection step of detecting a reference marker arranged in the manual scanning type printing device from a captured image of the printing device,
Based on the outer shape of the printing device in the captured image identified from the reference marker detected in the marker detection step and the pre-printed image that has been printed in advance on the print medium on which the content to be printed is printed, As an auxiliary image for assisting scanning, an image generating and displaying step of generating the already printed image of a portion hidden by the printing device , and displaying the auxiliary image by superimposing it on the captured image,
It is characterized by including.

In order to achieve the object of the present invention, one aspect of the program according to the present invention is
Computer,
Marker detecting means for detecting a reference marker arranged in a manual scanning type printing device from a captured image of the printing device,
Generating an auxiliary image for assisting the scanning of the printing device, based on a first arrangement position that is the position of the printing device in the captured image specified by the reference marker detected by the marker detection means, Image generation and display means for displaying the auxiliary image and the image showing the position of the printer head of the printing device in an overlapping manner on the captured image;
Function as
It is characterized by

In order to achieve the object of the present invention, one aspect of the program according to the present invention is
Computer,
Marker detecting means for detecting a reference marker arranged in a manual scanning type printing device from a captured image of the printing device,
On the basis of the outer shape of the printing device in the captured image specified by the reference marker detected by the marker detection unit and the pre-printed image that has been printed in advance on the print medium on which the content to be printed is printed, As an auxiliary image for assisting scanning, an image generation and display unit that generates the already printed image of a portion hidden by the printing device and displays the auxiliary image by superimposing it on the captured image.
Function as
It is characterized by

According to the present invention, it is possible to assist printing by a user when scanning and printing with a manual scanning type printing apparatus.

FIG. 1 is a diagram showing a printing system according to an embodiment of the present invention. It is a bottom view showing a handy printer according to an embodiment of the present invention. It is a block diagram showing a configuration of a handy printer according to an embodiment of the present invention. It is a block diagram which shows the structure of the terminal device which concerns on embodiment of this invention. 6 is a flowchart showing a print assist process executed by the terminal device according to the embodiment of the present invention. It is a flow chart which shows the display processing which the terminal unit concerning the embodiment of the invention performs. 7 is a flowchart showing a process of displaying an auxiliary image executed by the terminal device according to the embodiment of the present invention. 6 is a flowchart showing a printing process executed by the handy printer according to the embodiment of the present invention. It is a figure which shows the display of the display of the terminal device which concerns on embodiment of this invention. It is a figure which shows the display of the display of the terminal device which concerns on embodiment of this invention. It is a figure which shows the display of the display of the terminal device which concerns on embodiment of this invention. It is a figure which shows the display of the display of the terminal device which concerns on embodiment of this invention. It is a figure which shows the display of the display of the terminal device which concerns on embodiment of this invention. It is a figure which shows the display of the display of the terminal device which concerns on embodiment of this invention.

Hereinafter, a printing system according to an embodiment for carrying out the present invention will be described with reference to the drawings.

The printing system 1 according to the present embodiment prints contents such as characters, symbols, logos, characters and marks on a print medium. As shown in FIG. 1, the printing system 1 includes a handy printer (manual scanning type printing device) 100 that is scanned by a user to print content, and a terminal device (printing auxiliary device) 200 having a display 240. .. Hereinafter, unless otherwise specified, the terms “upper”, “lower”, “right”, and “left” will be used in correspondence with the vertical and horizontal directions in FIG. 1. The terminal device 200 includes, for example, a smartphone and a tablet PC (Personal Computer). The handy printer 100 and the terminal device 200 are configured to be able to communicate with each other through a wireless line. The terminal device 200 functions as an augmented reality (AR) display device, and an auxiliary image (auxiliary image) that assists printing by indicating or suggesting to the user how to scan the handy printer 100. Image) is displayed.

The housing 101 of the handy printer 100 has a size that can be easily gripped by a user. When the user holds the handy printer 100 and scans it in the sub-scanning direction shown in FIG. 1, the handy printer 100 prints the content on the print medium R.

On the upper surface of the handy printer 100, a reference marker 102 for the terminal device 200 to specify the position and the sub-scanning direction of the handy printer 100, and a start button 150 that receives a print start instruction are provided. The reference marker 102 is composed of a square black frame and dots indicating the direction of the mark arranged near the vertex D in the black frame. The line segment connecting the vertices A and B of this square and the line segment connecting the vertices C and D are provided in parallel to the sub-scanning direction. A line segment connecting the vertices A and C and a line segment connecting the vertices B and D are provided in parallel to the main scanning direction. In addition, a dot indicating the sub-scanning direction is provided near the vertex D. As described above, the reference marker 102 has a fixed arrangement position relative to the handy printer 100, and the position of the reference marker 102 also moves as the handy printer 100 scans.

As shown in FIG. 2, a scanning distance detector 130 and a print head 140 are provided on the bottom surface of the handy printer 100.

The scanning distance detection unit 130 includes a light source such as an LED (Light Emitting Diode) and an optical sensor, and detects the distance that the handy printer 100 is scanned in the sub-scanning direction.

The print head 140 includes a magenta print head 140M that ejects magenta ink, a cyan print head 140C that ejects cyan ink, a yellow print head 140Y that ejects yellow ink, and inks of each color not shown. It consists of a tank. The print heads 140M, 140C, and 140Y are provided with nozzles lined up in the main scanning direction. The main scanning direction is orthogonal to the sub scanning direction. The print head 140 selectively ejects ink from the nozzles by energization corresponding to the image of the content.

As shown in FIG. 3, the handy printer 100 has, in addition to the above-described scanning distance detection unit 130, print head 140, and start button 150, an electric configuration, a control unit 110, a communication unit (second unit). A communication unit) 120, a ROM (Read Only Memory) 160, and a RAM (Random Access Memory) 170 are provided.

The control unit 110 includes a CPU (Central Processing Unit) and the like. The control unit 110 functions as the content acquisition unit 111, the print head control unit 112, and the scanning distance output unit 113 by executing the program stored in the ROM 160.

The content acquisition unit 111 acquires the data of the content transmitted from the terminal device 200 via the communication unit 120 and stores it in the RAM 170.

The print head controller 112 causes the print head 140 to print an image based on the scanning distance scanned by the scanning distance detector 130 in the sub-scanning direction. Specifically, the print head control unit 112 scans the content image stored in the RAM 170 for each YMC (Yellow Magenta Cyan) every time the print head 140 scans a distance corresponding to one dot pitch in the sub-scanning direction. The print head 140 prints one dot line, which is a line of one pixel arranged in a line in the main scanning direction.

The scanning distance output unit 113 transmits data indicating the scanning distance detected by the scanning distance detecting unit 130 to the terminal device 200 via the communication unit 120.

The communication unit 120 receives the data indicating the content from the terminal device 200 and transmits the data indicating the scanning distance to the terminal device 200. The communication unit 120 is composed of a wireless communication module such as a wireless LAN (Local Area Network) or Bluetooth (registered trademark).

As described above, the scanning distance detection unit 130 detects the scanned distance, which is the distance that the print head 140 has scanned the print medium R, and outputs data indicating the detected scanned distance to the control unit 110. To do.

As described above, the print head 140 causes the nozzles provided in the print head 140 to selectively eject ink in accordance with print data under the control of the print head controller 112. The ejected ink adheres to the print medium R, and pixels (1 dot line) arranged in a line in the main scanning direction of the print head 140 are printed.

The start button 150 receives an instruction to start printing based on a user operation.

The ROM 160 is composed of a nonvolatile memory such as a flash memory, and stores the programs for the control unit 110 to realize various functions as described above. The RAM 170 is composed of a volatile memory and is used as a work area for the control unit 110 to execute programs for performing various processes. Further, the RAM 170 stores information such as YMC image data of content and scanning distance.

As shown in FIG. 4, the terminal device 200 includes a control unit 210, a communication unit (first communication unit) 220, a camera 230, a display 240, an operation unit 250, a ROM 260, and a RAM 270.

The control unit 210 is composed of a CPU and the like. The control unit 210 functions as the content acquisition unit 211, the image acquisition unit 212, the marker detection unit 213, and the image generation/display unit 214 by executing the program stored in the ROM 260.

The content acquisition unit 211 acquires data indicating the content received by the operation unit 250 or received by the communication unit 220. The content acquisition unit 211 also transmits content data to the handy printer 100 via the communication unit 220.

The image acquisition unit 212 acquires a live view image (captured image) LI in the physical space captured by the camera 230, and stores the acquired live view image LI in the RAM 270.

The marker detection unit 213 detects the reference marker 102 arranged in the handy printer 100 included in the live view image LI obtained by capturing the real space. Further, based on the reference marker 102, the position of the handy printer 100 in the image and the sub-scanning direction are specified. For example, the marker detection unit 213 determines whether or not the live view image LI includes the reference marker 102 that matches the pattern of the reference marker stored in the ROM 260 in advance. When it is determined that the reference marker 102 is included in the live view image LI, the marker detection unit 213 recognizes the four corners of the reference marker 102 shown in FIG. 1 (vertex A, vertex B, vertex C, vertex D). The marker detection unit 213 detects the position of the reference marker 102 included in the live view image LI from the positions of the four corners, and identifies the position of the handy printer 100 from the position of the reference marker 102. The marker detection unit 213 also determines the live view image LI based on the angle between the line segment AB and the line segment AC, the ratio of the length of the line segment AB to the length of the line segment CD, the direction of the line segment AB, and the like. The sub-scanning direction indicated by the reference marker 102 included in is detected, and the sub-scanning direction of the handy printer 100 in the live view image LI is specified from the sub-scanning direction indicated by the reference marker 102. Note that the upper, lower, left, and right sides of the reference marker 102 are recognized by the mark indicating the lower right attached near the vertex D.

Returning to FIG. 4, the image generation/display unit 214 displays the data indicating the content acquired by the content acquisition unit 211 and the arrangement position (first position) of the handy printer 100 in the live view image LI specified by the marker detection hand unit 213. Based on the (arrangement position) and the sub-scanning direction, an auxiliary image showing the handy printer 100 is generated, and this auxiliary image is displayed on the display 240 by superimposing it on the live view image LI in the real space captured by the camera 230. To do. Specifically, the image generation/display unit 214 obtains the remaining scanning distance until the printing is completed, based on the scanning distance required for printing the content and the scanned distance that the handy printer 100 has scanned. .. The image generation/display unit 214 also generates an auxiliary image of the handy printer 100 in the direction corresponding to the sub-scanning direction of the handy printer 100 specified by the marker detection unit 213. The image generation/display unit 214 is based on the position of the handy printer 100 identified by the marker detection unit 213 and the remaining scanning distance, and the handy printer 100 placed at the position when printing is completed (second placement position). Is displayed on the display 240 while being superimposed on the live view image LI.

The communication unit 220 transmits data indicating the content to the handy printer 100 and receives data indicating the scanning distance transmitted from the handy printer 100. The communication unit 220 is composed of a wireless communication module such as a wireless LAN or Bluetooth (registered trademark), similar to the communication unit 120 described above.

The camera 230 images the physical space. The camera 230 includes a lens, an image sensor, and the like, and outputs captured image data to the control unit 110. The image sensor is composed of a CMOS (Complementary Metal Oxide Semiconductor) image sensor or the like.

The display 240 displays an image generated by the image generation/display unit 214 and an image necessary for operation, and is configured by an LCD (Liquid Crystal Display) or the like.

The operation unit 250 accepts content data and instructions for starting and ending print processing based on a user input. The operation unit 250 and the display 240 constitute a touch panel display device.

The ROM 260 is composed of a non-volatile memory such as a flash memory, and stores a program for the control unit 110 to implement various functions, a reference marker pattern, an auxiliary image corresponding to the reference marker, and the like. The RAM 270 is composed of a volatile memory and is used as a work area for the control unit 110 to execute programs for performing various processes. The RAM 270 also stores information about the live view image LI captured by the camera 230, content, and scanning distance.

Next, the print assist process executed by the terminal device 200 having the above configuration will be described.

When the user captures an image of the handy printer 100 in the physical space with the camera 230 of the terminal device 200, the terminal device 200 displays a live view image of the arrangement position of the handy printer 100 when printing is finished before and during printing. The image is displayed on the display 240 as an image showing the augmented reality space superimposed on the LI. Therefore, the user can know to what position the handy printer 100 should be scanned before and during printing, so that the handy printer 100 can be easily scanned.

The terminal device 200 responds to the instruction by the user to start the process, and starts the print assist process shown in FIG. Hereinafter, the print assist process executed by the terminal device 200 will be described using a flowchart.

First, the content acquisition unit 211 receives the data of the print target content input to the operation unit 250 (step S101). The content acquisition unit 211 stores data indicating the received content in the RAM 270. Next, the content acquisition unit 211 determines whether or not an instruction to transmit the data indicating the content has been received (step S102). When the instruction to transmit the data has not been received (step S102; No), step S102 is repeated. When the instruction to transmit the data is received (step S102; Yes), the content acquisition unit 211 transmits the data indicating the content to the handy printer 100 via the communication unit 220 (step S103). Next, the controller 210 executes a display process (step S104).

In the display processing, as illustrated in FIG. 6, the image acquisition unit 212 causes the camera 230 to start capturing the live view image LI in the physical space (step S201). At this time, the user points the camera 230 toward the handy printer 100. Next, the image acquisition unit 212 acquires this live view image LI (step S202) and stores the acquired image in the RAM 270. Next, the marker detection unit 213 determines whether or not the live view image LI includes the reference marker 102 that matches the pattern of the reference marker stored in the ROM 260 in advance (step S203). When it is determined that the reference marker 102 is not included in the live view image LI (step S203; No), the live view image LI captured by the camera is displayed (step S204), and the process returns to step S202. When it is determined that the image includes the reference marker 102 (step S203; Yes), the image generation/display unit 214 acquires the current scanning distance data transmitted from the handy printer 100 (step S205). The current scanning distance transmitted from the handy printer 100 will be described later. The image generation/display unit 214 stores the data of the current scanning distance in the RAM 270.

The image generation/display unit 214 determines whether printing is completed (step S206). Specifically, the image generation/display unit 214 determines the scanning distance required for printing the content. Next, the image generation/display unit 214 determines whether the current scanning distance stored in the RAM 270 is less than the scanning distance required to print the content, and the current scanning distance is required to print the content. If it is less than the scanning distance, it is determined that printing is not completed. If it is determined that printing has not been completed (step S206; No), auxiliary image generation processing is executed (step S207).

In the auxiliary image generation process, as shown in FIG. 7, the marker detection unit 213 detects the position of the reference marker 102 included in the live view image LI and the sub-scanning direction indicated by the reference marker 102 (step S301). As shown in FIG. 9, the marker detection unit 213 recognizes the four corner vertices (vertex A, vertex B, vertex C, and vertex D) of the reference marker and recognizes the reference markers in the live-view image LI from the positions of the four corners. The position of 102 is detected. The marker detection unit 213 also determines the angle formed by the line segment connecting the vertices A and B and the line segment connecting the vertices A and C, and the length of the line segment connecting the vertices A and B. , The length of the line segment connecting the vertices C and D, the direction of the line segment connecting the vertices A and B, the sub-scanning direction indicated by the reference marker 102 in the live view image LI, and the like. To detect. Note that the upper, lower, left, and right sides of the reference marker 102 are recognized by the mark indicating the lower right attached near the vertex D.

Returning to FIG. 7, the image generation/display unit 214 detects the remaining scanning distance until the handy printer 100 completes printing the content (step S302). Specifically, the difference between the scanning distance required to print the entire content and the scanning distance acquired in step S205 is obtained. Next, the image generation/display unit 214 specifies the sub-scanning direction of the handy printer 100 in live view based on the sub-scanning direction indicated by the reference marker 102 detected by the marker detection unit 213 (step S303). The image generation/display unit 214 generates an auxiliary image of the handy printer 100 based on the sub-scanning direction of the handy printer 100 (step S304). Next, the image generation/display unit 214 uses this auxiliary for scanning the remaining scanning distance in the sub-scanning direction of the handy printer 100 in the live view image LI from the position of the handy printer 100 in the live view image LI. The image is superimposed on the live view image LI and displayed on the display 240 (step S305).

After that, the process of displaying the auxiliary image ends, the process returns to step S202 of the display process shown in FIG. 6, and the image acquisition unit 212 acquires the live view image LI captured by the camera 230. After that, step S202 to step S207 are repeated to generate an auxiliary image based on the reference marker 102 in the live view image LI captured by the camera, and display the generated auxiliary image on the display 240. When the image generation/display unit 214 determines that the printing is completed (step S206; Yes), the auxiliary image is cleared (step S208), the live view image LI is captured (step S209), and the display process is terminated. To do.

When the display process ends, the process returns to the print assist process shown in FIG. 5, and the content acquisition unit 211 determines whether or not an instruction to end the print assist process has been input (step S105). When the instruction to end the print assisting process is not input (step S105; No), the process returns to step S101, and the content acquisition unit 211 receives the content data. Then, steps S101 to S105 are repeated. When the instruction to end the print assisting process is input (step S105; Yes), the print assisting process is ended.

Next, the print processing executed by the handy printer 100 will be described.

In response to the user's instruction to start the print processing, the handy printer 100 starts the print processing shown in FIG.

First, the content acquisition unit 111 waits in a state where the data indicating the content can be received, and determines whether or not the data indicating the content has been received (step S401). When the data indicating the content has not been received (step S401; No), step S401 is repeated. When the data indicating the content is received (step S401; Yes), the print head control unit 112 stores the data indicating the content in the RAM 170, and waits in a state where printing can be started if the start button 150 is pressed (step). S402). Next, the print head controller 112 determines whether the start button 150 has been pressed (step S403). When it is determined that the button has not been pressed (step S403; No), the process returns to step S402, and steps S402 and S403 are repeated.

When the print head control unit 112 determines that the start button 150 has been pressed (step S403; Yes), the print head control unit 112 determines whether the print head 140 has scanned a distance corresponding to one dot pitch in the sub-scanning direction. (Step S404). When the distance corresponding to one dot pitch is not scanned (step S404; No), step S404 is repeated. When the distance corresponding to the 1-dot pitch is scanned (step S404; Yes), the print head control unit 112, the 1-dot line that is a line of 1 pixel arranged in a line in the main scanning direction of the image of the content. Is printed by the print head 140 (step S405). Next, the scanning distance output unit 113 transmits the scanning distance in which the print head 140 was scanned in the sub-scanning direction after the start button 150 was pressed, to the terminal device 200 via the communication unit 120 (step S406). .. Further, the scanning distance output unit 113 stores the scanning distance in the RAM 170.

Next, the print head controller 112 determines whether or not the content has been printed to the end (step S407). Specifically, when the scanning distance scanned by the handy printer 100 is less than the scanning distance required to print the entire content, it is determined that the content has not been printed to the end, and this scanning distance is If the scanning distance is equal to or larger than the scanning distance required to print all the contents, it is determined that the content is printed to the end. When it is determined that the content has not been printed to the end (step S407; No), steps S404 to S407 are repeated until the printing of the content is completed, each time a distance equivalent to the diameter of one dot is scanned. The print head controller 112 causes the print head 140 to print one dot line.

When the print head control unit 112 determines that the content has been printed to the end (step S407; Yes), the content acquisition unit 111 determines whether or not the user has received an instruction to end the print processing (step S408). When the instruction to end the print processing has not been received (step S408; No), the process returns to step S401, the content acquisition unit 111 waits in a state in which the content data can be received, and whether or not the content data is received. The determination is made, and steps S401 to S408 are repeated. When the instruction to end the print processing is received (step S408; Yes), the print processing ends.

Next, the processing executed by the printing system 1 according to the present embodiment will be described based on a specific example with reference to FIGS. 9 and 10.

The user captures the live view image LI of the handheld printer 100 with the camera 230 of the terminal device 200, so that the terminal device 200 arranges the handheld printer 100 at the position where the printing is finished before and during printing. Is displayed on the display 240 as an image superimposed on the live view image LI. The user can see how much the handy printer 100 should be scanned by looking at the display on the display 240, and can assist the user in printing the content on the print medium R.

In response to the user's operation, the handy printer 100 starts the printing process and the terminal device 200 starts the printing control process.

When the terminal device 200 starts the print assist process, the content acquisition unit 211 receives data indicating the content input to the operation unit 250 (step S101; FIG. 5). Next, when the content acquisition unit 211 receives an instruction to transmit data indicating content (step S102; Yes; FIG. 5), the content acquisition unit 211 transmits data indicating content to the handy printer 100 via the communication unit 220 (step S102). S103; FIG. 5). Next, since the handy printer 100 receives the data indicating the content, the content acquisition unit 111 determines that the data indicating the content has been received (step S401; Yes; FIG. 8), and the print head control unit 112 starts the process. If the button 150 is pressed, the printer stands by in a state where printing can be started (step S402; FIG. 8).

Next, the terminal device 200 executes a display process (step S104; FIG. 5). When the terminal device 200 executes the display process, the image acquisition unit 212 causes the camera 230 to start capturing an image of the physical space. At this time, the user images the reference marker 102 of the handy printer 100 with the camera 230. Next, the image acquisition unit 212 acquires the live view image LI (step S202; FIG. 6). Next, since the marker detection unit 213 determines that the reference marker 102 is included in the live view image LI (step S203; Yes; FIG. 6), the image generation display unit 214 is transmitted from the handy printer 100. The data of the scanning distance is acquired (step S205; FIG. 6). Since the handy printer 100 has not started printing at this point, the image generation/display unit 214 sets the scanning distance to zero. Since the scanning distance is 0, the image generation/display unit 214 determines that the printing is not completed (step S206; No; FIG. 6) and executes the auxiliary image display process (step S207; FIG. 6). The auxiliary image I shown in FIG. 9 in which the virtual image is superimposed on the live view image LI captured by the camera 230 is displayed on the display 240 (step S305; FIG. 7). In FIG. 9, the handy printer 100 arranged at the position at the end of printing shown by the dotted line and the line showing the movement path of the handy printer 100 are the auxiliary image I.

Next, when the start button 150 of the handy printer 100 is pressed (step S403; Yes; FIG. 8) and the distance corresponding to one dot pitch is scanned (step S404; Yes; FIG. 8), the print head control unit 112. Causes the print head 140 to print a 1-dot line, which is a line of 1 pixel arranged in a line in the main scanning direction of the image of the content (step S405; FIG. 8). Next, the scanning distance output unit 113 transmits the scanning distance at which the print head 140 was scanned in the sub-scanning direction after the start button 150 was pressed, to the terminal device 200 via the communication unit 120 (step). 406; FIG. 8).

Next, the terminal device 200 acquires the scanning distance data transmitted from the handy printer 100 (step S205; FIG. 6). When the received scanning distance is less than the scanning distance required to print the entire content, the image generation/display unit 214 determines that the printing is not completed (step S206; No; FIG. 6), and assists. Image display processing is executed (step S207; FIG. 6). In the auxiliary image display process, the image generation/display unit 214 calculates the remaining scanning distance until the printing is completed (step S302; FIG. 7). Next, as shown in FIG. 10, the image generation/display unit 214 superimposes the auxiliary image I on the live view image LI and displays it on the display 240 based on the position of the reference marker 102 and the remaining scanning distance. (Step S305; FIG. 7). At this time, the handy printer 100 is scanning and moving, but the auxiliary image of the handy printer 100 at the end of printing is displayed based on the remaining scanning distance, and therefore is shown at the same position as in FIG. Be done. In this way, the image of the handy printer 100 in the real space while the user is scanning the handy printer 100 and the auxiliary image of the handy printer 100 at the end of printing are superimposed and displayed on the display 240 in real time. ..

Next, when it is determined that the content has not been printed to the end (step S407; No; FIG. 8), steps S404 to S407 are repeated until the printing of the content is completed, and the distance corresponding to one dot pitch is scanned. Each time the printing is performed, the print head control unit 112 prints one dot line, which is a line of one pixel.

When the print head control unit 112 determines that the content has been printed to the end (step S407; Yes; FIG. 8), the content acquisition unit 111 receives an instruction to end the print processing (step S408; Yes; FIG. 8). ), the handy printer 100 ends the printing process. When the handy printer 100 finishes the printing process, the terminal device 200 determines that the printing is completed (step S206; Yes), clears the auxiliary image (step S208), and finishes capturing the live view image LI. Then, (step S209), the display process ends. After that, when an instruction to end the print assisting process is input (step S105; Yes), the print assisting process is ended.

As described above, according to the printing system 1 of the present embodiment, when the content is printed on the print medium R by the handy printer 100, the terminal device 200 functions as an augmented reality display device, and the handy printer 100 is provided to the user. An auxiliary image is displayed to inform the user of which position the printing should be completed. Further, even during printing, the auxiliary image of the handy printer 100 at the end of printing is sequentially displayed, so that the user can easily grasp the remaining scanning distance of the handy printer 100. In this way, by displaying the auxiliary image on the terminal device 200, it is possible to eliminate a failure such as stopping scanning during printing, and it is possible to assist the user in printing the content on the print medium R. .. Further, in the printing system 1, it is only necessary to dispose the reference marker 102 on the handy printer 100, and the handy printer 100 can be downsized as compared with the case where the handy printer 100 is provided with a display for displaying a preview of content. ..

(Modification)
In the printing system 1 of the above-described embodiment, the terminal device 200 displays the arrangement position and the movement route of the handy printer 100 at the end of printing as the auxiliary image, but the auxiliary image displayed by the terminal device 200 is The image is not particularly limited as long as it is an auxiliary image for the user to scan the handy printer 100. For example, the terminal device 200 displays a moving image scanned from the position where the handy printer 100 is currently arranged to the position where the handy printer 100 is located at the end of printing, by superimposing the moving image on the image captured by the handy printer 100. Good. By doing so, the user can know how to scan the handy printer 100 to print the content before printing, can prevent printing failure, and can print on the print medium R by the user. It is possible to assist the printing of the contents. Further, the terminal device 200 may display an arrangement position of the handy printer 100 at the end of printing with an arrow or the like. Further, the terminal device 200 may be shown by enclosing a region through which the handy printer 100 is scanned and passed by a line.

In addition, the terminal device 200, for example, as shown in FIG. 11, a preview image showing a print result when an unprinted portion of the print target content to be printed by the handy printer 100 is printed on the print medium R. The PI may be displayed as an auxiliary image so as to be superimposed on the live view image LI. In this example, the content is “ABCDE”. The image generation/display unit 214 generates a preview image PI indicating a print preview based on the position of the reference marker 102 in the live view image LI and the sub-scanning direction indicated by the reference marker 102. The image generation/display unit 214 superimposes the preview image PI on the live view image LI based on the position of the reference marker 102 in the live view image LI and the sub-scanning direction of the handy printer 100 specified by the reference marker 102. It is displayed on the display 240. By doing so, the user can know how to print the content before printing it, can prevent printing failure, and print the content on the print medium R by the user. Can be assisted.

Further, when the display terminal 200 displays a print image image, as shown in FIG. 12, when the user scans the handy printer 100 to print the content, the content to be printed is already printed by the handy printer 100. The printed image EI indicating the result of printing the printed portion on the print medium R may be superimposed on the live view image LI as an auxiliary image and may be identified and shown with respect to the preview image PI. Here, the printed content (AB portion) is displayed with hollow characters. The black portion (CDE portion) is the preview image PI of the content that has not been printed yet. Specifically, the image generation/display unit 214 identifies the printed content based on the scanning distance scanned by the handy printer 100, identifies the identified printed content from the content before printing, and displays the content. 240 is displayed. By doing so, the user can know to what part the print is currently performed, prevent printing failure such as stopping scanning during printing, and assist the user in printing the content on the print medium R. can do. Further, instead of changing the printed portion to a hollow portion, the printed portion may be indicated by a different color or the like.

In addition, the display terminal 200 may display an image image HI of the print head 140 that imitates the print head 140 of the handy printer 100, as shown in FIG. 13. The image generation/display unit 214 generates the image image HI of the print head 140 according to the position of the reference marker 102 in the image detected by the marker detection unit 213 and the sub-scanning direction indicated by the reference marker 102, and the live view image LI. It is displayed on the display 240 by superimposing it on the handy printer 100. By doing so, the user can easily grasp the position of the print head 140 of the handy printer 100, and easily align the print head 140. In this case, the image image HI of the print head 140 is composed of three short straight lines arranged in parallel as shown in FIG. 13, and each short straight line corresponds to the head of each color. When using the heads of all colors, one short straight line arranged on the rightmost side of the three lines corresponds to the position where the first printing is performed.

Further, in the above-described embodiment, the display terminal 200 superimposes and displays the auxiliary image on the live view image LI, but the display terminal 200 superimposes the auxiliary image on the still image as the captured image. It may be displayed.

Further, in the above-described embodiment, the marker detection unit 213 detects the reference marker 102 arranged in the handy printer 100 included in the live view image LI, and based on the reference marker 102, the handy printer 100 in the image. However, the present invention is not limited to this, and the whole or a part of the handy printer 100 included in the live view image LI, for example, the housing 101 of the handy printer 100 is detected by image recognition, and the handy printer 100 is detected. The position of the printer 100 and the sub-scanning direction may be specified. In this case, the housing 101 of the handy printer 100 functions as a reference marker. Here, the case 101 is assumed to perform the function of the reference marker, but a part thereof may function as the reference marker, or a part of the handy printer 100 other than the case 101 may function as the reference marker. Good.

In the printing system 1 of the above-described embodiment, the display image of the real space including the handy printer 100 is displayed on the display 240 of the terminal device 200, and each auxiliary image is displayed in the captured image. The configuration is not limited to this, and may be configured as follows. That is, the handy printer 100 is configured to include a display as an informing unit, and the above-mentioned captured image is transmitted to the control unit 110 of the handy printer 100 via the communication unit 220 of the terminal device 200 and the communication unit 120 of the handy printer 100. To the handy printer 100, and each auxiliary image may be displayed in the captured image. As described above, by displaying each auxiliary image in the captured image displayed on the display 240 of the handy printer 100, the user can be guided to move the handy printer 100 to the print start position. Further, in this case, the process of generating each auxiliary image may be performed by either the control unit 210 of the terminal device 200 or the control unit 110 of the handy printer 100, or between the terminal device 200 and the handy printer 100 as appropriate. The processing may be shared.

Although the reference marker 102 was displayed in the live view image LI at the start of printing, the reference marker 102 is displayed in the live view image LI by the hand of the user who manually scans the handy printer 100 during printing. When the image disappears, the image generation/display unit 215 of the terminal device 200 continues to display each auxiliary image using the part of the outer shape of the handy printer 100 displayed in the live view image LI as the reference marker 102. It may be displayed so as to be superimposed on the view image LI.

In addition, as shown in FIG. 14, the display terminal 200 displays an image image of a preprinted pattern (preprinted image) 300 such as a ruled line or a pattern of the print medium R that is hidden by the handy printer 100 in the live view image LI. The KI may be displayed. For example, based on the print medium R previously captured by the camera 230 and the image of the print medium R in the live view image LI that is not hidden by the handy printer 100, the print medium R is hidden by the handy printer 100 in the live view image LI. The image image KI of the portion of the print medium R is displayed so as to be superimposed on the live view image LI. In other words, the image generation display unit 214 of the display terminal 200 prints the outline of the handy printer 100 and the content to be printed in the live view image LI specified by the reference marker 102 detected by the marker detection unit 213. An image image KI of the print medium R in a portion hidden by the handy printer 100 is generated as an auxiliary image for assisting the scanning of the handy printer 100 based on the above-mentioned already-printed pattern 300 printed in advance on the print medium R. , The auxiliary image is displayed by being superimposed on the live view image LI. By doing so, the user can grasp the ruled lines, the pattern, etc. of the print medium in the portion hidden by the handy printer 100 during printing using the handy printer 100. Therefore, in the printing system 1 of the present modification example. Also, it becomes possible to assist the user in printing the content in accordance with the ruled lines or the pattern of the print medium.
Further, it goes without saying that the display form in the above-described embodiment and the display form in each modification may be appropriately combined.

Further, in the above-described embodiment, the auxiliary image is superimposed on the live view image LI showing the real space and displayed on the display 240 of the terminal device 200. However, the auxiliary image is a head mount such as a spectacles type display. It may be displayed on the display. In this case, the head mounted display displays the auxiliary image by superimposing it on the transparent real space. The user wearing the head mounted display can easily scan the handy printer 100 by seeing the auxiliary image superimposed on the transparent real space.

Further, the central part that performs the print assist process executed by the terminal device 200 including a CPU, a RAM, a ROM, etc. does not depend on a dedicated system, but is a normal information portable terminal (smartphone, tablet PC), personal computer. It can be executed using For example, a computer program for executing the above operation is stored in a computer-readable recording medium (flexible disc, CD (Compact Disc)-ROM, DVD (Digital Versatile Disc)-ROM, etc.) and distributed, An information terminal that executes the above-described processing may be configured by installing this computer program in an information portable terminal or the like. The computer program may be stored in a storage device included in a server device on a communication network such as the Internet and downloaded by a normal information processing terminal to configure the information processing device.

When the function of the print assisting device is realized by sharing of an OS (Operating System) and an application program or by cooperation between the OS and an application program, only the application program part is stored in a recording medium or a storage device. You may.

It is also possible to superimpose a computer program on a carrier wave and distribute it via a communication network. For example, this computer program may be posted on a bulletin board (BBS: Bulletin Board System) on a communication network, and the computer program may be distributed via the network. Then, this computer program may be started up and executed under the control of the OS in the same manner as other application programs so that the above-described processing can be executed.

Although the preferred embodiment of the present invention has been described above, the present invention is not limited to the specific embodiment, and the present invention includes the invention described in the claims and an equivalent range thereof. included. The inventions described in the initial claims of the present application will be additionally described below.

(Appendix 1)
Marker detection means for detecting a reference marker arranged in a manual scanning type printing device from a captured image of the printing device,
Generating an auxiliary image for assisting the scanning of the printing device, based on a first arrangement position that is the position of the printing device in the captured image specified by the reference marker detected by the marker detection means, Image generation and display means for displaying the auxiliary image by superimposing it on the captured image;
A print assisting device comprising:

(Appendix 2)
Marker detection means for detecting a reference marker arranged in a manual scanning type printing device from a captured image of the printing device,
On the basis of the outer shape of the printing device in the captured image specified by the reference marker detected by the marker detection unit and the pre-printed image that has been printed in advance on the print medium on which the content to be printed is printed, Image generation and display means for generating an auxiliary image for assisting scanning, and displaying the auxiliary image by superimposing it on the captured image;
A print assisting device comprising:

(Appendix 3)
The image generation/display unit has an arrangement position and a scanning direction of the printing device in the captured image specified by the reference marker detected by the marker detection unit, and a distance required to print the content to be printed. An image indicating a second arrangement position, which is the position of the printing device when the printing of the content by the printing device is completed, is generated based on a certain required scanning distance, and the image is set to the printing device. Display on the captured image,
The printing auxiliary apparatus according to appendix 1 or 2 characterized in that.

(Appendix 4)
The image generation/display unit displays a preview image showing a print result of the content to be printed based on the content to be printed, the first arrangement position which is the position of the printing device in the captured image, and the scanning direction. Is generated and the preview image is displayed by being superimposed on the captured image.
4. The print assisting device according to any one of appendices 1 to 3, characterized in that.

(Appendix 5)
The image generation/display unit acquires data indicating a scanned distance, which is a distance scanned after starting printing, from the printing apparatus, and prints the content to be printed and the position of the printing apparatus in the captured image. Based on the certain first arrangement position and scanning direction and the scanned distance, the printing apparatus generates a printed image showing a print result obtained by scanning the scanned distance and printing the printed image. Is displayed by being superimposed on the captured image,
5. The print assisting device according to any one of appendices 1 to 4, characterized in that.

(Appendix 6)
The image generation/display unit displays an image indicating a position of a printer head of the printing device in an overlapping manner on the captured image based on a first arrangement position which is a position of the printing device in the captured image.
6. The printing assisting device according to any one of appendices 1 to 5, characterized in that.

(Appendix 7)
A communication unit that receives the content to be printed from the print auxiliary apparatus according to any one of appendices 1 to 6;
Printing means for printing based on the content;
A printing apparatus comprising:

(Appendix 8)
A communication unit that receives the content to be printed from the print auxiliary apparatus according to any one of appendices 1 to 6;
A printing unit that performs printing based on the content,
The reference marker is arranged,
A printing device characterized by the above.

(Appendix 9)
A manual scanning type printing device,
From the print assisting device including an image capturing unit that captures a physical space including a reference marker arranged in the printing device and a first communication unit that transmits information about the captured image captured by the image capturing unit, Second communication means for receiving information,
Marker detection means for detecting the reference marker from the received captured image,
An image in which an image indicating the arrangement position and the scanning direction of the printing device in the captured image specified by the reference marker detected by the marker detecting unit is displayed by being superimposed on the captured image including the image of the printing device. Generation display means,
A printing apparatus comprising:

(Appendix 10)
A manual scanning type printing device,
Imaging means for imaging a real space including a reference marker arranged in the printing device, marker detection means for detecting the reference marker from a captured image, and the imaging specified from the reference marker detected by the marker detection means Second communication means for receiving information on the arrangement position and the scanning direction of the printing device from a print auxiliary apparatus, which comprises first communication means for transmitting information on the arrangement position and the scanning direction of the printing device in the image. ,
An image generation and display unit that displays an image indicating the arrangement position and the scanning direction of the printing device based on the received information of the arrangement position and the scanning direction of the printing device, the image being superimposed on the captured image.
A printing apparatus comprising:

(Appendix 11)
A print auxiliary device according to any one of appendices 1 to 6,
The printing device in which the reference marker is arranged,
A printing system comprising:

(Appendix 12)
A marker detection step of detecting a reference marker arranged in the manual scanning type printing device from a captured image of the printing device,
Generating an auxiliary image for assisting scanning of the printing device, based on a first arrangement position that is the position of the printing device in the captured image specified from the reference marker detected in the marker detection step, An image generation and display step of displaying the auxiliary image by superimposing it on the captured image;
A printing assistance method comprising:

(Appendix 13)
A marker detection step of detecting a reference marker arranged in the manual scanning type printing device from a captured image of the printing device,
Based on the outer shape of the printing device in the captured image identified from the reference marker detected in the marker detection step and the pre-printed image that has been printed in advance on the print medium on which the content to be printed is printed, An image generation and display step of generating an auxiliary image for assisting scanning, and displaying the auxiliary image by superimposing it on the captured image,
A printing assistance method comprising:

(Appendix 14)
Computer,
Marker detecting means for detecting a reference marker arranged in a manual scanning type printing device from a captured image of the printing device,
Generating an auxiliary image for assisting the scanning of the printing device, based on a first arrangement position that is the position of the printing device in the captured image specified by the reference marker detected by the marker detection means, Image generating and displaying means for displaying the auxiliary image by superimposing it on the captured image,
Function as
A program characterized by that.

(Appendix 15)
Computer,
Marker detecting means for detecting a reference marker arranged in a manual scanning type printing device from a captured image of the printing device,
On the basis of the outer shape of the printing device in the captured image specified by the reference marker detected by the marker detection unit and the pre-printed image that has been printed in advance on the print medium on which the content to be printed is printed, Image generation and display means for generating an auxiliary image for assisting scanning, and displaying the auxiliary image by superimposing it on the captured image,
Function as
A program characterized by that.

DESCRIPTION OF SYMBOLS 1... Printing system, 100... Handy printer, 101... Case, 102... Reference marker, 110, 210... Control part, 111, 211... Content acquisition part, 112... Print head control part, 113... Scanning distance output part, 120 , 220... Communication unit, 130... Scanning distance detection unit, 140... Print head, 140M... Magenta color print head, 140C... Cyan color print head, 140Y... Yellow print head, 150... Start button, 160, 260... ROM, 170 270... RAM, 200... Terminal device, 212... Image acquisition section, 213... Marker detection section, 214... Image generation display section, 230... Camera, 240... Display, 250... Operation section, 300... Preprinted pattern, R... Print medium, LI... Live view image, I... Auxiliary image, EI... Printed image, PI... Preview image, HI, KI... Image image

Claims (14)

Marker detection means for detecting a reference marker arranged in a manual scanning type printing device from a captured image of the printing device,
Generating an auxiliary image for assisting the scanning of the printing device, based on a first arrangement position that is the position of the printing device in the captured image specified by the reference marker detected by the marker detection means, Image generating and displaying means for displaying the auxiliary image and the image showing the position of the printer head of the printing device in an overlapping manner on the captured image;
A print assisting device comprising: Marker detection means for detecting a reference marker arranged in a manual scanning type printing device from a captured image of the printing device,
On the basis of the outer shape of the printing device in the captured image specified by the reference marker detected by the marker detection unit and the pre-printed image that has been printed in advance on the print medium on which the content to be printed is printed, As an auxiliary image for assisting scanning, an image generation and display unit that generates the already printed image of a portion hidden by the printing device, and displays the auxiliary image by superimposing it on the captured image,
A print assisting device comprising: The image generation/display unit has an arrangement position and a scanning direction of the printing device in the captured image specified by the reference marker detected by the marker detection unit, and a distance required to print the content to be printed. An image indicating a second arrangement position, which is the position of the printing device when the printing of the content by the printing device is completed, is generated based on a certain required scanning distance, and the image is set to the printing device. Display on the captured image,
The printing auxiliary apparatus according to claim 1, wherein the printing auxiliary apparatus is a printing auxiliary apparatus. The image generation/display unit displays a preview image showing a print result of the content to be printed based on the content to be printed, the first arrangement position which is the position of the printing device in the captured image, and the scanning direction. Is generated and the preview image is displayed by being superimposed on the captured image.
The print assisting device according to claim 1, wherein the print assisting device is a print assisting device. The image generation/display unit acquires data indicating a scanned distance, which is a distance scanned after starting printing, from the printing apparatus, and prints the content to be printed and the position of the printing apparatus in the captured image. Based on the certain first arrangement position and scanning direction and the scanned distance, the printing apparatus generates a printed image showing a print result obtained by scanning the scanned distance and printing the printed image. Is displayed by being superimposed on the captured image,
The print assisting device according to claim 1, wherein the print assisting device is a print assisting device. The image generation/display unit displays an image indicating a position of a printer head of the printing device in an overlapping manner on the captured image based on a first arrangement position which is a position of the printing device in the captured image.
The print assisting device according to claim 2, wherein the print assisting device is a print assisting device. A communication unit that receives the content to be printed from the print auxiliary apparatus according to claim 1.
Printing means for printing based on the content;
A printing apparatus comprising: A communication unit that receives the content to be printed from the print auxiliary apparatus according to claim 1.
A printing unit that performs printing based on the content,
The reference marker is arranged,
A printing device characterized by the above. A manual scanning type printing device,
From the print assisting device including an image capturing unit that captures a physical space including a reference marker arranged in the printing device and a first communication unit that transmits information about the captured image captured by the image capturing unit, Second communication means for receiving information,
Marker detection means for detecting the reference marker from the received captured image,
An image showing the arrangement position and the scanning direction of the printing device and the image showing the position of the printer head of the printing device in the captured image specified by the reference marker detected by the marker detecting means are the images of the printing device. Image generation and display means for displaying the captured image in a superposed manner,
A printing apparatus comprising: A print auxiliary device according to any one of claims 1 to 6,
The printing device in which the reference marker is arranged,
A printing system comprising: A marker detection step of detecting a reference marker arranged in the manual scanning type printing device from a captured image of the printing device,
Generating an auxiliary image for assisting scanning of the printing device, based on a first arrangement position that is the position of the printing device in the captured image specified from the reference marker detected in the marker detection step, An image generation and display step of displaying the auxiliary image and an image showing the position of the printer head of the printing device in an overlapping manner on the captured image;
A printing assistance method comprising: A marker detection step of detecting a reference marker arranged in the manual scanning type printing device from a captured image of the printing device,
Based on the outer shape of the printing device in the captured image identified from the reference marker detected in the marker detection step and the pre-printed image that has been printed in advance on the print medium on which the content to be printed is printed, As an auxiliary image for assisting scanning, an image generating and displaying step of generating the already printed image of a portion hidden by the printing device , and displaying the auxiliary image by superimposing it on the captured image,
A printing assistance method comprising: Computer,
Marker detecting means for detecting a reference marker arranged in a manual scanning type printing device from a captured image of the printing device,
Generating an auxiliary image for assisting the scanning of the printing device, based on a first arrangement position that is the position of the printing device in the captured image specified by the reference marker detected by the marker detection means, Image generation and display means for displaying the auxiliary image and the image showing the position of the printer head of the printing device in an overlapping manner on the captured image;
Function as
A program characterized by that. Computer,
Marker detecting means for detecting a reference marker arranged in a manual scanning type printing device from a captured image of the printing device,
On the basis of the outer shape of the printing device in the captured image specified by the reference marker detected by the marker detection unit and the pre-printed image that has been printed in advance on the print medium on which the content to be printed is printed, As an auxiliary image for assisting scanning, an image generation and display unit that generates the already printed image of a portion hidden by the printing device and displays the auxiliary image by superimposing it on the captured image.
Function as
A program characterized by that.

Images