JP2017004076A - Information processing apparatus, image formation system and image formation apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing apparatus which can resume image formation even when a user erroneously floats a hand-held printer in printing, and which suppresses cost increase.SOLUTION: An information processing apparatus (image data output apparatus 11) which can communicate with an image formation apparatus (hand-held printer 20) forming an image on the basis of image data while moving on a recording medium comprises: an imaging unit 52 which acquires imaging data in which a position detection mark formed on the recording medium is imaged; a mark detection unit 53 which detects the image formation apparatus and formed image from the imaging data; a position decision unit 55 which estimates the position of the image formation apparatus on the recording medium by using the image data used for formation of the formed image and the relative position of the image formation apparatus with respect to the formed image detected by the mark detection unit; and a transmission/reception unit 51 which transmits the position estimated by the position estimation means to the image formation apparatus.SELECTED DRAWING: Figure 6

Description

  The present invention relates to an information processing apparatus, an image forming system, and an image forming apparatus.

  There is known a printer device that forms an image by ejecting ink or the like at a timing when the sheet is conveyed and the sheet reaches an image forming position. On the other hand, a printer device (hereinafter referred to as a handheld printer) reduced in size by deleting the paper transport system from the printer device is being put into practical use.

  FIG. 1 is an example of a diagram schematically illustrating image formation by the handheld printer 20. Image data is transmitted to the handheld printer 20 from an image data output device 11 such as a smartphone or a PC (Personal Computer). The user holds the handheld printer 20 and scans the print medium 12 (for example, standard paper or notebook) with a freehand. Since the handheld printer 20 has a position detection mechanism, when the handheld printer 20 moves to the target discharge position, ink of a color to be discharged at the target discharge position is discharged. Since the area where ink has already been ejected is masked (because it is not a target for ink ejection), the user can form an image by scanning the handheld printer 20 in any direction on the print medium 12.

  However, since scanning of the handheld printer 20 is left to the user, the user may accidentally lift the handheld printer 20 from the print medium 12. In this case, the position detected by the handheld printer 20 may not be accurate. In addition, the user may scan the handheld printer 20 at a speed higher than the specification. In this case, the position may not be accurate. In these cases, it becomes difficult for the handheld printer 20 to lose sight of the current position and resume image formation.

  For such inconvenience, a handheld printer 20 that re-recognizes the current position has been devised (see, for example, Patent Document 1). Patent Document 1 discloses a handheld printer that periodically prints position information with invisible ink, reads the position information with a scanner unit when the current position is lost, recognizes the current position, and resumes image formation. Is disclosed.

  However, the handheld printer described in Patent Document 1 has a problem of increasing costs because it is necessary to mount a unit for ejecting invisible ink and a scanner unit. In addition, the operability may be reduced due to the increase in weight and the size of the housing.

  In view of the above problems, an object of the present invention is to provide an information processing apparatus capable of resuming image formation while suppressing an increase in cost.

  In view of the above problems, the present invention provides an information processing apparatus capable of communicating with an image forming apparatus that forms an image based on image data while moving a position on a print medium, wherein the image forming apparatus and the image forming apparatus are Used for forming the formed image, an acquisition unit that acquires imaging data in which a formed image formed on a print medium is captured, an image processing unit that detects the image forming apparatus and the formed image from the imaged data, and Position estimation means for estimating the position of the image forming apparatus on the print medium using the image data and a relative position of the image forming apparatus with respect to the formed image detected by the image processing means; and the position estimation Transmitting means for transmitting the position estimated by the means to the image forming apparatus.

  It is possible to provide an information processing apparatus capable of suppressing the increase in cost and the like and restarting image formation.

It is an example of the figure which shows typically image formation by a handheld printer. It is an example of the figure explaining the outline of the procedure which acquires the present position which the handheld printer lost sight of. 1 is an example of a system configuration diagram of an image forming system. It is an example of the hardware block diagram of an image data output device and a handheld printer. It is an example of the figure explaining the structure of a control part. It is an example of a functional block diagram of an image data output device and a handheld printer. FIG. 3 is an example of a diagram illustrating nozzle positions and the like in an IJ recording head. It is an example of the figure explaining the position of a handheld printer. It is a figure which shows the example of arrangement | positioning of the position detection mark in the top view of a handheld printer. It is an example of the figure which shows the relative position of a position detection mark and a navigation sensor. It is an example of the figure explaining the position of the handheld printer in imaging data. It is an example of the figure which illustrates the pattern matching using the image for collation with respect to image data typically. It is a figure which shows an example of the inquiry screen displayed on the display apparatus. It is a figure which shows an example of the pattern matching which paid its attention to the length of the edge | side of the image for collation. It is an example of the figure explaining the position of the handheld printer in image data. FIG. 3 is an example of a flowchart illustrating an operation procedure of the image forming system. (Example 2) which is an example of the functional block diagram of an image data output device. It is an example of the figure explaining distortion of imaging data. It is an example of the figure which each shows the image for collation before and after conversion. It is a figure which shows an example of the non-front mark arrange | positioned with a handheld printer. It is an example of the figure which illustrates typically imaging of a handheld printer by an image data output device. (Example 3) which is an example of the flowchart figure which shows the imaging procedure of an image data output device. (Example 4) which is an example of the flowchart figure which shows the imaging procedure of an image data output device.

  DESCRIPTION OF EMBODIMENTS Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.

<Outline of acquisition of handheld printer position>
First, an outline of a procedure for acquiring the current position when the handheld printer 20 loses sight of the current position will be described with reference to FIG. FIG. 2 is an example of a diagram illustrating an outline of a procedure for acquiring the current position that the handheld printer 20 has lost.
(i) As shown in FIG. 2A, the user scans the handheld printer 20 on a print medium (an example of a recording medium) to form an image. It is assumed that the handheld printer 20 has lost sight of the current position before all the images are formed.
(ii) The user grasps the state of the handheld printer 20 with an LED lamp or an alarm sound. Then, the user places the handheld printer 20 at an appropriate location on the print medium 12, and images the handheld printer 20 and the print medium 12 with the image data output device 11 (FIG. 2B). Here, position detection marks M1 and M2 (an example of a first mark) are formed in the handheld printer 20 in advance.
(iii) The left diagram in FIG. 2C shows the imaged data 14 of the handheld printer 20 and the print medium 12 that have been imaged. The image data output device 11 detects the position detection marks M1 and M2 of the handheld printer 20 from the imaging data 14. Further, the image data output unit 11 acquires an image around the handheld printer 20 or near the image (hereinafter referred to as a collation image 14 a) from the imaging data 14. Then, the image data output unit 11 searches for a matching image 13a that matches the matching image 14a from the image data 13 used for image formation (pattern matching).
(iv) FIG. 2D shows the image data 13. As shown in FIG. 2C, since the relative positions of the position detection marks M1, M2 with respect to the matching image 14a are known in the imaging data 14, the position detection marks M1, M2 with respect to the coincidence image 13a also in the image data 13. The relative position can be specified. That is, the position of the handheld printer 20 in the image data 13 can be specified.
(v) The image data output device 11 transmits the position of the handheld printer 20 (or the position of the position detection marks M1 and M2 itself) to the handheld printer 20. As a result, as shown in FIG. 2E, the handheld printer 20 can acquire the current position and resume image formation.

  Therefore, even if the handheld printer 20 of the present embodiment loses sight of the position, it can suppress the increase in cost, acquire the position, and resume image formation.

<Terminology>
Terms used in this embodiment will be described.
Image data is data used for image formation, and is held by an image data output device.
The imaging data is data obtained by imaging the print medium 12 and the handheld printer 20.

  The loss of the position of the handheld printer 20 means that the current position cannot be calculated or that the position is extremely inaccurate. Factors include the fact that the handheld printer 20 floats from the paper and that the user scans the handheld printer 20 at a speed higher than the specification. However, in this embodiment, the factor is not limited.

  The print medium 12 is, for example, a standard paper or a notebook, but is not limited thereto, and a fixed object such as a desk may be used as the print medium 12, or a sheet-like member other than paper may be used as the print medium 12. Further, the print medium 12 is not necessarily arranged horizontally.

<Configuration example>
FIG. 3 shows an example of a system configuration diagram of the image forming system 1. The image forming system 1 in FIG. 3A includes an image data output device 11 and a handheld printer 20 that can communicate with each other. In such a system configuration, the image data output unit 11 transmits the image data 13 to the handheld printer 20, and when the handheld printer 20 loses its position, the image data output unit 11 captures the handheld printer 20 and the print medium 12. To do. The image data output unit 11 detects the position of the handheld printer 20 from the imaging data 14 and transmits it to the handheld printer 20.

  In the system configuration of FIG. 3A, the handheld printer 20 may detect the position of the handheld printer 20. In this case, the image data output device 11 transmits the imaging data 14 to the handheld printer 20. The handheld printer 20 holds at least a part of the image data 13, but acquires the entire image data 13 from the image data output unit 11 if necessary. The handheld printer 20 detects the position of the handheld printer 20 in the same manner as the image data output device 11.

  The image forming system 1 in FIG. 3B includes an image data output device 11, a handheld printer 20, and an imaging device 8. In such a system configuration, when the handheld printer 20 loses its position, the imaging device 8 images the handheld printer 20 and the print medium 12. Since the imaging device 8 transmits the imaging data 14 to the image data output device 11, the image data output device 11 detects the position of the handheld printer 20 from the imaging data 14 and transmits it to the handheld printer 20.

  Also in this case, the handheld printer 20 may detect the position of the handheld printer 20. In this case, the imaging device 8 transmits the imaging data 14 to the handheld printer 20 (may be transmitted via the image data output unit 11). The handheld printer 20 detects the position of the handheld printer 20 as in FIG.

  Further, the imaging device 8 may detect the position of the handheld printer 20. In this case, the imaging device 8 acquires the image data 13 from the image data output device 11 and detects the position of the handheld printer 20. The imaging device 8 transmits the detected position to the handheld printer 20.

  In the image forming system 1 of the present embodiment, the position of the handheld printer 20 can be detected as long as the apparatus acquires the image data 13 and the imaging data 14 as described above. Therefore, as illustrated in FIG. 3C, the server 9 may detect the position of the handheld printer 20 by the image data output unit 11 transmitting the image data 13 and the imaging data 14 to the server 9. The server 9 transmits the detected position to the image data output unit 11, and the image data output unit 11 transmits this position to the handheld printer 20.

  Hereinafter, for convenience of explanation, it is assumed that the image data output unit 11 detects the position of the handheld printer 20.

  Note that the image data output unit 11 is an information processing apparatus. For example, a smart phone, a PC (Personal Computer), a tablet terminal, a PDA (Personal Digital Assitant), a mobile phone, or a wearable PC. Alternatively, an MFP (Multifunctional Peripheral / Printer / Product), various cameras, an electronic blackboard, a game device, a car navigation terminal, a projector, or the like may be used.

<< Hardware configuration >>
The hardware configuration of the image data output device 11 and the handheld printer 20 will be described with reference to FIG. The image data output unit 11 is realized by a hardware configuration as shown in FIG. The image data output unit 11 includes an input device 101, a display device 102, an external I / F 103, a camera 110, a RAM 104, a ROM 105, a CPU 106, a communication I / F 107, an SSD (Solid State Drive) 108, a short-range wireless communication device 109, and the like. Are connected to each other via a bus B.

  The input device 101 is a touch panel, for example, and is used to input each operation signal to the image data output unit 11. The input device 101 may be a keyboard or a mouse. The display device 102 is, for example, an LCD (Liquid Crystal Display) or the like, and displays a processing result by the image data output unit 11.

  The external I / F 103 is an interface with an external device. The external device includes a recording medium 103a. The recording medium 103a can store a program (an output device program to be described later) for realizing the functions of the present embodiment. The image data output device 11 can read and / or write to the recording medium 103 a via the external I / F 103.

  The recording medium 103a is a recording medium such as an SD memory card. The recording medium 103a may be a recording medium such as a USB memory (Universal Serial Bus memory), a DVD (Digital Versatile Disk), a CD (Compact Disk), or a flexible disk.

  The camera 110 is an image pickup apparatus including a lens for forming an image, an aperture, an image pickup device such as a CMOS or a CCD. It can capture moving images as well as still images.

  The RAM 104 is a volatile semiconductor memory (storage device) that temporarily stores programs and data. The ROM 105 is a nonvolatile semiconductor memory (storage device) that can retain programs and data even when the power is turned off. The ROM 105 stores programs and data such as BIOS (Basic Input / Output System), OS settings, and network settings that are executed when the image data output device 11 is started.

  The CPU 106 is an arithmetic unit that realizes control and functions of the entire image data output unit 11 by reading a program and data from a storage device such as the ROM 105 and the SSD 108 onto the RAM 104 and executing processing.

  The communication I / F 107 is an interface for performing communication. The communication I / F 107 communicates with the handheld printer 20 using a communication standard such as Bluetooth (registered trademark) or wireless LAN. Note that there are Wi-Fi Direct, ad hoc mode, and the like as wireless LAN communication standards that do not use an access point. The communication I / F 107 may be an interface for connecting to a mobile phone network.

  The SSD 108 is a nonvolatile storage device that stores programs and data. The stored programs and data include, for example, an OS (Operating System) that is basic software for controlling the entire image data output device 11, and application software that provides various functions on the OS (including an output device program to be described later). and so on. Note that the image data output device 11 may include an HDD (Hard Disk Drive) or the like instead of the SSD 108 or in combination with the SSD 108.

  The short-range wireless communication device 109 performs communication according to a communication standard using an IC chip such as NFC (Near Field Communication) or TransferJet (registered trademark), for example. The image data output device 11 transmits the image data 13 to the handheld printer 20 using the communication I / F or the short-range wireless communication device 109. In this case, the handheld printer 20 can also communicate with the IC chip.

  Next, the hardware configuration of the handheld printer 20 will be described with reference to FIG. The handheld printer 20 is an example of an image forming apparatus that forms an image on the print medium 12. The entire operation of the handheld printer 20 is controlled by a control unit 25, and a communication I / F 27, an IJ recording head drive circuit 23, an OPU 26, a ROM 28, a DRAM 29, and a navigation sensor 30 are electrically connected to the control unit 25. ing. Further, since the handheld printer 20 is driven by electric power, it has a power supply 22 and a power supply circuit 21. The power generated by the power supply circuit 21 is transmitted to the communication I / F 27, the IJ recording head drive circuit 23, the OPU 26, the ROM 28, the DRAM 29, the IJ recording head 24, the control unit 25, and the navigation sensor 30 through a wiring indicated by a dotted line 22a. Have been supplied.

  The power source 22 is mainly a battery. A solar cell, a commercial power source (AC power source), a fuel cell, or the like may be used. The power supply circuit 21 distributes the power supplied from the power supply 22 to each part of the handheld printer 20. Further, the voltage of the power supply 22 is stepped down or boosted to a voltage suitable for each part. In addition, when the power source 22 can be charged by a battery, the power source circuit 21 detects the connection of the AC power source and connects it to the battery charging circuit so that the power source 22 can be charged.

  The communication I / F 27 receives image data from the image data output device 11 such as a smartphone or a PC. The communication I / F 27 is a communication device that supports communication standards such as wireless LAN, Bluetooth (registered trademark), NFC (Near Field Communication), infrared, 3G (mobile phone), or LTE (Long Term Evolution). In addition to such wireless communication, a communication device that supports wired communication using a wired LAN, a USB cable, or the like may be used.

  The ROM 28 stores firmware for controlling the hardware of the handheld printer 20, drive waveform data of the IJ recording head 24 (data defining voltage change for discharging droplets), initial setting data of the handheld printer 20, and the like. doing.

  The DRAM 29 is used for storing image data received by the communication I / F 27 and storing firmware developed from the ROM 28. Therefore, it is used as a work memory when the CPU 31 executes firmware.

  The navigation sensor 30 is a sensor that detects the position of the handheld printer 20. The navigation sensor 30 includes, for example, a light source such as a light emitting diode (LED) or a laser, an image sensor that images the print medium 12, or a sensor that images interference fringes of incident / reflected light. When the handheld printer 20 is scanned over the print medium 12, minute edges of the print medium 12 are detected (imaged) one after another, and the amount of movement is obtained by analyzing the distance between the edges. The navigation sensor 30 is mounted in at least two places of the handheld printer 20. When distinguishing both, it is called navigation sensor S0, S1. Further, as the navigation sensor 30, a multi-axis acceleration sensor, a gyro sensor, or the like may be used, or the position of the handheld printer 20 may be detected only by the acceleration sensor or the gyro sensor.

  An OPU (Operation panel Unit) 26 has an LED for displaying the state of the handheld printer 20, a switch for the user to instruct the handheld printer 20 to form an image, and the like. However, the present invention is not limited to this, and may have a liquid crystal display and may further have a touch panel. Further, it may have a voice input function.

  The IJ recording head driving circuit 23 generates a driving waveform (voltage) for driving the IJ recording head 24 using the above driving waveform data. A drive waveform corresponding to the ink droplet size can be generated.

  The IJ recording head 24 is a head for ejecting ink. In the figure, four colors of CMYK ink can be ejected, but it may be a single color or ejecting five or more colors. A plurality of ink ejection nozzles (described later) arranged in a row (may be two or more rows) for each color. Further, the ink ejection method may be a piezo method or a thermal method, or any other method.

  Based on the amount of movement detected by the navigation sensor 30, the control unit 25 determines the position of each nozzle of the IJ recording head 24, determines the image to be formed according to the position, determines whether or not the discharge nozzle is available (the nozzle position is the target discharge position). To determine whether it is within a predetermined range). Details of the control unit 25 will be described below.

  FIG. 5 is an example of a diagram illustrating the configuration of the control unit 25. The control unit 25 has a SoC 50 and an ASIC / FPGA 40. The SoC 50 and the ASIC / FPGA 40 communicate via buses 45 and 46. This means that the ASIC / FPGA 40 may be designed by any mounting technology, and may be configured by other mounting technology other than the ASIC / FPGA 40. Further, the SoC 50 and the ASIC / FPGA 40 may be configured as a single chip or substrate without using separate chips. Alternatively, it may be implemented with three or more chips or substrates.

  The SoC 50 has functions such as a CPU 31, a position calculation circuit 32, a memory CTL (controller) 35, and a ROM CTL (controller) 36 connected via a bus 46. In addition, the component which SoC50 has is not restricted to these.

  The ASIC / FPGA 40 includes an Image RAM 37, a DMAC 38, a rotator 39, an interrupt controller 41, a navigation sensor I / F 42, a print / sensor timing generation unit 43, and an IJ recording head control unit 44 connected via a bus 45. have. In addition, the component which ASIC / FPGA40 has is not restricted to these.

  The CPU 31 executes firmware (program) or the like developed from the ROM 28 to the DRAM 29, and controls operations of the position calculation circuit 32, the memory CTL 35, and the ROM CTL 36 in the SoC 50. Further, it controls the operations of the Image RAM 37, DMAC 38, rotator 39, interrupt controller 41, navigation sensor I / F 42, print / sensor timing generation unit 43, and IJ recording head control unit 44 in the ASIC / FPGA 40.

  The position calculation circuit 32 calculates the position (coordinate information) of the handheld printer 20 based on the movement amount for each sampling cycle detected by the navigation sensor 30. Strictly speaking, the position of the handheld printer 20 is the position of the nozzle 61. However, if the position of the navigation sensor 30 is known, the position of the nozzle 61 can be calculated. In this embodiment, the position of the navigation sensor 30 is the position of the navigation sensor S0, S1 unless otherwise specified. The position calculation circuit 32 calculates a target discharge position.

  As will be described later, the position of the navigation sensor 30 is calculated based on, for example, a predetermined origin (an initial position of the handheld printer 20 when image formation is started). Further, the position calculation circuit 32 estimates the moving speed and the moving direction based on the difference between the past position and the newest position, and predicts the position at the next calculation timing, for example. In this way, it is possible to discharge ink while suppressing a delay with respect to scanning by the user.

  The memory CTL 35 is an interface with the DRAM 29, requests data from the DRAM 29, sends the acquired firmware to the CPU 31, and sends the acquired image data to the ASIC / FPGA 40.

  The ROM CTL 36 is an interface with the ROM 28, requests data from the ROM 28, and sends the acquired data to the CPU 31 and the ASIC / FPGA 40.

  The DMAC 38 acquires the image data 13 around each nozzle of the IJ recording head 24 via the memory CTL 35 based on the position information calculated by the position calculation circuit 32. That is, image data around the position where the handheld printer 20 is present with respect to the print medium 12 is acquired.

  The rotator 39 rotates the image data acquired by the DMAC 38 in accordance with the head that ejects ink and the nozzle position in the head. The DMAC 38 outputs the rotated image data to the IJ recording head control unit 44. For example, the rotator 39 can acquire the rotation angle θ calculated when the position calculation circuit 32 calculates the position, and can rotate the image using the rotation angle θ.

  The Image RAM 37 temporarily stores the image data 13 acquired by the DMAC 38. That is, a certain amount of image data 13 is buffered and read according to the position of the handheld printer 20.

  The IJ recording head control unit 44 performs dither processing on the image data (bitmap data) and converts the image data 13 into a set of points representing an image with size and density. As a result, the image data 13 becomes data of the ejection position and the dot size. The IJ recording head control unit 44 outputs a control signal corresponding to the dot size to the IJ recording head drive circuit 23. The IJ recording head drive circuit 23 generates a drive waveform (voltage) using the drive waveform data corresponding to the control signal as described above.

  The navigation sensor I / F 42 communicates with the navigation sensor 30, receives movement amounts ΔX ′ and ΔY ′ (which will be described later) as information from the navigation sensor 30, and stores the values in an internal register.

  The print / sensor timing generation unit 43 notifies the navigation sensor I / F 42 of the timing for reading the information of the navigation sensor 30 and notifies the IJ recording head control unit 44 of the drive timing. The IJ recording head control unit 44 determines whether or not the ejection nozzle is available, and determines that ink is ejected if there is a target ejection position where ink should be ejected, and that ejection is not performed if there is no target ejection position.

  The interrupt controller 41 detects that the navigation sensor I / F 42 has completed communication with the navigation sensor 30, and outputs an interrupt signal for notifying the SoC 50 of the completion. By this interruption, the CPU 31 acquires ΔX ′ and ΔY ′ stored in the internal register by the navigation sensor I / F 42. In addition, it has a status notification function for errors and the like.

<Functions of image data output device 11 and handheld printer>
Next, functions of the image data output unit 11 and the handheld printer 20 will be described with reference to FIG. FIG. 6 is an example of a functional block diagram of the image data output device 11 and the handheld printer 20.

<< Image data output device >>
The image data output device 11 includes a transmission / reception unit 51, an imaging unit 52, a mark detection unit 53, a pattern matching unit 54, a position determination unit 55, and a storage / readout unit 59. Each of these units included in the image data output unit 11 operates according to an instruction from the CPU 106 according to the output unit program 1001 expanded from the SSD 108 onto the RAM 104. A function realized by or a means to be functioned.

  Further, the image data output unit 11 has a storage unit 1000 constructed by the SSD 106, the RAM 104, or the recording medium 103a shown in FIG. The storage unit 1000 stores an output device program 1001, imaging data 14, and image data 13. The output device program 1001 may be distributed in a state stored in the recording medium 103a, or may be distributed from a server device that distributes the program.

  The transmission / reception unit 51 is realized by a command from the CPU 106 illustrated in FIG. 4 and a communication I / F 107, and transmits / receives various data to / from the handheld printer 20.

  The imaging unit 52 is realized by the command from the CPU 106 illustrated in FIG. 4 and the camera 110, and images the print medium 12 and the handheld printer 20. Note that imaging may be triggered by a user operation, or triggered by a condition described later in the third embodiment.

  The mark detection unit 53 is realized by a command or the like from the CPU 106 shown in FIG. 4, detects the position detection marks M1 and M2 from the imaging data 14, and detects the position detection marks M1 and M1 with respect to the matching image 14a in the imaging data 14. The relative position of M2 is specified.

  The pattern matching unit 54 is realized by a command or the like from the CPU 106 shown in FIG. 4, performs pattern matching on the image data 13 using the matching image 14 a, and generates a matching image 13 a that matches the matching image 14 a. To detect.

  The position determination unit 55 determines the positions of the position detection marks M1 and M2 with respect to the coincidence image 13a of the image data 13 based on the relative positions of the position detection marks M1 and M2 with respect to the verification image 14a in the imaging data 14. This is the current position of the handheld printer 20 in the image data 13 (position at the time of imaging). The position of the handheld printer 20 is transmitted to the handheld printer 20 via the transmission / reception unit 51.

  The storage / reading unit 59 is realized by the instruction from the CPU 106 shown in FIG. 4 and the SSD 108, and stores various data in the storage unit 1000 and reads various data stored in the storage unit 1000. Process.

<< Handheld printer >>
The handheld printer 20 includes a transmission / reception unit 65, a position loss detection unit 66, a position setting unit 67, and a storage / reading unit 69. Each of these units included in the handheld printer 20 is operated by a command from the CPU 31 according to the printer program 2001 developed from the ROM 28 onto the DRAM 29 by any one of the components shown in FIGS. A function realized by or a means to be functioned.

  The handheld printer 20 has a storage unit 2000 constructed by the ROM 28 or DRAM 29 shown in FIG. The storage unit 2000 stores a printer program 2001.

  The transmission / reception unit 65 is realized by a command from the CPU 31 illustrated in FIG. 5, a communication I / F 27, and the like, and transmits / receives various data to / from the image data output unit 11.

  The position loss detection unit 66 is realized by a command from the CPU 31 shown in FIG. 5, a navigation sensor I / F 42, a position calculation circuit 32, and the like. Detect missing. The navigation sensor 30 collects the reflected light reflected from the print medium 12 by the light from the light source with a lens and takes an image with an image array. When the handheld printer 20 floats from the print medium 12, the distance between the print medium 12 and the image array becomes large, and the reflected light cannot be detected. As a result, the position loss detection unit 66 detects that the navigation sensor 30 has floated or that the current position has been lost. Further, for example, it may be detected that the current position is lost by continuously detecting the maximum speed that can be calculated by the position calculating circuit 32.

  The position setting unit 67 is realized by the command from the CPU 31 shown in FIG. 5 and the position calculation circuit 32, and sets the position of the handheld printer 20 transmitted from the image data output unit 11 in the position calculation circuit 32. .

  The storage / reading unit 69 is realized by the instruction from the CPU 31 shown in FIG. 5, the ROM 28, the DRAM 29, and the like, and stores various data in the storage unit 2000 and reads out various data stored in the storage unit 2000. Perform the process.

<Regarding the nozzle position in the IJ recording head>
Next, the nozzle position and the like in the IJ recording head 24 will be described with reference to FIG. FIG. 7A is an example of a plan view of the handheld printer 20. FIG. 7B is an example for explaining only the IJ recording head 24. The surface shown is the surface facing the print medium 12.

  The handheld printer 20 of the present embodiment has two or more navigation sensors 30. Since there are two or more navigation sensors 30, the rotation angle θ can be detected even if the navigation sensor 30 rotates relative to the print medium 12 during image formation. In FIG. 7A, two navigation sensors S0 and S1 are arranged apart from each other in the arrangement direction of the nozzles 61. The length between the two navigation sensors S0 and S1 is a distance L. The longer the distance L, the better. This is because the minimum rotation angle θ that can be detected becomes smaller as the distance L becomes longer, and the position error of the handheld printer 20 becomes smaller.

  The distances from the navigation sensor 30 to the IJ recording head 24 are distances a and b, respectively. The distance a and the distance b may be equal. Further, as shown in FIG. 7B, the distance from the end of the IJ recording head 24 to the first nozzle 61 is a distance d, and the distance between adjacent nozzles is a distance e. The values a to e are stored in advance in the ROM 28 or the like.

  Therefore, if the position calculation circuit 32 or the like calculates the position of the navigation sensor 30, the position calculation circuit 32 can calculate the position of the nozzle 61 using the distance a, the distance b, the distance d, and the distance e.

  In this embodiment, the horizontal direction to the print medium 12 is set as the X axis, and the vertical direction is set as the Y axis. These coordinates will be referred to as print medium coordinates. On the other hand, the navigation sensor 30 outputs a position on the following coordinate axes (X ′ axis, Y ′ axis). That is, the direction in which the nozzles 61 are arranged (the direction connecting the two navigation sensors S0 and S1) is the Y ′ axis, and the direction orthogonal to the Y ′ axis is the X ′ axis. The position is calculated by the position calculation circuit 32.

<About the position of the handheld printer on the print medium>
Next, the position of the handheld printer 20 will be described with reference to FIG. In FIG. 8A, the handheld printer 20 is rotated θ with respect to the print medium 12 clockwise. If there is no rotation, X = X ′ and Y = Y ′. However, when the handheld printer 20 rotates by the rotation angle θ with respect to the print medium 12, the outputs of the navigation sensors S0 and S1 and the actual position of the handheld printer 20 on the print medium 12 do not match.

  For this reason, as shown in FIG. 8A, ΔX ′ and ΔY ′ output from the navigation sensors S0 and S1 correspond to the print medium coordinates X and Y as follows. In FIG. 8A, the movement amounts ΔX ′ and ΔY detected by the navigation sensors S0 and S1 (the outputs are equal because of parallel movement) when the handheld printer 20 with the rotation angle θ moves in the X direction with the same rotation angle θ. The correspondence between 'and X, Y is shown. ΔX ′ output from the navigation sensors S0 and S1 is reflected in X1, and ΔY ′ is reflected in X2.

  In FIG. 8B, the movement amounts ΔX ′ and ΔY detected by the navigation sensors S0 and S1 (the outputs are equal because of parallel movement) when the handheld printer 20 with the rotation angle θ moves in the Y direction with the same rotation angle θ. The correspondence between 'and X, Y is shown. ΔY ′ output from the navigation sensors S0 and S1 is reflected in Y1, and −ΔX ′ is reflected in Y2.

Therefore, when the handheld printer 20 moves in the X direction and the Y direction with the rotation angle θ, ΔX ′ and ΔY ′ output from the navigation sensors S0 and S1 can be converted into X and Y of the print medium coordinates as follows.
X = ΔX′cos θ + ΔY′sin θ (1)
Y = −ΔX′sin θ + ΔY′cos θ (2)
As described above, if the rotation angle θ is known based on the starting position of image formation, the positions of the print medium coordinates of the navigation sensors S0 and S1 can be obtained from the equations (1) and (2). The image formation start position is a position at which the user starts image formation by pressing a button or the like of the handheld printer 20, for example, slightly inside the upper left corner of the print medium 12.

The rotation angle dθ during the sampling time of ΔX ′ and ΔY ′ can be obtained as follows from the difference between the outputs (ΔX ′ ₀ , ΔX ′ ₁ ) of the two navigation sensors S0 and S1. By accumulating dθ, the rotation angle θ can be obtained.
dθ = arcsin {(ΔX ′ ₀ −ΔX ′ ₁ ) / L} (3)
<Position detection mark>
The position detection marks M1 and M2 will be described with reference to FIG. FIG. 9 shows an arrangement example of the position detection marks M1 and M2 in the top view of the handheld printer 20. If the positions of at least two points of the handheld printer 20 are known, the position and orientation of the handheld printer 20 can be estimated. For this reason, the position detection marks M1 and M2 are arranged at two locations so that the positions of two points can be specified.

  The position detection marks M1, M2 may be formed integrally with the HHP 20, or may be affixed like a seal. Moreover, when OPU26 is a liquid crystal display etc., OPU26 may display.

  In FIG. 9A, position detection marks M1 and M2 are arranged at the upper right corner and the lower left corner of the handheld printer 20, respectively. Since both the position detection marks M1 and M2 indicate the coordinates of one point of the handheld printer 20, for example, the position of the upper left vertex P1 of the position detection mark M1 is detected from the imaging data 14, and the upper vertex P2 of the position detection mark M2 is detected. The position of is detected. Vertices other than these may be detected.

  9A, the position detection marks M1 and M2 have different directions (the position detection mark M2 has a rhombus shape), but the position detection marks M1 and M2 as shown in FIG. 9B. The direction may be the same. Further, as shown in FIG. 9C, position detection marks M <b> 1 and M <b> 2 may be arranged at the upper right corner and the upper left corner of the handheld printer 20, respectively.

  In FIG. 9D, only one position detection mark M1 is arranged in the upper right corner of the handheld printer 20. The position detection mark M1 has a rectangular solid portion at the center and a square surrounding it. In such a position detection mark M1, the ratio of the lengths of the black and white pixels through which a straight line passes is different in the vertical direction and the horizontal direction. For example, 1 (black) in the vertical direction: 1 (white): 3 (black): 1 (white): 1 (black), 1 (black) in the horizontal direction: 1 (white): 7 (black): 1 ( White): 1 (black). When such a predetermined ratio of black and white pixels is detected by two straight lines that are 90 degrees apart, it may be determined that the position detection mark M1 has been detected, and therefore the detection accuracy of the position detection mark M1 Can be improved. For this reason, even when only one position detection mark M1 is arranged, for example, the upper left vertex P3 and the lower right vertex P4 of the position detection mark M1 can be adopted as the two positions of the handheld printer 20, respectively.

  Similarly, even if only one of the rectangular position detection marks M1 and M2 in FIGS. 9A to 9C is arranged, the orientation of the position detection mark M1 (or M2) such as a rectangle is known. If it is a shape, the positions of two or more vertices of the position detection mark M1 (or M2) may be adopted as the coordinates of the two points of the handheld printer 20.

  Further, the position detection marks M1 and M2 may be triangular or star-shaped. When two position detection marks M1 and M2 are arranged, the position detection marks M1 and M2 may be circular.

Next, a method for detecting the position detection marks M1 and M2 will be described. The position detection marks M1 and M2 are formed in a color different from the color of the housing of the handheld printer 20. Therefore, the mark detection unit 53 can detect the position detection marks M1 and M2 from the imaging data 14 by the following procedure, for example. Hereinafter, a method for detecting the position detection marks M1 and M2 in FIGS. 9A to 9C will be described.
(i) Binarize the image to white or black.
(ii) Find consecutive black pixels.
(iii) Detect edges in the vertical and horizontal directions.
(iv) A straight line is detected by the Hough transform, and a black pixel surrounded by four straight lines is detected.
(v) When two black pixels of (iv) are found (when more than two black pixels are found, they are determined by the aspect ratio of four straight lines surrounding the rectangle), and the position detection marks M1 and M2 are determined.

  Note that such a detection method of the position detection mark is merely an example. For example, the vertex of the position detection marks M1 and M2 may be detected by using the Harris corner detection method in (iv). Further, for example, an identification device that has learned the position detection marks M1 and M2 by machine learning may detect the position detection marks.

  FIG. 10 is an example of a diagram illustrating the relative positions of the position detection marks M1 and M2 and the navigation sensor 30. FIG. As shown in FIG. 10, the distance α and the distance β from the upper left vertex P1 of the position detection mark M1 to the navigation sensor S0 are known. Therefore, if the positions of the position detection marks M1, M2 are detected, the position of the navigation sensor S0 can be specified.

  It is also possible to obtain the position of the navigation sensor S0 from the position detection mark M2. Since the relative position of S1 with respect to the navigation sensor S0 is also known, if the navigation sensor S0 or S1 is known, the position of the navigation sensor S1 or S0 can also be specified.

<Acquisition of verification image from imaging data>
Next, the acquisition of the verification image from the imaging data will be described. First, the image data 13 included in the image data output unit 11 will be described with reference to FIG. FIG. 11A shows an example of the image data 13 together with the coordinates. The image data output device transmits the image data 13 of FIG. 11A to the handheld printer 20. Further, since the image data output unit 11 holds the entire image data 13, the coordinates of each pixel of the image data 13 can also be specified. For convenience of explanation, the upper left corner of the image data 13 is the origin (0, 0), but the origin may be anywhere.

  If the handheld printer 20 forms an image halfway, but the user loses sight of the current position by the end of the image formation, the user images the handheld printer 20 and the print medium 12. FIG. 11B is a diagram illustrating the posture of the image data output unit 11 at the time of imaging. As shown in FIG. 11 (b), the optical axis 7 of the image data output device 11 is preferably perpendicular to the print medium 12. Thereby, since distortion (projective deformation) caused by the inclination of the optical axis 7 does not occur in the imaging data 14, comparison between the imaging data 14 and the image data 13 is facilitated.

<< Scaling of imaging data >>
Here, the size of the imaging data 14 may vary depending on how far the image data output unit 11 has imaged the print medium 12 and the focal length (magnification). On the other hand, the size when the image data 13 is formed on the print medium 12 is determined (for example, a size that fits in A4). Therefore, the pattern matching unit 54 captures the imaging data 14 so that the distance between the position detection marks M1 and M2 (the length of the side when only one position detection mark is present) is the actual length at the time of image formation. Scaling For example, it is assumed that one pixel (including a pitch interval between pixels) corresponds to the length of s [mm] in real space. Further, it is assumed that the actual length of the position detection marks M1 and M2 is 50 mm, and the distance between the position detection marks M1 and M2 in the imaging data 14 is 300 pixels. In this case, since the distance between the position detection marks M1 and M2 at the time of image formation is a length of “s × 300”, the imaging data 14 is scaled so that this is 50 [mm]. Therefore, by referring to a table or the like in which the value of s is predetermined for each focal length, the pattern matching unit 54 can change the magnification so that the imaging data 14 has the same size as the image data 13. Note that the image data 13 may have the same size as the image data 13 by scaling the image data 13.

  The pattern matching unit 54 may calculate the distance between the image data output unit 11 and the print medium 12 from the number of pixels of the position detection marks M1 and M2. Then, the imaging data 14 may be scaled according to this distance. The relationship between the distance and the variable magnification is set in advance so that the sizes of the imaging data 14 and the image data 13 match.

<< Acquisition image and pattern matching >>
FIG. 11C shows an example of the imaging data 14 captured by the image data output unit 11. In this image data 14, a formed image 71 and a handheld printer 20 are imaged on a print medium. The relative positions of the formed image 71 and the handheld printer 20 are clear. Therefore, if the formed image 71 can be detected from the image data 13, the position of the handheld printer 20 in the image data 13 can be determined using this relative position.

  The pattern matching unit 54 extracts a part or the whole of the formed image 71 around or near the handheld printer 20 as a verification image 14a. First, the verification image 14a including a character string will be described. The use of the character string is for convenience of explanation, and the verification image 14a may be an image.

  The pattern matching unit 54 extracts a character string by image processing. For example, the character region is extracted as a region whose spatial frequency is obtained and whose height is equal to or higher than a threshold value. Further, a circumscribed rectangle of continuous black pixels is obtained, and it can be determined that the character string is a circumscribed rectangle having a substantially constant height arranged in one direction at substantially constant intervals. In this embodiment, it is assumed that the verification image 14a including the characters “HAND HELD PRIN” is extracted.

  FIG. 12 is an example of a diagram for schematically explaining pattern matching using the matching image 14 a for the image data 13. FIG. 12A shows a collation image 14 a and image data 13. For example, the pattern matching unit 54 calculates the difference for each pixel (or for each pixel block) by aligning the upper left corner of the verification image 14a with the origin (0, 0). If both the image data 13 and the verification image 14a are black pixels or white pixels, the difference between the pixels is zero. On the other hand, if only one of the image data 13 and the verification image 14a is a black pixel or a white pixel, the absolute value of the difference between the pixels is 1. Therefore, the smaller the total difference for the area indicated by the verification image 14a is, the more the verification image 14a matches the image data 13.

  As shown in FIG. 12B, the pattern matching unit 54 calculates the sum of the differences while shifting the image for comparison 14a pixel by pixel (or pixel blocks) with respect to the image data 13. When the sum of the differences is the smallest, the area overlapping the verification image 14a is the matching image 13a.

  In FIG. 12C, the verification image 14 a matches the image data 13. In this way, the coincidence image 13a is specified from the image data 13.

  Note that pattern matching can be similarly performed for an image (image) instead of a character. That is, pattern matching may be performed using an arbitrary rectangular area around or near the handheld printer 20 as the matching image 14a.

  However, the time required for detecting the coincidence image 13a can be shortened by specifying whether the collation image 14a is a character or an image (image). That is, pattern matching may be performed only in the character area or the image area in the image data 13. The image area may be determined as an area that is not a character area, or can be detected by clustering pixels into a color space.

  When pattern matching is performed using characters, the pattern matching unit 54 may perform pattern matching on a character basis, instead of pattern matching on a pixel basis. For example, the pattern matching unit 54 performs OCR (Optical Character Reader) on the verification image 14a. As a result, the character code included in the verification image 14a is known, and the pattern matching unit 54 performs OCR on the image data 13 to search for a character string in which the same character code is arranged. Thereby, the location of the coincidence image 13a of the image data 13 is known.

<< When matching image 13a cannot be determined uniquely >>
Here, consider a case where the coincidence image 13a cannot be determined uniquely. If the image data 13 already formed is extremely small (for example, only 2 to 3 characters are formed and there are a plurality of the same characters in the image data), or the image data 13 has a plurality of the same character patterns, the matching image 13a cannot be specified. In the former case, the pattern matching unit 54 determines that the character is at a position close to the image formation start position. In the latter case, the pattern matching unit 54 displays a plurality of character patterns on the display device 102 and asks the user to select them.

  FIG. 13 is a diagram showing an example of an inquiry screen 1300 displayed on the display device 102. The inquiry screen 1300 displays image data 13 and imaging data 14. The pattern matching unit 54 selects the matching image 14 a from the imaging data 14 and detects four matching images 13 b 1 to 13 b 4 from the image data 13. The pattern matching unit 54 emphasizes the four matching images 13b1 to 13b4 by surrounding them with a rectangle. Further, a message 1301 such as “Please select a character string corresponding to the character string enclosed in the right square” is displayed on the inquiry screen 1300. The user determines which of the matching images 13b1 to 13b4 corresponds to the matching image 14a of the imaging data 14, and selects (touches) one of the four matching images 13b1 to 13b4.

<< Another example of pattern matching >>
Another method of pattern matching will be described with reference to FIG. FIG. 14 is a diagram illustrating an example of pattern matching focusing on the length of the side of the verification image 14a.

  This method is a method of detecting a straight line (or each vertex) connecting a short side (a-b), a long side (b-c), and a diagonal line (a-c) of a matching image 14a including characters (here, "HAND"). Compared to pattern matching at the pixel level, the processing load can be reduced.

  The pattern matching unit 54 takes out a collection of characters delimited by a certain margin or more from the imaging data 14. This aggregate is a collation image 14a. In the case of English, it may be cut out for each word. In languages such as Japanese where there are no spaces between words, characters between punctuation marks are cut out. This is because the character string is taken out by eliminating the space. Further, by focusing on words and punctuation marks, a set of characters can be extracted from the image data 13 with the same delimiter. This aggregate becomes a candidate for the coincidence image 13b. It does not include character strings that extend over two lines due to line breaks.

  Depending on the image data 13, there are cases where there is little space between lines, etc., so the division of character chunks is generally not decided, but in order to treat them as chunks (aggregates), a certain threshold is set to detect the chunks. That's fine.

  In the example of FIG. 14A, a triangle is formed by the vertices abc of the verification image 14a including a collection of characters “HAND”. In FIG. 14B, among the candidate rectangular areas of the coincidence image 13a including the character set “HAND”, a triangle is formed by the vertices a ′, b ′, and c ′.

  Therefore, if a triangle having the same side as the triangle abc of the matching image 14a is found from the candidates for the matching image 13a, it is highly likely that it is the matching image 13a.

  In the present embodiment, it is assumed that the imaging data 14 is not rotated with respect to the image data 13. Even if the imaging data 14 is rotated, the imaging unit 52 pays attention to a character string or the like and corrects the tilt, whereby the image data 13 and Differences in the angle of the imaging data 14 need not be considered.

  However, as shown in FIG. 14C, when the imaging data 14 is rotated with respect to the image data 13, the pattern matching unit 54 may confirm by rotating the verification image 14 a. First, the coordinates of point a and point a ′ are matched, and rotation is performed so that the coordinates of point c and point c ′ match. Instead, the points b and b ′ may be used, but the calculation accuracy can be improved as the distance increases.

  Here, the length of the distance is caused by the accuracy, which is related to the number of bits in the program. In terms of computation, the value handled is a finite number of bits (for example, the C language double type is 16 bits). Therefore, an operation deviation of the minimum unit occurs. Similarly, when the image data 13 is coordinated, it is affected by the finite number of bits. From the above, it can be seen that the minimum unit that can be shifted is the same, but when the distance to be expressed is different, the influence of the minimum unit error is reduced when the distance is relatively long.

<Position of handheld printer in image data>
FIG. 15 is an example of a diagram illustrating the position of the handheld printer 20 in the image data 13. FIG. 15A shows the relative position of the handheld printer 20 in the imaging data 14. The position detection marks M1 and M2 are detected from the imaging data 14, and the relative position with the verification image 14a is also clarified. For example, the position of the position detection marks M1, M2 is represented by the distance from the upper left vertex P5 of the verification image 14a and the azimuth φ.
Position detection mark M1: distance k1, direction φ1
Position detection mark M2: distance k2, azimuth φ2
FIG. 15B shows the positions of the position detection marks M1 and M2 estimated as the coincidence image 13a in the image data 13. In the image data 13, the position detection mark M1 is at a distance k1 and an azimuth φ1 with respect to the upper left vertex P6 of the coincidence image 13a, and the position detection mark M2 is at a distance k2 and an azimuth φ2 with respect to the upper left vertex P6 of the coincidence image 13a. Therefore, the positions (coordinates) of the position detection marks M1 and M2 in the image data 13 can be specified using the upper left vertex P6 of the coincidence image 13 detected by pattern matching as a base point.

  The position determination unit 55 determines the coordinates (M0x, M0y) of the point P1 at the distance k1 and the azimuth φ1 from the upper left vertex P6 of the coincidence image 13 as the coordinates of the position detection mark M1. Similarly, the coordinates (M1x, M1y) of the point P2 at the distance k2 and the azimuth φ2 from the upper left vertex of the coincidence image are determined as the coordinates of the position detection mark M2. The position determination unit 55 transmits (M0x, M0y) (M1x, M1y) determined in this way, or the positions of the navigation sensors S0, S1 to the handheld printer 20.

<Operation procedure>
FIG. 16 is an example of a flowchart illustrating an operation procedure of the image forming system 1 of the present embodiment.

  First, the user presses the power button of the handheld printer 20 (U01).

  The handheld printer 20 is activated when the power is turned on, and initializes the SoC 50 and the ASIC / FPGA 40 (H01).

  Thereafter, when the initialization is completed (Yes in H02), the handheld printer 20 turns on an LED indicating that image formation can be started (H03).

  Next, the user operates the image data output unit 11 to select image data 13 to be used for printing (U02).

  Further, the user operates the image data output unit 11 to execute a print job (U03).

  On the other hand, the storage / reading unit 59 of the image data output device 11 stores the image data 13 in the storage unit 1000 (D01).

  The pattern matching unit 54 of the image data output device 11 holds the coordinates of the image data 13 (D02). That is, the coordinates of each pixel when the image data 13 is formed on the print medium 12 are obtained. Note that the transmission / reception unit 51 of the image data output device 11 transmits the image data 13 to the handheld printer 20.

  Next, the handheld printer 20 blinks the LED when it receives the image data 13 (H04). Thereby, the user grasps that the image can be formed.

  The user determines the initial position of the handheld printer 20 (U04). Then, the print start determination button is pressed (U05), and the operation is performed freehand (U06).

  The handheld printer 20 forms an image by ejecting ink according to the current position (H05).

  Then, it is assumed that the user has lifted the handheld printer 20 during scanning (U07).

  As a result, the position loss detection unit 66 of the handheld printer 20 acquires the position information and the floating information and stores them in the internal memory (H06).

  Thereafter, the user performs an operation necessary for acquiring the position. That is, the already formed image data 13 and the handheld printer 20 are imaged into one imaging data 14 (U08).

  The mark detection unit 53 of the image data output unit 11 detects the position detection marks M1 and M2 from the imaging data 14 (D03).

  Then, the pattern matching unit 54 acquires the matching image 14a from around or near the position detection mark, and performs pattern matching on the image data 13 (D04). The verification image 14a may be a character or an image.

  Next, the position determination unit 55 determines the relative positions of the position detection marks M1, M2 with respect to the coincidence image 13a that coincides with the matching image 14a in the image data 13 (D05).

  Then, the positions of the navigation sensors S0, S1 or the positions of the position detection marks M1, M2 are transmitted to the handheld printer 20 (D06).

  The position setting unit 67 of the handheld printer 20 sets the positions of the navigation sensors S0 and S1 in the position calculation circuit 32 (H07). Thereby, the position calculation circuit 32 can calculate the position of the nozzle. The handheld printer 20 notifies the user that the image formation can be resumed by turning on the LED. Therefore, the user can resume image formation.

  As described above, in the image forming system 1 of this embodiment, even if the handheld printer 20 loses sight, the image data output unit 11 can calculate the position and resume image formation. There is no need to install a unit that ejects invisible ink or a scanner unit.

  In Example 1, the image data output device 11 captured the formed image and the handheld printer 20 from the front of the print medium 12. In this embodiment, an image data output unit 11 that can calculate the position of the handheld printer 20 even when the optical axis at the time of imaging is inclined with respect to the print medium 12 will be described.

  FIG. 17 shows an example of a functional block diagram of the image data output device 11 of the present embodiment. In the present embodiment, the components denoted by the same reference numerals in FIG. 6 perform the same function, and therefore, only the main components of the present embodiment may be mainly described. The image data output device 11 of this embodiment has a matrix calculation unit 56.

  The matrix calculation unit 56 is realized by an instruction from the CPU 106 shown in FIG. 4 and calculates a projective transformation matrix to be described later.

<Projection transformation matrix>
The projective transformation matrix will be described. Consider that the print medium and the imaging data 14 are distorted when the optical axis of the image data output unit 11 is tilted. FIG. 18 is an example of a diagram illustrating the distortion of the imaging data 14. 18A shows an example of the image data 13 held by the image data output unit 11, and FIG. 18B shows an example of the imaging data 14 captured by the image data output unit 11.

Since the imaging data 14 is formed based on the image data 13, the imaging data 14 has points corresponding to the image data 13. If the coordinates of the image data 13 are m _p (x _p , y _p ) and the coordinates of the imaging data 14 are m _c (x _c , y _c ), m _p and m _c are the projective transformation matrix H (an example of correction information). Can be expressed as follows. mp (xp, yp) is a coordinate system coordinate having the upper left vertex of the image data 13 as the origin, and _mc (x _c , y _c ) is a coordinate system coordinate having the upper left vertex of the imaging data 14 as the origin. .

The following constraint is obtained from one corresponding point of m _p (x _p , y _p ) and m _c (x _c , y _c ).

With one constraint, two constraint equations are obtained for eight unknown parameters. Therefore, if four corresponding points are obtained from the image data 13 and the imaging data 14, all elements of the projective transformation matrix H can be determined.

  In FIG. 18, the correspondence between the image data 13 and the imaging data 14 is not necessarily clear. For this reason, for example, the image data output unit 11 displays the image data 13 and the imaging data 14 on the display device 102 and allows the user to input corresponding points. In this case, for example, the user designates the same location (for example, the upper left vertex of H, the upper right vertex of N, the lower right vertex of N, the lower left vertex of H, etc.) of the corresponding characters in the image data 13 and the imaging data 14, respectively.

  Alternatively, the image data output unit 11 acquires the image-formed pixels in the image data 13 from the handheld printer 20 and extracts corresponding points as follows. As described above, since the handheld printer 20 records the pixels from which the ink has been ejected, the pixels on which the image is formed are obvious.

In the example of FIG. 18, the characters “HAND HELD PRIN” have been formed in the image data 13. The matrix calculation unit 56 employs the upper left vertex, upper right vertex, lower right vertex, and lower left vertex of the formed area as corresponding points. Therefore, the matrix calculation unit 56 sets the upper left vertex m _p1 , N upper right vertex m _p2 , N lower right vertex m _p3 , and H lower left vertex m _p4 of the image data 13 as corresponding points. Therefore, the matrix calculation unit 56 acquires that the image “HAND HELD PRIN” has been formed from the handheld printer 20 as coordinate data, and determines four vertices m _p1 , m _p2 , m _p3 , and m _p4 from the coordinate data. (The coordinates of the points m _{p1 to} m _p4 of the handheld printer 20 are the same as the coordinates of the points m _{c1 to} m _c4 of the image data 13).

In the imaging data 14, the points m _c1 , m _c2 , m _c3 , and m _c4 corresponding to the points m _p1 , m _p2 , m _p3 , and m _p4 are obtained as follows. That is, the point m _c1 is obtained by searching the upper left pixel, the point m _c2 is the upper right pixel, the point m _c3 is the lower right pixel, and the point m _c4 is the lower left pixel. Note that the imaging data 14 is preferably smoothed by a filter or the like in order to reduce the influence of noise such as a single point.

  In the above description, the case where the image for matching 14a is a character string has been described. Even when an image (image) is formed, the upper left vertex, the upper right vertex, the upper right vertex, and the lower left vertex of the image (image) can be specified in the same manner. Corresponding points can be obtained. In addition, since H can be calculated | required more accurately, the one where the area enclosed by a corresponding point is wider can determine | require the corresponding point of the wide range as much as possible.

<Acquisition of verification image>
As in the first embodiment, the mark detection unit 53 can detect the position detection marks M1 and M2. The pattern matching unit 54 determines an image in a predetermined range around or near the position detection marks M1 and M2 as the matching image 14a.

<Pattern matching>
Since the pattern matching is preferably performed on the matching image 14a without distortion, the pattern matching unit 54 corrects the distortion by applying all of the pixels of the matching image 14a to Equation (4) (imaged from the front). To state). FIG. 19A shows a collation image 14a before conversion, and FIG. 19B shows a collation image 14a after conversion. The distortion of the verification image 14a is reduced by the conversion. Therefore, the pattern matching unit 54 can perform pattern matching as in the first embodiment. In FIG. 19, the range of the image data 13 for which the projection transformation matrix H is obtained and the collation image 14a are the same, but they may be different.

  In the coordinate system of the verification image 14a, the pattern matching unit 54 converts the coordinates of the position detection marks M1 and M2 by the equation (4). Thereby, it is possible to calculate how much the distances and azimuths of the position detection marks M1 and M2 with respect to the verification image 14a become when they are captured from the front (that is, in the image data 13). Accordingly, as shown in FIG. 19B, the positions of the position detection marks M1 and M2 whose distortion is corrected with respect to the matching image 14a whose distortion is corrected are known.

<Position of handheld printer in image data>
It is assumed that the pattern matching unit 54 detects the coincidence image 13a in the image data 13. The position determination unit 55 applies the distances k1 and k2 and the orientations φ1 and φ2 obtained as shown in FIG. 19B to the coincidence image 13a to determine the positions of the position detection marks M1 and M2 in the image data 13.

  Thus, according to the present embodiment, the image data output device 11 can calculate the position of the handheld printer 20 and resume image formation even if the image formed with the print medium 12 is not picked up from the front.

  In this embodiment, an image forming system 1 using a handheld printer 20 in which non-front marks are arranged so as to be imaged from the front will be described. The functional block diagram of the present embodiment may be the same as FIG. 6 of the first embodiment.

  The non-front marks Q1 to Q4 (an example of the second mark) arranged on the handheld printer 20 will be described with reference to FIG. FIG. 20 is a diagram illustrating an example of the handheld printer 20 and the non-frontal marks Q1 to Q4 arranged on the handheld printer 20. The position detection marks M1 and M2 are the same as those in the first and second embodiments. In contrast, in this embodiment, non-frontal marks Q1 to Q4 are arranged on the four side surfaces of the handheld printer 20. Since the non-front marks Q <b> 1 to Q <b> 4 have no thickness, they are not captured in the imaging data 14 when the optical axis of the image data output device 11 is perpendicular to the handheld printer 20. In other words, it means that the optical axis is not vertical while the non-front marks Q1 to Q4 are shown.

  The non-front mark is described as being the same color as the position detection marks M1 and M2, but may be a different color. In the case of different colors, only the non-front marks Q1 to Q4 can be detected. In the case of the same color, the non-front mark is specified by using the non-front marks Q1 to Q4 within a predetermined distance from the position detection marks M1 and M2.

  FIG. 21 is an example of a diagram for schematically explaining imaging of the handheld printer 20 by the image data output device 11, and FIG. 22 is an example of a flowchart showing an imaging procedure of the image data output device 11.

  The user keeps the angle of view while adjusting the angle so that the image data output unit 11 does not copy the non-frontal marks Q1 to Q4. During this time, the imaging unit 52 of the image data output device 11 regularly captures images (S10).

  The imaging unit 52 of the image data output device 11 detects the non-frontal marks Q1 to Q4 similarly to the position detection marks M1 and M2 (S20).

  The imaging unit 52 determines whether only two marks are captured (S30). Since the position detection marks M1 and M2 are arranged in the front, they are always imaged. Therefore, when only two marks are captured, it can be determined that the optical axis is the front of the print medium 12.

  When only two marks are captured (when the non-front marks Q1 to Q4 cannot be detected), the imaging unit 52 acquires the imaging data 14 (S40).

  Therefore, the position of the handheld printer 20 can be calculated by performing enlargement / reduction in the same manner as in the first embodiment using the imaging data 14 thus captured and pattern matching with the image data 13.

  Although the non-front marks Q1 to Q4 have been described as having no thickness, the non-front marks Q1 to Q4 may have a thickness. For example, assuming that the thicknesses of the plurality of non-front marks Q1 to Q4 are the same, it is possible to confirm that the optical axis is vertical when the thicknesses of the non-front marks Q1 to Q4 are the same in the imaging data 14. .

  In this embodiment, the imaging unit 52 automatically determines that the non-frontal marks Q1 to Q4 are not captured, but the user may determine and capture the image (the shutter button may be pressed). . However, the optical axis shift due to camera shake or the like can be suppressed by automatically capturing an image.

  In the present embodiment, in addition to the configuration of the third embodiment, an image forming system 1 that does not require the image data output device 11 to enlarge / reduce the imaging data 14 will be described. The functional block diagram of the present embodiment may be the same as FIG. 6 of the first embodiment.

  In the process in which the imaging unit 52 captures the handheld printer 20 as shown in FIG. 21, characters that have already been imaged are captured in the imaging data 14. Therefore, the imaging unit 52 acquires characters around or near the position detection marks M1 and M2 from the imaging data 14 and searches the image data 13 for the same characters. Here, pattern matching or OCR may be performed. For example, when the character string “HAND” is detected from the imaging data 14, the character string “HAND” is searched from the image data 13. If both sizes are the same, it is not necessary to enlarge or reduce the imaging data 14 after imaging.

  In the case of an image (image), a matching image 14a is acquired and a matching image is detected by pattern matching. And it images so that both size may become the same.

  FIG. 23 is an example of a flowchart when the image data output unit 11 captures an image. In the description of FIG. 23, differences from FIG. 22 will be mainly described.

  When only two marks are imaged in step S30, the imaging unit 52 determines whether the sizes of the imaging data 14 and the image data 13 are the same (S35). The imaging unit 52 outputs a voice such as “Please raise the camera” or “Lower the camera” so that the sizes of the imaging data 14 and the image data 13 are the same. It is preferable to induce.

  When the sizes become the same, the imaging unit 52 acquires the imaging data 14 (S40).

  Therefore, the image data output unit 11 of the present embodiment can omit enlargement / reduction in the process of calculating the position of the handheld printer 20. In addition, since a matching image that matches the matching image 14a is specified during imaging, the waiting time of the user can be shortened.

  In the present embodiment, the character string sizes are compared, but the distance between the position detection marks M1 and M2 may be compared between the imaging data 14 and the image data 13. That is, using the relationship between the number of pixels and the length of the image to be formed as in the first embodiment, the distance (pixel) between the position detection marks M1 and M2 in the imaging data 14 is a predetermined value (that is, the image When the number of pixels reaches 50 [mm] at the time of formation, the imaging data 14 is acquired.

<Other application examples>
The best mode for carrying out the present invention has been described above with reference to the embodiments. However, the present invention is not limited to these embodiments, and various modifications can be made without departing from the scope of the present invention. And substitutions can be added.

  For example, although the inquiry screen 1300 in FIG. 13 has been described as being displayed when a plurality of matching images 13a are detected, the inquiry screen 1300 may be displayed when the matching image 14a is detected. In this case, the user selects a matching image that matches the matching image 14a from the image data 13 in a state where there is no candidate. In addition, the user may specify the verification image 14 a from the imaging data 14.

  In the present embodiment, the position of the handheld printer 20 is detected by the position detection marks M1 and M2. However, the handheld printer 20 may be recognized from the imaging data by an identification device that learns the weight of the classifier using a photograph of the entire shape or pattern of the handheld printer 20 as a teacher signal. Also in this case, since the range in which the handheld printer 20 is shown is known, the position of the navigation sensor 30 can be specified.

  For example, the constituent elements of the SoC 50 and the ASIC / FPGA 40 described above may be included in any of them depending on the CPU performance, the circuit scale of the ASIC / FPGA 40, and the like.

  The functions of the handheld printer 20 described with reference to FIGS. 4 and 5 and the like may be realized by the CPU 31 executing a program.

  Further, in the present embodiment, a handheld printer 20 that detects positions in three directions (for example, X / Y / R axis, R: rotation) and can freely scan on a plane (free hand scanning) will be described as an example. did. However, the present embodiment can also be applied to the handheld printer 20 that detects a position in one direction (for example, the X-axis direction) and scans only in one direction to form an image. The present embodiment can also be applied to the handheld printer 20 that detects positions in two directions (for example, the X / Y axis directions) and scans only in two directions to form an image.

  In the present embodiment, it has been described that an image is formed by ejecting ink. However, an image may be formed by irradiation with visible light, ultraviolet light, infrared light, laser, or the like. In this case, for example, a print medium 12 that reacts to heat or light is used. Further, a transparent liquid may be discharged. In this case, visible information is obtained when light in a specific wavelength region is irradiated. That is, it is only necessary to form visual information that can be finally confirmed with the naked eye through chemical and physical actions.

  Further, in the present embodiment, it has been described that the user scans with a freehand when forming an image. However, the handheld printer 20 may run on a print medium by using a motor or the like as power.

DESCRIPTION OF SYMBOLS 1 Image formation system 11 Image data output device 12 Print medium 13 Image data 14 Imaging data 20 Handheld printer 52 Imaging part 53 Mark detection part 54 Pattern matching part 55 Position determination part 56 Matrix calculation part

Japanese Patent No. 5033247

Claims (14)

An information processing apparatus capable of communicating with an image forming apparatus that forms an image based on image data while moving a position on a recording medium,
An acquisition unit configured to acquire imaging data in which a formed image formed on a recording medium by the image forming apparatus and the image forming apparatus is captured;
An image processing unit for detecting the image forming apparatus and the formed image from the imaging data;
The position on the recording medium of the image forming apparatus is estimated using the image data used for forming the formed image and the relative position of the image forming apparatus with respect to the formed image detected by the image processing unit. Position estimation means for
Transmitting means for transmitting the position estimated by the position estimating means to the image forming apparatus;
An information processing apparatus.   The position estimating means specifies a matching image that matches the formed image in the image data used for forming the formed image, and estimates the position specified by the relative position based on the matching image. Item 4. The information processing apparatus according to Item 1.   The position estimation unit extracts the formed image around the image forming apparatus, and specifies the matching image by performing pattern matching on the image data using the surrounding formed image. The information processing apparatus described.   The said position estimation means performs pattern matching with respect to the said image data using the said formation image containing a character string, The said matching image containing the same character string as the said character string is specified from the said image data. Information processing device. The position estimating means connects a plurality of straight line lengths obtained by connecting vertices of circumscribed rectangles of character strings included in the formed image, and vertices of circumscribed rectangles of character strings included in the image data. And compare with a set of multiple straight line lengths
The information processing apparatus according to claim 3, wherein the matching image in which a plurality of sets of straight line lengths match is specified from the image data.   The information processing apparatus according to claim 2, wherein the position estimation unit displays the formed image and the image data on a display device and receives selection of the matching image corresponding to the formed image. The image processing means detects a first mark that can estimate the position and orientation of the image forming apparatus arranged in the image forming apparatus from the imaging data,
The information processing apparatus according to claim 1, wherein the position estimation unit determines a relative position of the first mark with respect to the formed image as the relative position of the image forming apparatus with respect to the formed image. The image forming apparatus and an imaging unit that captures the formed image formed by the image forming apparatus,
A second mark is arranged on the image forming apparatus,
The imaging unit is configured to form the image formed by the image forming apparatus and the image forming apparatus when the second mark is not captured in the captured image data or when the second mark is captured in a predetermined state. The information processing apparatus according to any one of claims 1 to 7, The imaging means is further triggered by the fact that the size of the formed image in the imaging data is substantially the same as the size of the coincidence image that matches the formed image in the image data.
The information processing apparatus according to claim 8, wherein the image forming apparatus and the formed image formed by the image forming apparatus are captured.   In the imaging data, the length information obtained from the first mark capable of estimating the position and orientation of the image forming apparatus arranged in the image forming apparatus is set to a predetermined number of pixels in the imaging data. The information processing apparatus according to claim 8, wherein the information forming apparatus captures the image forming apparatus and the formed image formed by the image forming apparatus. Correction information calculating means for calculating correction information for correcting distortion of the formed image that has occurred during imaging;
The position estimation unit performs pattern matching on the image data using the formed image whose distortion has been corrected using the correction information calculated by the correction information calculation unit, and identifies the matching image. 3. The information processing apparatus according to 3. An image forming system having an image forming apparatus that forms an image based on image data while moving a position on a recording medium, and an information processing apparatus that can communicate with the image forming apparatus,
An acquisition unit configured to acquire imaging data in which a formed image formed on a recording medium by the image forming apparatus and the image forming apparatus is captured;
An image processing unit for detecting the image forming apparatus and the formed image from the imaging data;
The position on the recording medium of the image forming apparatus is estimated using the image data used for forming the formed image and the relative position of the image forming apparatus with respect to the formed image detected by the image processing unit. Position estimation means for
Transmitting means for transmitting the position estimated by the position estimating means to the image forming apparatus;
An image forming system. An image forming apparatus that forms an image based on image data while moving the position of a recording medium,
Position detecting means for detecting a position relative to the recording medium;
Acquisition means for acquiring imaging data in which a formed image formed on the recording medium by the image forming apparatus and the image forming apparatus is captured;
An image processing unit for detecting the image forming apparatus and the image from the imaging data acquired by the acquisition unit;
The position on the recording medium of the image forming apparatus is estimated using the image data used for forming the formed image and the relative position of the image forming apparatus with respect to the formed image detected by the image processing unit. Position estimation means for
And an image forming unit that forms the formed image using the position estimated by the position estimating unit. An information processing apparatus capable of communicating with an image forming apparatus that forms an image based on image data while moving a position on a recording medium,
An acquisition unit configured to acquire imaging data in which a formed image formed on a recording medium by the image forming apparatus and the image forming apparatus is captured;
An image processing unit for detecting the image forming apparatus and the formed image from the imaging data;
The position on the recording medium of the image forming apparatus is estimated using the image data used for forming the formed image and the relative position of the image forming apparatus with respect to the formed image detected by the image processing unit. Position estimation means for
Transmitting means for transmitting the position estimated by the position estimating means to the image forming apparatus;
Program to function as.