JP6507865B2 - Information processing apparatus, image forming system, image forming apparatus - Google Patents

Description

  The present invention relates to an information processing apparatus, an image forming system, and an image forming apparatus.

  2. Description of the Related Art There is known a printer apparatus that discharges ink or the like at a timing when a sheet is conveyed and the sheet reaches a position where an image is formed, to form an image. On the other hand, a printer apparatus (hereinafter, referred to as a hand-held printer) which is miniaturized by removing the paper conveyance system from the printer apparatus is being put to practical use.

  FIG. 1 is an example of a diagram schematically showing image formation by the hand-held printer 20. As shown in FIG. Image data is transmitted to the handheld printer 20 from an image data output unit 11 such as a smartphone or a PC (Personal Computer), for example. The user holds the hand-held printer 20 and scans the print medium 12 (for example, fixed paper, notebook, etc.) freehandedly. Since the hand-held printer 20 has a position detection mechanism, when the hand-held printer 20 moves to the target ejection position, it ejects ink of the color to be ejected at the target ejection position. The location where ink has already been ejected is masked (as it is not the target of ink ejection), so the user can form an image by scanning the hand-held printer 20 in any direction on the print medium 12.

  However, since the scanning of the hand-held printer 20 is left to the user, the user may accidentally float the hand-held printer 20 off the print medium 12. In this case, the position detected by the handheld printer 20 may not be accurate. In addition, the user may scan the hand-held printer 20 at a speed faster than the specification, and also in this case, the position may not be accurate. In these cases, the handheld printer 20 loses its current position, making it difficult to resume image formation.

  To cope with such a disadvantage, a hand-held printer 20 has been devised which recognizes the current position again (see, for example, Patent Document 1). In Patent Document 1, a position information is periodically printed with invisible ink, and when the current position is lost, the scanner unit reads the position information, recognizes the current position, and resumes image formation. Is disclosed.

  However, the hand-held printer described in Patent Document 1 needs to be equipped with a unit that discharges invisible ink and a scanner unit, which causes a problem of cost increase. In addition, there is a possibility that the operability may be reduced because the housing becomes heavy and the housing becomes large.

  An object of the present invention is to provide an information processing apparatus capable of resuming image formation while suppressing an increase in cost and the like in view of the above problems.

In view of the above problems, the present invention is an information processing apparatus capable of communicating with an image forming apparatus that forms an image based on image data while moving a position on a recording medium, and the image forming apparatus and the image forming apparatus An acquisition means for acquiring imaging data in which a formation image formed on a recording medium is imaged, an image processing means for detecting the image forming apparatus and the formation image from the imaging data, and the method is used for forming the formation image In the image data, a matched image that matches the formed image is specified, the relative position of the image forming apparatus with respect to the formed image in the captured data is determined, and the relative position based on the matched image in the image data is determined a position estimation means for estimating a position of the image forming apparatus in the image data on the basis of, the position of the position estimating unit has estimated And a transmission means for transmitting to the serial image forming apparatus.

  It is possible to provide an information processing apparatus capable of resuming image formation while suppressing cost increase and the like.

It is an example of the figure showing typically the image formation by a hand-held printer. FIG. 6 is an example of a diagram for explaining an outline of a procedure for acquiring a current position at which a handheld printer has lost sight. FIG. 1 is an example of a system configuration diagram of an image forming system. It is an example of the hardware block diagram of an image data output device and a hand-held printer. It is an example of the figure explaining the composition of a control part. It is an example of a functional block diagram of an image data output device and a hand-held printer. FIG. 7 is an example of a diagram for explaining the nozzle position and the like in the IJ recording head. It is an example of a figure explaining the position of a handheld printer. FIG. 6 is a view showing an arrangement example of position detection marks in the top view of the hand-held printer. It is an example of the figure which shows the relative position of a position detection mark and a navigation sensor. FIG. 7 is an example of a diagram for explaining the position of a handheld printer in imaging data. FIG. 7 is an example of a diagram schematically illustrating pattern matching using an image for matching on image data. It is a figure which shows an example of the inquiry screen displayed on the display apparatus. It is a figure which shows an example of the pattern matching which paid its attention to the side length of the image for collation. FIG. 2 is an example of a diagram for explaining the position of a handheld printer in image data. FIG. 6 is an example of a flowchart showing an operation procedure of the image forming system. It is an example of the functional block diagram of an image data output device (Example 2). It is an example of a figure explaining distortion of image pick-up data. It is an example of the figure which shows the image for collation before and after conversion, respectively. It is a figure which shows an example of the non-front mark arrange | positioned with a hand-held printer. It is an example of the figure which illustrates imaging of a handheld printer by an image data output machine typically. It is an example of the flowchart figure which shows the imaging procedure of an image data output device (Example 3). It is an example of the flowchart figure which shows the imaging procedure of an image data output device (Example 4).

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<Summary of Acquisition of Position of Handheld Printer>
First, an outline of a procedure for acquiring the current position when the handheld printer 20 loses the current position will be described with reference to FIG. FIG. 2 is an example of a diagram for explaining the outline of the procedure for acquiring the current position at which the handheld printer 20 has lost sight.
(i) As shown in FIG. 2A, the user scans the hand-held printer 20 on a print medium (an example of a recording medium) to form an image. Then, it is assumed that the handheld printer 20 has lost the current position by the time all the images are formed.
(ii) The user grasps the state of the hand-held printer 20 by the LED lamp and the alarm sound. Then, the user places the hand-held printer 20 at an appropriate place on the print medium 12 and picks up the hand-held printer 20 and the print medium 12 with the image data output unit 11 (FIG. 2 (b)). Here, in the handheld printer 20, position detection marks M1 and M2 (an example of a first mark) are formed in advance.
(iii) The left view of FIG. 2C shows the captured data 14 of the hand-held printer 20 and the print medium 12 captured. The image data output unit 11 detects the position detection marks M1 and M2 of the handheld printer 20 from the imaging data 14. Further, the image data output unit 11 acquires an image around the hand-held printer 20 (hereinafter, referred to as a verification image 14 a) from the imaging data 14. Then, the image data output unit 11 searches the image data 13 used for image formation for a matching image 13a that matches the image for matching 14a (pattern matching is performed).
(iv) FIG. 2 (d) shows the image data 13. As shown in FIG. 2C, the relative positions of the position detection marks M1 and M2 with respect to the verification image 14a in the imaging data 14 are known, so that the position detection marks M1 and M2 with respect to the matching image 13a in the image data 13 as well. The relative position can be identified. That is, the position of the handheld printer 20 in the image data 13 can be specified.
(v) The image data output unit 11 transmits the position of the hand-held printer 20 (or the position of the position detection marks M1 and M2 itself) to the hand-held printer 20. As a result, as shown in FIG. 2E, the handheld printer 20 can acquire the current position and resume image formation.

  Therefore, even if the handheld printer 20 according to the present embodiment loses the position, it can acquire the position and resume the image formation while suppressing the cost increase and the like.

<About terms>
The terms used in the present embodiment will be described.
Image data is data used for image formation, and is held by an image data output unit.
The imaging data is data obtained by imaging the print medium 12 and the handheld printer 20.

  If the handheld printer 20 loses position, it means that the current position can not be calculated or it is extremely inaccurate. The factors include that the hand-held printer 20 floats from the sheet, and the user causes the hand-held printer 20 to scan faster than the specifications, but the factors are not limited in this embodiment.

  The print medium 12 is, for example, a fixed-size sheet or a notebook, but the invention is not limited thereto. A fixed object such as a desk may be used as the print medium 12 or a sheet-like member other than paper may be used as the print medium 12. Also, the print medium 12 may not necessarily be arranged horizontally.

<Configuration example>
FIG. 3 shows an example of a system configuration diagram of the image forming system 1. The image forming system 1 of FIG. 3A includes an image data output unit 11 and a hand-held printer 20 that can communicate with each other. In such a system configuration, the image data output unit 11 transmits the image data 13 to the hand-held printer 20, and when the hand-held printer 20 loses position, the image data output unit 11 picks up the hand-held printer 20 and the print medium 12 Do. The image data output unit 11 detects the position of the hand-held printer 20 from the imaging data 14 and transmits the position to the hand-held printer 20.

  Further, in the system configuration of FIG. 3A, the handheld printer 20 may detect the position of the handheld printer 20. In this case, the image data output unit 11 transmits the imaging data 14 to the hand-held printer 20. The hand-held printer 20 holds at least a part of the image data 13, but acquires the entire image data 13 from the image data output unit 11 if necessary. The handheld printer 20 detects the position of the handheld printer 20 in the same manner as the image data output unit 11.

  The image forming system 1 shown in FIG. 3B includes an image data output unit 11, a hand-held printer 20, and an imaging device 8. In such a system configuration, the imaging device 8 images the handheld printer 20 and the print medium 12 when the handheld printer 20 loses its position. Since the imaging device 8 transmits the imaging data 14 to the image data output unit 11, the image data output unit 11 detects the position of the handheld printer 20 from the imaging data 14 and transmits it to the handheld printer 20.

  Also in this case, the handheld printer 20 may detect the position of the handheld printer 20. In this case, the imaging device 8 transmits the imaging data 14 to the handheld printer 20 (may be transmitted via the image data output unit 11). The hand-held printer 20 detects the position of the hand-held printer 20 as in FIG. 3 (a).

  Also, the imaging device 8 may detect the position of the hand-held printer 20. In this case, the imaging device 8 acquires the image data 13 from the image data output unit 11 and detects the position of the hand-held printer 20. The imaging device 8 transmits the detected position to the hand-held printer 20.

  In the image forming system 1 according to the present embodiment, the position of the hand-held printer 20 can be detected as long as the apparatus acquires the image data 13 and the imaging data 14 as described above. Therefore, as shown in FIG. 3C, the server 9 may detect the position of the handheld printer 20 by the image data output unit 11 transmitting the image data 13 and the imaging data 14 to the server 9. The server 9 transmits the detected position to the image data output unit 11, and the image data output unit 11 transmits this position to the hand-held printer 20.

  In the following, for convenience of explanation, the image data output unit 11 is described as detecting the position of the handheld printer 20.

  The image data output unit 11 is an information processing apparatus. For example, a smartphone, a personal computer (PC), a tablet terminal, a personal digital assistant (PDA), a mobile phone, or a wearable PC may be used. Alternatively, MFP (Multifunctional Peripheral / Printer / Product), various cameras, electronic blackboards, game devices, car navigation terminals, projectors, etc. may be used.

<< Hardware Configuration >>
The hardware configuration of the image data output unit 11 and the hand-held printer 20 will be described with reference to FIG. The image data output unit 11 is realized by, for example, a hardware configuration as shown in FIG. The image data output unit 11 includes an input device 101, a display device 102, an external I / F 103, a camera 110, a RAM 104, a ROM 105, a CPU 106, a communication I / F 107, a solid state drive (SSD) 108, and a short distance wireless communication device 109. , And each is connected to one another by a bus B.

  The input device 101 is, for example, a touch panel, and is used to input each operation signal to the image data output unit 11. The input device 101 may be a keyboard, a mouse or the like. The display device 102 is, for example, an LCD (Liquid Crystal Display) or the like, and displays the processing result by the image data output unit 11.

  The external I / F 103 is an interface with an external device. The external device is, for example, a recording medium 103a. The recording medium 103a can store a program for realizing the function of the present embodiment (a program for an output device described later). The image data output unit 11 can read and / or write the recording medium 103 a via the external I / F 103.

  The recording medium 103a is, for example, a recording medium such as an SD memory card. The recording medium 103 a may be a recording medium such as a USB memory (Universal Serial Bus memory), a DVD (Digital Versatile Disk), a CD (Compact Disk), a flexible disk, or the like.

  The camera 110 is an imaging device provided with a lens for forming an image, an aperture, and an imaging element such as a CMOS or a CCD. Not only still pictures but also moving pictures can be taken.

  The RAM 104 is a volatile semiconductor memory (storage device) that temporarily holds programs and data. The ROM 105 is a non-volatile semiconductor memory (storage device) capable of holding programs and data even after the power is turned off. The ROM 105 stores programs and data such as a BIOS (Basic Input / Output System), an OS setting, and a network setting which are executed when the image data output unit 11 is started.

  The CPU 106 is an arithmetic device that implements control and functions of the entire image data output unit 11 by reading programs and data from a storage device such as the ROM 105 or the SSD 108 onto the RAM 104 and executing processing.

  The communication I / F 107 is an interface for performing communication. The communication I / F 107 communicates with the handheld printer 20 according to a communication standard such as Bluetooth (registered trademark) or a wireless LAN, for example. In addition, Wi-Fi Direct, an ad hoc mode, etc. exist as a communication standard of wireless LAN which does not go through an access point. Communication I / F 107 may be an interface for connecting to a mobile telephone network.

  The SSD 108 is a non-volatile storage device storing programs and data. The stored programs and data include, for example, an OS (Operating System) which is basic software for controlling the entire image data output unit 11, and application software for providing various functions on the OS (including an output unit program to be described later) and so on. The image data output unit 11 may include an HDD (Hard Disk Drive) or the like instead of or in combination with the SSD 108.

  The short distance wireless communication device 109 communicates in accordance with a communication standard using an IC chip such as NFC (Near Field Communication) or TransferJet (registered trademark), for example. The image data output unit 11 transmits the image data 13 to the handheld printer 20 using the communication I / F or the short distance wireless communication device 109. In this case, the handheld printer 20 can also communicate with the IC chip.

  Subsequently, the hardware configuration of the handheld printer 20 will be described with reference to FIG. The handheld printer 20 is an example of an image forming apparatus that forms an image on the print medium 12. The entire operation of the hand-held printer 20 is controlled by the control unit 25, and the communication I / F 27, the IJ recording head drive circuit 23, the OPU 26, the ROM 28, the DRAM 29 and the navigation sensor 30 are electrically connected to the control unit 25. ing. Further, since the hand-held printer 20 is driven by electric power, it has a power supply 22 and a power supply circuit 21. The power generated by the power supply circuit 21 is supplied to the communication I / F 27, the IJ recording head drive circuit 23, the OPU 26, the ROM 28, the DRAM 29, the IJ recording head 24, the control unit 25, and the navigation sensor 30. It is supplied.

  The power source 22 mainly uses a battery. A solar cell, a commercial power source (AC power source), a fuel cell or the like may be used. The power supply circuit 21 distributes the power supplied by the power supply 22 to each part of the handheld printer 20. Further, the voltage of the power supply 22 is stepped down or boosted to a voltage suitable for each part. Further, when the power supply 22 can be charged by the battery, the power supply circuit 21 detects the connection of the AC power supply and connects it to the charging circuit of the battery to enable charging of the power supply 22.

  The communication I / F 27 receives image data from the image data output unit 11 such as a smartphone or a PC. The communication I / F 27 is, for example, a communication device compatible with a communication standard such as wireless LAN, Bluetooth (registered trademark), NFC (Near Field Communication), infrared, 3G (mobile phone), or LTE (Long Term Evolution). Further, in addition to such wireless communication, a communication device compatible with wired communication using a wired LAN, a USB cable or the like may be used.

  The ROM 28 stores firmware for performing hardware control of the hand-held printer 20, drive waveform data of the IJ recording head 24 (data defining a voltage change for discharging a droplet), initial setting data of the hand-held printer 20, etc. doing.

  The DRAM 29 is used to store image data received by the communication I / F 27 and to store firmware developed from the ROM 28. Therefore, it is used as a work memory when the CPU 31 executes the firmware.

  The navigation sensor 30 is a sensor that detects the position of the handheld printer 20. The navigation sensor 30 has, for example, a light source such as a light emitting diode (LED) or a laser, an imaging sensor for imaging the print medium 12, or a sensor for imaging interference fringes of incident and reflected light. As the hand-held printer 20 is scanned over the print medium 12, minute edges of the print medium 12 are detected one after another (imaged), and the distance between the edges is analyzed to obtain the movement amount. The navigation sensor 30 is mounted on at least two places of the hand-held printer 20. When the two are distinguished, they are referred to as navigation sensors S0 and S1. A multi-axis acceleration sensor or a gyro sensor may be used as the navigation sensor 30, or the position of the hand-held printer 20 may be detected only by the acceleration sensor or the gyro sensor.

  An operation panel unit (OPU) 26 has an LED for displaying the state of the hand-held printer 20, a switch for the user to instruct the hand-held printer 20 to form an image, and the like. However, the present invention is not limited to this, and may have a liquid crystal display, and may further have a touch panel. In addition, it may have a voice input function.

  The IJ recording head drive circuit 23 generates a drive waveform (voltage) for driving the IJ recording head 24 using the above-described drive waveform data. It is possible to generate a drive waveform according to the size of the ink droplet and the like.

  The IJ recording head 24 is a head for ejecting ink. In the drawing, the four color inks of CMYK can be discharged, but a single color or five or more colors may be discharged. A plurality of ink discharge nozzles (described later) arranged in one line (or two or more lines) may be arranged for each color. Also, the ink ejection method may be a piezo method, a thermal method, or another method.

  The control unit 25 determines the position of each nozzle of the IJ recording head 24 based on the movement amount detected by the navigation sensor 30, and determines an image to be formed according to the position, and determines whether the discharge nozzle is possible (the position of the nozzle is the target discharge position And the like) and the like. The details of the control unit 25 will be described below.

  FIG. 5 is an example of a diagram for explaining the configuration of the control unit 25. As shown in FIG. The control unit 25 includes a SoC 50 and an ASIC / FPGA 40. The SoC 50 and the ASIC / FPGA 40 communicate via buses 45 and 46. The ASIC / FPGA 40 means that it may be designed with either implementation technology, and may be configured with other implementation technologies other than the ASIC / FPGA 40. Also, the SoC 50 and the ASIC / FPGA 40 may be configured as one chip or base without being separate chips. Alternatively, it may be implemented with three or more chips or platforms.

  The SoC 50 has functions of a CPU 31, a position calculation circuit 32, a memory CTL (controller) 35, a ROM CTL (controller) 36, and the like connected via a bus 46. The components of SoC 50 are not limited to these.

  The ASIC / FPGA 40 also includes an Image RAM 37, a DMAC 38, a rotator 39, an interrupt controller 41, a navigation sensor I / F 42, a print / sensor timing generator 43, and an IJ recording head controller 44 connected via a bus 45. have. The components of the ASIC / FPGA 40 are not limited to these.

  The CPU 31 executes firmware (program) or the like expanded from the ROM 28 to the DRAM 29, and controls the operation of the position calculation circuit 32, the memory CTL 35, and the ROM CTL 36 in the SoC 50. Further, it controls the operations of the image RAM 37, the DMAC 38, the rotator 39, the interrupt controller 41, the navigation sensor I / F 42, the print / sensor timing generator 43, and the IJ print head controller 44 in the ASIC / FPGA 40.

  The position calculation circuit 32 calculates the position (coordinate information) of the hand-held printer 20 based on the movement amount for each sampling cycle detected by the navigation sensor 30. The position of the hand-held printer 20 is strictly the position of the nozzle 61, but if the position of the navigation sensor 30 is known, the position of the nozzle 61 can be calculated. In this embodiment, the positions of the navigation sensors S0 and S1 are referred to as the position of the navigation sensor 30 unless otherwise specified. Further, the position calculation circuit 32 calculates a target discharge position.

  The position of the navigation sensor 30 is calculated based on, for example, a predetermined origin (the initial position of the hand-held printer 20 when the image formation is started) as described later. The position calculation circuit 32 estimates the moving speed and the moving direction based on the difference between the past position and the most recent position, and predicts, for example, the position at the next calculation timing. By doing this, it is possible to discharge the ink while suppressing the delay with respect to the scan of the user.

  The memory CTL 35 is an interface with the DRAM 29, requests data from the DRAM 29, sends the obtained firmware to the CPU 31, and sends the obtained image data to the ASIC / FPGA 40.

  The ROM CTL 36 is an interface with the ROM 28, requests data from the ROM 28, and sends the acquired data to the CPU 31 or the ASIC / FPGA 40.

  The DMAC 38 acquires the image data 13 around each nozzle of the IJ recording head 24 via the memory CTL 35 based on the position information calculated by the position calculation circuit 32. That is, image data around the position where the handheld printer 20 is present with respect to the print medium 12 is acquired.

  The rotator 39 rotates the image data acquired by the DMAC 38 in accordance with the head that ejects ink and the position of the nozzle in the head. The DMAC 38 outputs the image data after rotation to the IJ recording head control unit 44. The rotator 39 can obtain, for example, the rotation angle θ calculated when the position calculation circuit 32 calculates the position, and can rotate the image using the rotation angle θ.

  The image RAM 37 temporarily stores the image data 13 acquired by the DMAC 38. That is, a certain amount of image data 13 is buffered and read out according to the position of the hand-held printer 20.

  The IJ recording head control unit 44 subjects the image data (bit map data) to dithering or the like to convert the image data 13 into a set of points representing the image in size and density. Thereby, the image data 13 becomes data of the ejection position and the size of the point. The IJ recording head control unit 44 outputs a control signal corresponding to the size of the point to the IJ recording head drive circuit 23. As described above, the IJ recording head drive circuit 23 generates a drive waveform (voltage) using the drive waveform data corresponding to the control signal.

  The navigation sensor I / F 42 communicates with the navigation sensor 30, receives movement amounts ΔX ′ and ΔY ′ (these will be described later) as information from the navigation sensor 30, and stores the values in an internal register.

  The print / sensor timing generation unit 43 notifies the navigation sensor I / F 42 of the timing for reading the information of the navigation sensor 30, and notifies the IJ recording head control unit 44 of the drive timing. The IJ print head control unit 44 determines whether or not the ejection nozzle is to be ejected, and when the target ejection position where the ink should be ejected is present, the ink is ejected, and when the target ejection position is not present, it is determined that ejection is not performed.

  The interrupt controller 41 detects that the navigation sensor I / F 42 has completed communication with the navigation sensor 30, and outputs an interrupt signal for notifying the SoC 50 of it. The CPU 31 acquires ΔX ′ and ΔY ′ stored in the internal register by the navigation sensor I / F 42 by this interrupt. In addition, it also has a status notification function such as an error.

<Functions of Image Data Output Device 11 and Handheld Printer>
Subsequently, functions of the image data output unit 11 and the hand-held printer 20 will be described with reference to FIG. FIG. 6 is an example of a functional block diagram of the image data output unit 11 and the hand-held printer 20. As shown in FIG.

<< Image data output unit >>
The image data output unit 11 includes a transmission / reception unit 51, an imaging unit 52, a mark detection unit 53, a pattern matching unit 54, a position determination unit 55, and a storage / readout unit 59. Each of the units included in the image data output unit 11 operates according to an instruction from the CPU 106 according to the output unit program 1001 expanded on the RAM 104 from the SSD 108 and any one of the components shown in FIG. Function or means implemented.

  Further, the image data output unit 11 has a storage unit 1000 constructed by the SSD 106, the RAM 104, or the recording medium 103a shown in FIG. The storage unit 1000 stores an output unit program 1001, imaging data 14, and image data 13. The output unit program 1001 may be distributed from the storage device 103 a as well as from the server device that distributes the program.

  The transmission / reception unit 51 is realized by an instruction from the CPU 106 shown in FIG. 4 and the communication I / F 107, and transmits / receives various data to / from the handheld printer 20.

  The imaging unit 52 is realized by an instruction from the CPU 106 shown in FIG. 4 and the camera 110, and images the print medium 12 and the hand-held printer 20. The imaging may be triggered by the user's operation or triggered by satisfying the conditions described later in the third embodiment and the like.

  The mark detection unit 53 is realized by an instruction or the like from the CPU 106 shown in FIG. 4, detects the position detection marks M1 and M2 from the imaging data 14, and detects the position detection marks M1 and M2 with respect to the comparison image 14a in the imaging data 14. Identify the relative position of M2.

  The pattern matching unit 54 is realized by an instruction or the like from the CPU 106 shown in FIG. 4, performs pattern matching on the image data 13 using the matching image 14a, and matches the matching image 13a that matches the matching image 14a. To detect.

  The position determination unit 55 determines the positions of the position detection marks M1 and M2 with respect to the matching image 13a of the image data 13 based on the relative positions of the position detection marks M1 and M2 with respect to the verification image 14a in the imaging data 14. This is taken as the current position (position at the time of imaging) of the handheld printer 20 in the image data 13. The position of the hand-held printer 20 is transmitted to the hand-held printer 20 via the transmission / reception unit 51.

  The storage / readout unit 59 is realized by an instruction from the CPU 106 shown in FIG. 4 and the SSD 108 etc., and stores various data in the storage unit 1000 and reads out various data stored in the storage unit 1000. Do the processing.

<< Handheld Printer >>
The hand-held printer 20 has a transmitting / receiving unit 65, a position loss detecting unit 66, a position setting unit 67, and a storage / reading unit 69. These units included in the hand-held printer 20 operate according to an instruction from the CPU 31 according to the printer program 2001 expanded on the DRAM 29 from the ROM 28 and any of the components shown in FIGS. Function or means implemented.

  The hand-held printer 20 also has a storage unit 2000 constructed by the ROM 28 or the DRAM 29 shown in FIG. The storage unit 2000 stores a printer program 2001.

  The transmission / reception unit 65 is realized by an instruction from the CPU 31 shown in FIG. 5 and the communication I / F 27 etc., and transmits / receives various data to / from the image data output unit 11.

  The position loss detection unit 66 is realized by an instruction from the CPU 31 shown in FIG. 5, the navigation sensor I / F 42, the position calculation circuit 32, and the like, and the handheld printer 20 floats from the print medium 12 or the current position. Detect lost sight. The navigation sensor 30 condenses the reflected light, which is the light of the light source reflected by the printing medium 12, with a lens, and captures an image with an image array. When the hand-held printer 20 floats from the print medium 12, the distance between the print medium 12 and the image array increases and the reflected light can not be detected. As a result, the position loss detection unit 66 detects that the navigation sensor 30 has floated or lost the current position. Also, for example, the current position may be detected by detecting the maximum speed that can be calculated by the position calculation circuit 32 continuously.

  The position setting unit 67 is realized by an instruction from the CPU 31 and the position calculation circuit 32 shown in FIG. 5 and sets the position of the handheld printer 20 transmitted from the image data output unit 11 in the position calculation circuit 32. .

  Storage / readout unit 69 is realized by an instruction from CPU 31 shown in FIG. 5, ROM 28 and DRAM 29 etc., stores various data in storage unit 2000, reads out various data stored in storage unit 2000, etc. Do the process.

<About the nozzle position in IJ recording head>
Next, nozzle positions and the like in the IJ recording head 24 will be described with reference to FIG. FIG. 7A is an example of a plan view of the hand-held printer 20. FIG. FIG. 7B is an example of a diagram for explaining only the IJ recording head 24. The surface shown is the surface facing the print medium 12.

  The handheld printer 20 of the present embodiment has two or more navigation sensors 30. With two or more navigation sensors 30, the rotation angle θ can be detected even if the navigation sensor 30 rotates with respect to the print medium 12 during image formation. In FIG. 7A, two navigation sensors S0 and S1 are disposed apart from each other in the direction in which the nozzles 61 are arranged. The length between the two navigation sensors S0, S1 is the distance L. The longer the distance L, the better. This is because the longer the distance L, the smaller the minimum detectable rotation angle θ, and the smaller the position error of the hand-held printer 20.

  The distances from the navigation sensor 30 to the IJ recording head 24 are distances a and b, respectively. The distance a and the distance b may be equal. Further, as shown in FIG. 7B, the distance from the end of the IJ recording head 24 to the first nozzle 61 is a distance d, and the distance between adjacent nozzles is a distance e. The values a to e are stored in advance in the ROM 28 or the like.

  Therefore, if the position calculation circuit 32 or the like calculates the position of the navigation sensor 30, the position calculation circuit 32 can calculate the position of the nozzle 61 using the distances a, b, d, and e.

  In the present embodiment, the direction horizontal to the print medium 12 is set to the X axis, and the direction perpendicular to the print medium 12 is set to the Y axis. This coordinate is referred to as print medium coordinate. On the other hand, the navigation sensor 30 outputs the position on the following coordinate axes (X 'axis, Y' axis). That is, the arrangement direction of the nozzles 61 (the direction connecting the two navigation sensors S0 and S1) is taken as the Y 'axis, and the direction orthogonal to the Y' axis is taken as the X 'axis. The calculation of the position is performed by the position calculation circuit 32.

<About the position of the handheld printer on the print medium>
Next, the position of the handheld printer 20 will be described with reference to FIG. In FIG. 8A, the handheld printer 20 rotates clockwise by θ with respect to the print medium 12. If it is not rotating at all, X = X 'and Y = Y'. However, when the hand-held printer 20 is rotated relative to the print medium 12 by the rotation angle θ, the outputs of the navigation sensors S 0, S 1 do not match the actual position of the hand-held printer 20 in the print medium 12.

  Therefore, as shown in FIG. 8A, ΔX ′ and ΔY ′ output by the navigation sensors S0 and S1 correspond to X and Y of the print medium coordinates as follows. In FIG. 8A, the movement amounts ΔX ′ and ΔY detected by the navigation sensors S0 and S1 (the output is equal because they are parallel movement) when the hand-held printer 20 with the rotation angle θ moves in the X direction only with the same rotation angle θ. 'Shows the correspondence between X and Y. ΔX ′ output from the navigation sensors S0 and S1 is reflected on X1, and ΔY ′ is reflected on X2.

  In FIG. 8B, the movement amounts ΔX ′ and ΔY detected by the navigation sensors S0 and S1 (parallel displacements are equal) when the hand-held printer 20 with the rotation angle θ moves with the same rotation angle θ only in the Y direction. 'Shows the correspondence between X and Y. ΔY ′ output from the navigation sensors S0 and S1 is reflected in Y1, and −ΔX ′ is reflected in Y2.

Therefore, when the hand-held printer 20 moves in the X direction and the Y direction with the rotation angle θ, ΔX ′ and ΔY ′ output by the navigation sensors S0 and S1 can be converted into X and Y of print medium coordinates as follows.
X = ΔX ′ cos θ + ΔY ′ sin θ (1)
Y = −ΔX ′ sin θ + ΔY ′ cos θ (2)
As described above, if the rotation angle θ is known based on the image formation start position, the position of the print medium coordinates of the navigation sensors S0 and S1 can be obtained from the equations (1) and (2). The start position of the image formation is a position where the user presses a button or the like of the hand-held printer 20 to start the image formation, and is, for example, slightly inside the upper left corner of the print medium 12.

The rotation angle dθ at the sampling time of ΔX ′ and ΔY ′ can be determined as follows from the difference between the outputs (ΔX ′ ₀ , ΔX ′ ₁ ) of the two navigation sensors S0 and S1. The rotation angle θ can be determined by accumulating dθ.
dθ = arcsin {(ΔX ′ ₀ −ΔX ′ ₁ ) / L} (3)
<Position detection mark>
The position detection marks M1 and M2 will be described with reference to FIG. FIG. 9 shows an arrangement example of the position detection marks M1 and M2 in the top view of the hand-held printer 20. If the positions of at least two points of the hand-held printer 20 are known, the position and attitude of the hand-held printer 20 can be estimated. For this reason, the position detection marks M1 and M2 are arranged at two positions so that the positions of the two points can be specified.

  The position detection marks M1 and M2 may be integrally formed on the HHP 20 or may be attached like a seal. Further, when the OPU 26 is a liquid crystal display or the like, the OPU 26 may display.

  In FIG. 9A, position detection marks M1 and M2 are arranged at the upper right corner and lower left corner of the hand-held printer 20, respectively. Since both of the position detection marks M1 and M2 indicate the coordinates of one point of the hand-held printer 20, for example, the position of the upper left vertex P1 of the position detection mark M1 is detected from the imaging data 14, and the upper vertex P2 of the position detection mark M2 is detected. Position is detected. Other vertices may be detected.

  Further, in FIG. 9A, the orientations of the position detection marks M1 and M2 are different (the position detection marks M2 are in a rhombus shape), but as shown in FIG. 9B, the position detection marks M1 and M2 are shown. The direction of may be the same. Further, as shown in FIG. 9C, position detection marks M1 and M2 may be arranged at the upper right corner and the upper left corner of the hand-held printer 20, respectively.

  Further, in FIG. 9D, only one position detection mark M1 is disposed at the upper right corner of the hand-held printer 20. The position detection mark M1 has a solid rectangular portion at the center and a square surrounding the solid portion. In such a position detection mark M1, the ratio of lengths of black and white pixels through which a straight line passes in the vertical direction and the horizontal direction is different. For example, 1 (black): 1 (white): 3 (black): 1 (black) in the vertical direction, 1 (black): 1 (white): 7 (black): 1 (black) in the left-right direction White): 1 (black). When such a predetermined ratio of black and white pixels is detected by two straight lines having different angles by 90 degrees, it may be determined that the position detection mark M1 has been detected, so the detection accuracy of the position detection mark M1 is Can be improved. Therefore, even when only one position detection mark M1 is disposed, for example, the upper left vertex P3 and the lower right vertex P4 of the position detection mark M1 can be adopted as the two positions of the handheld printer 20, respectively.

  Similarly, even if only one of the rectangular position detection marks M1 and M2 shown in FIGS. 9A to 9C is disposed, the position detection mark M1 (or M2) can recognize the direction such as a rectangle. If it is a shape, the position of any two or more vertices of the position detection mark M1 (or M2) may be adopted as the coordinates of two points of the handheld printer 20.

  The position detection marks M1 and M2 may be triangles, stars or the like. When two position detection marks M1 and M2 are arranged, the position detection marks M1 and M2 may be circular.

Subsequently, a method of detecting the position detection marks M1 and M2 will be described. The position detection marks M 1 and M 2 are formed in a color different from the color of the housing of the handheld printer 20. Therefore, the mark detection unit 53 can detect the position detection marks M1 and M2 from the imaging data 14 in the following procedure, for example. Below, the detection method of position detection mark M1, M2 of FIG. 9 (a)-(c) is demonstrated.
(i) Binarize the image into white or black.
(ii) Search for continuous black pixels.
(iii) Detect edges in the vertical and horizontal directions.
(iv) A straight line is detected by the Hough transform, and a black pixel surrounded by four straight lines is detected.
(v) When two black pixels in (iv) are found (if it is found in more than two places, discrimination is made based on the aspect ratio of four straight lines surrounding the rectangle), each is determined as the position detection marks M1 and M2.

  Note that such a method of detecting the position detection mark is only an example, and for example, each vertex of the position detection marks M1 and M2 may be detected using a corner detection method of Harris in (iv). Further, for example, the identification device which has learned the position detection marks M1 and M2 by machine learning may detect the position detection marks.

  FIG. 10 is an example of a diagram showing the relative positions of the position detection marks M 1 and M 2 and the navigation sensor 30. As shown in FIG. 10, the distance α and the distance β from the upper left vertex P1 of the position detection mark M1 to the navigation sensor S0 are known. Therefore, if the positions of the position detection marks M1 and M2 are detected, the position of the navigation sensor S0 can be identified.

  It is also possible to obtain the position of the navigation sensor S0 from the position detection mark M2. Further, since the relative position of S1 with respect to the navigation sensor S0 is also known, if the navigation sensor S0 or S1 is known, the position of the navigation sensor S1 or S0 can also be specified.

<Acquisition of image for comparison from imaging data>
Next, acquisition of a comparison image from imaging data will be described. First, with reference to FIG. 11A, the image data 13 included in the image data output unit 11 will be described. FIG. 11A shows an example of the image data 13 with coordinates. The image data output unit transmits the image data 13 of FIG. Further, since the image data output unit 11 holds the entire image data 13, the coordinates of each pixel of the image data 13 can also be specified. In addition, although the upper left corner of the image data 13 is made into origin (0, 0) for convenience of explanation, an origin is good for anywhere.

  When the hand-held printer 20 forms an image halfway but loses the current position by the end of the image formation, the user picks up the hand-held printer 20 and the print medium 12. FIG. 11B is a view showing the attitude of the image data output unit 11 at the time of imaging. As shown in FIG. 11B, the optical axis 7 of the image data output unit 11 is preferably perpendicular to the print medium 12. As a result, distortion (projective deformation) caused by the tilt of the optical axis 7 does not occur in the imaging data 14, so that the comparison between the imaging data 14 and the image data 13 becomes easy.

<< Scaling of imaging data >>
Here, the size of the imaging data 14 may vary depending on the distance at which the image data output device 11 performs imaging with respect to the printing medium 12 and the focal distance (magnification). On the other hand, the size when the image data 13 is formed on the print medium 12 is fixed (for example, the size fits in A4). For this reason, the pattern matching unit 54 sets the imaging data 14 so that the distance between the position detection marks M1 and M2 (the length of the side when there is only one position detection mark) becomes the actual length at the time of image formation. Scale the For example, it is assumed that one pixel (including a pitch interval between pixels) is known to correspond to the length of s [mm] in real space. Further, it is assumed that the actual length of the position detection marks M1 and M2 is 50 mm, and the distance between the position detection marks M1 and M2 in the imaging data 14 is 300 pixels. In this case, since the distance between the position detection marks M1 and M2 at the time of image formation is “s × 300”, the imaging data 14 is scaled so as to be 50 [mm]. Therefore, by referring to a table or the like in which the value of s is predetermined for each focal length, the pattern matching unit 54 can perform scaling so that the imaging data 14 has the same size as the image data 13. The image data 13 may be scaled to have the same size as the image data 13 by scaling the image data 13.

  The pattern matching unit 54 may calculate the distance between the image data output unit 11 and the print medium 12 from the number of pixels of the position detection marks M1 and M2. Then, the imaging data 14 may be scaled according to the distance. The relationship between the distance and the scaling factor is preset so that the sizes of the imaging data 14 and the image data 13 match.

<< Acquisition of matching image and pattern matching >>
FIG. 11C illustrates an example of the imaging data 14 captured by the image data output unit 11. In the imaging data 14, the formed image 71 and the hand-held printer 20 are imaged on a print medium. The relative position of the formed image 71 and the hand-held printer 20 is apparent. Therefore, if the formed image 71 can be detected from the image data 13, the relative position can be used to determine the position of the hand-held printer 20 in the image data 13.

  The pattern matching unit 54 extracts a part or all of the periphery or near the hand-held printer 20 in the formed image 71 as a verification image 14 a. First, the verification image 14a including the character string will be described. The use of the character string is for the convenience of explanation, and the image for matching 14a may be an image (image).

  The pattern matching unit 54 extracts a character string by image processing. The character region is, for example, obtained as a region for which the spatial frequency is obtained and the height is equal to or more than a threshold. Further, a circumscribed rectangle of continuous black pixels is determined, and when the circumscribed rectangles of substantially constant height are arranged in one direction at substantially constant intervals, it can be determined as a character string. In this embodiment, it is assumed that the verification image 14a including the characters "HAND HELD PRIN" is extracted.

  FIG. 12 is an example of a diagram schematically illustrating pattern matching using the matching image 14a on the image data 13. As shown in FIG. FIG. 12A shows the verification image 14 a and the image data 13. For example, the pattern matching unit 54 calculates the difference for each pixel (or for each pixel block) by aligning the upper left corner of the image for matching 14a with the origin (0, 0). If both the image data 13 and the verification image 14a are black pixels or white pixels, the difference between the pixels is zero. On the other hand, if only one of the image data 13 and the verification image 14a is a black pixel or a white pixel, the absolute value of the difference between the pixels is one. Therefore, the smaller the sum of the differences for the area included in the matching image 14a, the smaller the matching image 14a matches the image data 13.

  As shown in FIG. 12B, the pattern matching unit 54 calculates the sum of differences while shifting the comparison image 14a with respect to the image data 13 by one pixel (or while shifting by pixel blocks). An area overlapping the image for comparison 14a when the sum of the differences is the smallest is the matched image 13a.

  In FIG. 12C, the matching image 14a matches the image data 13. Thus, the matching image 13a is specified from the image data 13.

  In the case of an image (image) instead of a character, pattern matching can be similarly performed. That is, pattern matching may be performed using any rectangular area around or near the hand-held printer 20 as the verification image 14a.

  However, the time required for detecting the matching image 13a can be shortened by specifying whether the image for matching 14a is a character or an image (image). That is, in the image data 13 as well, pattern matching may be performed only in the character area or the image area. The image area may be determined as an area that is not a character area, or can be detected by clustering pixels into a color space.

  When pattern matching is performed using characters, the pattern matching unit 54 may perform pattern matching in character units instead of performing pattern matching in pixel units. For example, the pattern matching unit 54 performs an optical character reader (OCR) on the comparison image 14a. As a result, since the character code included in the collation image 14a is known, the pattern matching unit 54 performs the OCR also on the image data 13, and searches for a character string in which the same character code is arranged. Thereby, the location of the matched image 13a of the image data 13 is known.

<< When the match image 13a can not be determined uniquely >>
Here, it is assumed that the matching image 13a can not be determined uniquely. If the image data 13 already formed is extremely small (for example, only 2 or 3 characters are formed and the same character is plural in the image data) or if the image data 13 has a plurality of identical character patterns, a matching image 13a can not be identified. In the former case, the pattern matching unit 54 determines that the character is located near the image formation start position. In the latter case, the pattern matching unit 54 causes the display device 102 to display a plurality of character patterns for the user to select.

  FIG. 13 is a view showing an example of the inquiry screen 1300 displayed on the display device 102. As shown in FIG. In the inquiry screen 1300, image data 13 and imaging data 14 are displayed. The pattern matching unit 54 selects the matching image 14 a from the imaging data 14 and detects four matching images 13 b 1 to 13 b 4 from the image data 13. The pattern matching unit 54 emphasizes the four matching images 13b1 to 13b4 by enclosing the four matching images 13b1 to 13b4 with a rectangle. Further, a message 1301 such as “Please select a character string corresponding to the character string surrounded by the right box” is displayed on the inquiry screen 1300. The user determines which of the matching images 13b1 to 13b4 the matching image 14a of the imaging data 14 corresponds to, and selects (touches) any one of the four matching images 13b1 to 13b4.

<< Another example of pattern matching >>
Another method of pattern matching will be described with reference to FIG. FIG. 14 is a diagram showing an example of pattern matching focusing on the side length of the matching image 14a.

  This method is a method of detecting a straight line (or each vertex) connecting short sides (ab), long sides (bc) and diagonals (ac) of the collation image 14a (here, "HAND") including characters. The arithmetic processing load can be reduced compared to pattern matching at the pixel level.

  The pattern matching unit 54 extracts, from the imaging data 14, a set of characters divided by a predetermined margin or more. This group is the verification image 14a. In the case of English, it may be cut out for each word. In languages such as Japanese where there is no space between words, the characters between punctuation marks are cut out. This is to remove the space and retrieve the string. In addition, by focusing on words and punctuation marks, it is possible to extract a set of characters from the image data 13 at the same division. This aggregate is a candidate for the matching image 13b. In addition, the character string which spread over 2 lines by line feed is not included.

  Since some image data 13 may have little space between lines, etc., the division of character chunks is not generally decided, but a certain threshold is set to detect chunks in order to be treated as a chunk (aggregate). Just do it.

  In the example of FIG. 14A, each vertex abc of the matching image 14a including a set of characters “HAND” forms a triangle. In FIG. 14B, a triangle is formed by the vertices a ′, b ′ and c ′ in the candidate rectangular regions of the matching image 13a including the character assembly “HAND”.

  Therefore, if a triangle having a side with the same length as the triangle abc in the matching image 14a is found from the candidates of the matching image 13a, it is highly likely that it is the matching image 13a.

  In the present embodiment, it is assumed that the imaging data 14 is not rotated with respect to the image data 13, and even if it is rotated, the imaging unit 52 pays attention to a character string or the like and performs inclination correction. The difference in the angle of the imaging data 14 may not be taken into consideration.

  However, as shown in FIG. 14C, when the imaging data 14 is rotated with respect to the image data 13, the pattern matching unit 54 may rotate the verification image 14a for confirmation. First, the coordinates of the point a and the point a ′ are aligned, and the coordinates of the points c and c ′ are rotated so as to match. Alternatively, points b and b 'may be used, but the longer the distance, the higher the calculation accuracy.

  Here, it is related to the number of bits on the program that the distance length is due to the precision. Arithmetically, the value to be handled is a finite number of bits (for example, C language double type is 16 bits). Therefore, the operation deviation of the minimum unit occurs. In addition, when the image data 13 is co-ordinated, it is similarly influenced by the finite number of bits. From the above, it can be seen that although the minimum unit that can be shifted is the same, the influence of the minimum unit error is reduced as the distance is relatively longer when the distances to be represented are different.

<Position of handheld printer in image data>
FIG. 15 is an example of a diagram for explaining the position of the hand-held printer 20 in the image data 13. FIG. 15A shows the relative position of the hand-held printer 20 in the imaging data 14. The position detection marks M1 and M2 are detected from the imaging data 14, and the relative position with the verification image 14a is also clarified. For example, the positions of the position detection marks M1 and M2 are represented by the distance from the upper left vertex P5 of the verification image 14a and the azimuth φ.
Position detection mark M1: distance k1, azimuth φ1
Position detection mark M2: distance k2, azimuth φ 2
FIG. 15B shows the positions of the position detection marks M1 and M2 estimated to be the coincident image 13a in the image data 13. In the image data 13, the position detection mark M1 is at a distance k1 and azimuth φ1 with respect to the upper left vertex P6 of the coincident image 13a, and the position detection mark M2 is at a distance k2 and azimuth φ2 with respect to the upper left vertex P6 of the coincidence image 13a. Therefore, the positions (coordinates) of the position detection marks M1 and M2 in the image data 13 can be specified based on the upper left vertex P6 of the matching image 13 detected by pattern matching.

  The position determination unit 55 determines the coordinates (M0x, M0y) of the point P1 located at a distance k1 from the upper left vertex P6 of the matching image 13 and in the azimuth φ1 as the coordinates of the position detection mark M1. Similarly, the coordinates (M1x, M1y) of the point P2 at a distance k2 from the upper left vertex of the coincident image and in the orientation φ2 are determined as the coordinates of the position detection mark M2. The position determination unit 55 transmits the positions of (M0x, M0y) (M1x, M1y) thus determined or the positions of the navigation sensors S0 and S1 to the hand-held printer 20.

<Operation procedure>
FIG. 16 is an example of a flowchart showing an operation procedure of the image forming system 1 of the present embodiment.

  First, the user presses the power button of the handheld printer 20 (U01).

  The hand-held printer 20 is activated by power ON, and initializes the SoC 50 and the ASIC / FPGA 40 (H01).

  Thereafter, when the initialization is completed (Yes in H02), the handheld printer 20 turns on an LED indicating that the image formation can be started (H03).

  Next, the user operates the image data output unit 11 to select the image data 13 to be used for printing (U02).

  The user operates the image data output unit 11 to execute a print job (U03).

  On the other hand, the storage / readout unit 59 of the image data output unit 11 stores the image data 13 in the storage unit 1000 (D01).

  Further, the pattern matching unit 54 of the image data output unit 11 holds the coordinates of the image data 13 (D02). That is, the coordinates of each pixel when the image data 13 is formed on the print medium 12 are obtained. The transmitting and receiving unit 51 of the image data output unit 11 transmits the image data 13 to the handheld printer 20.

  Next, the hand-held printer 20 blinks the LED upon receiving the image data 13 (H04). Thus, the user recognizes that the image can be formed.

  The user determines the initial position of the handheld printer 20 (U04). Then, the printing start determination button is pressed (U05), and the operation is performed freehand (U06).

  The handheld printer 20 discharges ink according to the current position to form an image (H05).

  Then, it is assumed that the user floats the handheld printer 20 during scanning (U07).

  Thereby, the position loss detection unit 66 of the handheld printer 20 acquires the position information and the floating information and stores the information in the internal memory (H06).

  Thereafter, the user performs an operation necessary for acquiring the position. That is, the image data 13 already formed and the hand-held printer 20 are imaged into one image pickup data 14 (U08).

  The mark detection unit 53 of the image data output unit 11 detects the position detection marks M1 and M2 from the imaging data 14 (D03).

  Then, the pattern matching unit 54 acquires the verification image 14a from around or near the position detection mark, and performs pattern matching on the image data 13 (D04). The collation image 14a may be a character or an image (image).

  Next, the position determining unit 55 determines the relative positions of the position detection marks M1 and M2 with respect to the matching image 13a that matches the image for matching 14a in the image data 13 (D05).

  Then, the positions of the navigation sensors S0 and S1 or the positions of the position detection marks M1 and M2 are transmitted to the hand-held printer 20 (D06).

  The position setting unit 67 of the hand-held printer 20 sets the positions of the navigation sensors S0 and S1 in the position calculation circuit 32 (H07). Thus, the position calculation circuit 32 can calculate the position of the nozzle. The hand-held printer 20 notifies the user that image formation can be resumed by turning on an LED or the like. Thus, the user can resume image formation.

  As described above, in the image forming system 1 of the present embodiment, even if the position of the handheld printer 20 is lost, the image data output unit 11 can calculate the position and restart the image formation. There is no need to mount a unit or a scanner unit that ejects invisible ink.

  In the first embodiment, the image data output unit 11 captures the formed image and the hand-held printer 20 from the front of the print medium 12. In this embodiment, an image data output unit 11 capable of calculating the position of the hand-held printer 20 even when the optical axis at the time of imaging is inclined with respect to the print medium 12 will be described.

  FIG. 17 shows an example of a functional block diagram of the image data output unit 11 of this embodiment. Note that, in the present embodiment, the components given the same reference numerals in FIG. 6 perform the same functions, so only the main components of the present embodiment may be described. The image data output unit 11 of the present embodiment has a matrix calculation unit 56.

  The matrix calculation unit 56 is realized by an instruction or the like from the CPU 106 shown in FIG. 4 and calculates a projection transformation matrix described later.

<Projective transformation matrix>
The projective transformation matrix will be described. Consider that the print medium and the imaging data 14 are distorted when the optical axis of the image data output unit 11 is inclined. FIG. 18 is an example of a diagram for explaining distortion of the imaging data 14. 18 (a) shows an example of the image data 13 held by the image data output unit 11, and FIG. 18 (b) shows an example of the imaging data 14 imaged by the image data output unit 11. As shown in FIG.

Since the imaging data 14 is formed based on the image data 13, the imaging data 14 has a point corresponding to the image data 13. Assuming that the coordinates of the image data 13 are m _p (x _p , y _p ) and the coordinates of the imaging data 14 are m _c (x _c , y _c ), m _p and m _c are the projective transformation matrix H (an example of correction information) Can be expressed as follows. mp (xp, yp) is a coordinate of a coordinate system having the upper left vertex of the image data 13 as an origin, and m _c (x _c , y _c ) is a coordinate of a coordinate system having an upper left vertex of the imaging data 14 as an origin .

Then, the following constraint is obtained from one corresponding point of m _p (x _p , y _p ) and m _c (x _c , y _c ).

With one constraint, two constraint equations are obtained for eight unknown parameters. Therefore, if four corresponding points are obtained by the image data 13 and the imaging data 14, all elements of the projective transformation matrix H can be determined.

  The correspondence between the image data 13 and the imaging data 14 in FIG. 18 is not necessarily clear. Therefore, for example, the image data output unit 11 displays the image data 13 and the imaging data 14 on the display device 102, and causes the user to input corresponding points. In this case, the user indicates, for example, the same place of the corresponding character in the image data 13 and the imaging data 14 (for example, the upper left vertex of H, the upper right vertex of N, the lower right vertex of N, the lower left vertex of H, etc.).

  Alternatively, the image data output unit 11 acquires the image-formed pixels of the image data 13 from the hand-held printer 20, and extracts corresponding points as follows. As described above, since the hand-held printer 20 records the pixels which have ejected ink, the pixels on which the image is formed are clear.

In the example of FIG. 18, the characters “HAND HELD PRIN” of the image data 13 have been formed. The matrix calculation unit 56 adopts the upper left vertex, the upper right vertex, the lower right vertex, and the lower left vertex of the formed area as corresponding points. Therefore, the matrix calculation unit 56 sets the upper left vertex m _p1 of the image data 13, the upper right vertex m _p2 of N, the lower right vertex m _{p3 of} N, and the lower left vertex m _{4 of} H as corresponding points. Therefore, the matrix calculation unit 56 acquires, as coordinate data, that the image “HAND HELD PRIN” has been formed from the handheld printer 20, and determines four vertices m _p1 , m _p2 , m _p3 and m _p4 from the coordinate data. (The coordinates of the points m _{p1 to} m _p4 possessed by the hand-held printer 20 are the same as the coordinates of the points m _{c1 to} m _c4 of the image data 13).

The points m _c1 , m _c2 , m _c3 and m _c4 corresponding to the points m _p1 , m _p2 , m _p3 and m _p4 in the imaging data 14 are obtained as follows. That is, the point _mc1 is obtained by searching for the top left pixel, the point _mc2 the top right pixel, the point _mc3 the right bottom pixel, and the point _mc4 the bottom left pixel. In order to reduce the influence of noise such as a single point, preferably, the imaging data 14 is smoothed by a filter or the like.

  Although the case where the matching image 14a is a character string has been described above, the upper left vertex, the upper right vertex, the upper right vertex, and the lower left vertex of the image (image) can be specified as well if the image (image) is formed. Corresponding points can be determined. In addition, since a larger area surrounded by the corresponding points can obtain H with high accuracy, it is preferable to determine the corresponding points as wide as possible.

<Acquisition of image for comparison>
As in the first embodiment, the mark detection unit 53 can detect the position detection marks M1 and M2. The pattern matching unit 54 determines an image of a predetermined range around or near the position detection marks M1 and M2 as the verification image 14a.

<Pattern matching>
Since pattern matching is preferably performed on the verification image 14a without distortion, the pattern matching unit 54 applies all of the pixels of the verification image 14a to Equation (4) to correct distortion (image taken from the front) Convert to state). FIG. 19A shows the collation image 14a before conversion, and FIG. 19B shows the collation image 14a after conversion. The transformation reduces the distortion of the verification image 14a. Therefore, the pattern matching unit 54 can perform pattern matching as in the first embodiment. In FIG. 19, the range of the image data 13 for which the projective transformation matrix H is obtained is the same as the matching image 14a, but may be different.

  The pattern matching unit 54 converts the coordinates of the position detection marks M1 and M2 in the coordinate system of the verification image 14a according to equation (4). This makes it possible to calculate how much the distance and orientation of the position detection marks M1 and M2 with respect to the verification image 14a are obtained from the front (that is, in the image data 13). Therefore, as shown in FIG. 19B, the positions of the position detection marks M1 and M2 whose distortion is corrected with respect to the distortion-corrected verification image 14a are known.

<Position of handheld printer in image data>
It is assumed that the pattern matching unit 54 detects the matching image 13 a in the image data 13. The position determination unit 55 applies the distances k1 and k2 and the azimuths φ1 and φ2 obtained as shown in FIG. 19B to the coincident image 13a to determine the positions of the position detection marks M1 and M2 in the image data 13.

  As described above, according to the present embodiment, even if the image data output unit 11 does not capture the image formed with the print medium 12 from the front, the position of the handheld printer 20 can be calculated to resume image formation.

  In this embodiment, an image forming system 1 using a hand-held printer 20 in which non-front marks are arranged so as to be imaged from the front will be described. The functional block diagram of this embodiment may be the same as that of FIG. 6 of the first embodiment.

  Non-front marks Q1 to Q4 (an example of a second mark) disposed on the hand-held printer 20 will be described with reference to FIG. FIG. 20 shows an example of the hand-held printer 20 and non-front marks Q1 to Q4 arranged on the hand-held printer 20. As shown in FIG. The position detection marks M1 and M2 are the same as in the first and second embodiments. On the other hand, in the present embodiment, non-front marks Q1 to Q4 are arranged on the four side surfaces of the hand-held printer 20. Since the non-front marks Q1 to Q4 have no thickness, they do not appear in the imaging data 14 when the optical axis of the image data output unit 11 is perpendicular to the handheld printer 20. In other words, it means that the optical axis is not vertical while the non-front marks Q1 to Q4 appear.

  Although the non-front mark is described as being the same color as the position detection marks M1 and M2, it may be a different color. In the case of different colors, only the non-front marks Q1 to Q4 can be detected. If the colors are the same, the non-front mark is identified using the non-front marks Q1 to Q4 within a predetermined distance from the position detection marks M1 and M2.

  FIG. 21 is an example of a diagram schematically illustrating imaging of the hand-held printer 20 by the image data output unit 11. FIG. 22 is an example of a flowchart showing an imaging procedure of the image data output unit 11.

  The user adjusts the angle so that the image data output unit 11 does not capture the non-front marks Q1 to Q4 and the angle of view is reduced. During this time, the imaging unit 52 of the image data output unit 11 periodically captures an image (S10).

  The imaging unit 52 of the image data output unit 11 detects the non-front marks Q1 to Q4 similarly to the position detection marks M1 and M2 (S20).

  The imaging unit 52 determines whether only two marks have been imaged (S30). Since the position detection marks M1 and M2 are disposed on the front, they are always imaged. Therefore, when only two marks are imaged, it can be determined that the optical axis is in front of the print medium 12.

  When only some marks are imaged (when the non-front marks Q1 to Q4 can not be detected), the imaging unit 52 acquires the imaging data 14 (S40).

  Therefore, the position of the hand-held printer 20 can be calculated by performing enlargement / reduction as in the first embodiment and performing pattern matching with the image data 13 using the imaging data 14 imaged in this manner.

  Although it has been described that the non-front marks Q1 to Q4 have no thickness, the non-front marks Q1 to Q4 may have a thickness. For example, assuming that the thicknesses of the plurality of non-front marks Q1 to Q4 are the same, it can be confirmed that the optical axis is vertical when the thicknesses of the non-front marks Q1 to Q4 are the same in the imaging data 14 .

  Further, in the present embodiment, the imaging unit 52 automatically determines that the non-front surface marks Q1 to Q4 are not captured, but the user may determine and may image (the shutter button may be pressed). . However, the automatic imaging can suppress the deviation of the optical axis due to camera shake or the like.

  In this embodiment, in addition to the configuration of the third embodiment, an image forming system 1 in which the image data output unit 11 does not need to enlarge or reduce the imaging data 14 will be described. The functional block diagram of this embodiment may be the same as that of FIG. 6 of the first embodiment.

  As shown in FIG. 21, in the process in which the imaging unit 52 captures the hand-held printer 20, characters and the like on which an image has been formed are captured in the captured data 14. Therefore, the imaging unit 52 acquires characters around or near the position detection marks M1 and M2 from the imaging data 14, and searches for the same characters from the image data 13. Here, pattern matching or OCR may be performed. For example, when a character string “HAND” is detected from the imaging data 14, a character string “HAND” is searched from the image data 13. If the sizes of the two are the same, it is not necessary to enlarge or reduce the imaging data 14 after imaging.

  In the case of an image (image), a matching image 14a is acquired, and a matching image is detected by pattern matching. Then, imaging is performed so that the sizes of both are the same.

  FIG. 23 is an example of a flowchart of the image data output unit 11 at the time of imaging. In the description of FIG. 23, differences from FIG. 22 will be mainly described.

  When only two marks are imaged in step S30, the imaging unit 52 determines whether the sizes of the imaging data 14 and the image data 13 are the same (S35). Note that the imaging unit 52 outputs a voice such as "Please raise the camera" or "Please lower the camera" so that the size of the imaging data 14 and the image data 13 are the same, for example. It is preferable to induce.

  If the sizes become the same, the imaging unit 52 acquires the imaging data 14 (S40).

  Therefore, the image data output unit 11 of the present embodiment can omit enlargement / reduction in the process of calculating the position of the hand-held printer 20. In addition, since a matching image that matches the matching image 14a is specified during imaging, the user's waiting time can be shortened.

  In the present embodiment, the sizes of the character strings are compared, but the distance between the position detection marks M1 and M2 may be compared between the imaging data 14 and the image data 13. That is, using the relationship between the number of pixels and the length of the image formed as in the first embodiment, the distance (pixel) between the position detection marks M1 and M2 in the imaging data 14 is a predetermined value (that is, the image). The imaging data 14 is acquired when the number of pixels becomes 50 [mm] at the time of formation.

<Other application examples>
As described above, the best mode for carrying out the present invention has been described using the examples, but the present invention is not limited to these examples at all, and various modifications can be made without departing from the scope of the present invention. And substitution can be added.

  For example, although it has been described that the inquiry screen 1300 of FIG. 13 is displayed when a plurality of matching images 13a are detected, the inquiry screen 1300 may be displayed when the matching image 14a is detected. In this case, the user selects a matching image that matches the matching image 14a from the image data 13 in a state where there is no candidate. Further, the user may designate the image for comparison 14 a from the imaging data 14.

  Further, in the present embodiment, the position of the hand-held printer 20 is detected by the position detection marks M1 and M2. However, the handheld printer 20 may be recognized from the imaging data by the identification device that has learned the weights of the classifiers using a photograph in which the entire shape or pattern of the handheld printer 20 has been imaged as a teacher signal. Also in this case, since the range in which the hand-held printer 20 appears is known, the position of the navigation sensor 30 can be specified.

  For example, the above-described components of SoC 50 and ASIC / FPGA 40 may be included in either of them depending on CPU performance, circuit size of ASIC / FPGA 40, and the like.

  The functions of the hand-held printer 20 described with reference to FIGS. 4 and 5 may be realized by the CPU 31 executing a program.

  Further, in the present embodiment, a description is given using a handheld printer 20 which detects positions in three directions (for example, X / Y / R axes, R: rotation) and can freely scan on a plane (free hand scan) as an example. did. However, the present embodiment can also be applied to a hand-held printer 20 that detects the position in one direction (for example, the X-axis direction) and scans only in one direction to form an image. The present embodiment can also be applied to a hand-held printer 20 that detects positions in two directions (for example, X / Y axis directions) and scans only in two directions to form an image.

  Further, although it has been described in the present embodiment that ink is ejected to form an image, the image may be formed by irradiating visible light, ultraviolet light, infrared light, laser, or the like. In this case, as the print medium 12, for example, one that responds to heat or light is used. Also, a transparent liquid may be discharged. In this case, visible light is obtained when light in a specific wavelength range is irradiated. That is, it is only necessary to form visual information that can be finally confirmed with the naked eye through chemical and physical actions.

  Further, in the present embodiment, it has been described that the user scans freehand when forming an image, but the hand-held printer 20 may run on the print medium by using a motor or the like as power.

DESCRIPTION OF SYMBOLS 1 image forming system 11 image data output unit 12 printing medium 13 image data 14 imaging data 20 handheld printer 52 imaging unit 53 mark detection unit 54 pattern matching unit 55 position determination unit 56 matrix calculation unit

Patent No. 5033247

Claims (13)

An information processing apparatus capable of communicating with an image forming apparatus that forms an image based on image data while moving a position on a recording medium,
The image forming apparatus and an acquiring unit for acquiring imaging data obtained by imaging the formed image formed on the recording medium by the image forming apparatus;
An image processing unit that detects the image forming apparatus and the formed image from the imaging data;
In the image data used to form the formed image, a matching image that matches the formed image is specified, and the relative position of the image forming apparatus to the formed image in the imaging data is determined.
Position estimation means for estimating the position of the image forming apparatus in the image data based on the relative position based on the matching image in the image data ;
Transmission means for transmitting the position estimated by the position estimation means to the image forming apparatus;
An information processing apparatus having Wherein the position estimating means, the match by extracting the formed image that is reflected around the image forming apparatus performs pattern matching to the image data by using the formed image of the ambient of the image forming apparatus The information processing apparatus according to claim 1 , wherein the image is specified. The position estimating unit performs pattern matching to the image data by using the formed image including a character string, wherein the matching image including the same character string as the character string to claim 2 for identifying from the image data Information processing equipment. The position estimation means connects a set of lengths of a plurality of straight lines obtained by connecting vertices of a circumscribed rectangle of the character string included in the formed image and a vertex of the circumscribed rectangle of the character string included in the image data Compare the set of linear lengths obtained by
The information processing apparatus according to claim 2 , wherein the matching image in which a plurality of sets of straight line lengths match is specified from the image data. The information processing apparatus according to any one of claims 1 to 4, wherein the position estimation means causes the display device to display the formed image and the image data, and receives selection of the matched image corresponding to the formed image. The image processing means detects, from the imaging data, a first mark capable of estimating a position of the image forming apparatus disposed in the image forming apparatus and an attitude on a recording medium .
The position estimation unit according to any one of claims 1 to 5 , wherein the relative position of the first mark to the formed image in the imaging data is determined as the relative position of the image forming apparatus to the coincident image in the image data. The information processing apparatus described in the item. The image forming apparatus and an imaging unit configured to capture the formed image formed by the image forming apparatus;
A second mark is arranged on the image forming apparatus,
The image forming device and the image are triggered by the fact that the second mark is not imaged or the plurality of second marks are imaged with the same thickness while imaging the image forming apparatus periodically. The information processing apparatus according to any one of claims 1 to 6, wherein the formed image formed by the forming apparatus is imaged to generate the imaging data . The imaging means further in response to the size of the formed image in the imaging data, the size of the matching image to be consistent with the form image in the image data is almost the same,
The information processing apparatus according to claim 7 , wherein the formed image formed by the image forming apparatus and the image forming apparatus is captured. In the imaging data, the imaging unit further predetermines, in advance, length information obtained from a first mark which can estimate the position of the image forming apparatus disposed in the image forming apparatus and the attitude on the recording medium. 8. The information processing apparatus according to claim 7 , wherein the formed image formed by the image forming apparatus and the image forming apparatus is captured when the number of pixels reaches a determined number. It has correction information calculation means for calculating correction information for correcting the distortion of the formed image generated at the time of imaging,
Wherein the position estimating means, the claims distortion using the correction information calculated by the correction information calculating unit performs pattern matching to the image data by using the formed image corrected, identifying the matching image The information processing apparatus according to 2 . An image forming system comprising: an image forming apparatus for forming an image based on image data while moving a position on a recording medium; and an information processing apparatus capable of communicating with the image forming apparatus,
The image forming apparatus and an acquiring unit for acquiring imaging data obtained by imaging the formed image formed on the recording medium by the image forming apparatus;
An image processing unit that detects the image forming apparatus and the formed image from the imaging data;
In the image data used to form the formed image, a matching image that matches the formed image is specified, and the relative position of the image forming apparatus to the formed image in the imaging data is determined.
Position estimation means for estimating the position of the image forming apparatus in the image data based on the relative position based on the matching image in the image data ;
Transmission means for transmitting the position estimated by the position estimation means to the image forming apparatus;
An imaging system having: An image forming apparatus for forming an image based on image data while moving the position of a recording medium, comprising:
Position detection means for detecting the position relative to the recording medium;
The image forming apparatus and an acquisition unit for acquiring imaging data of the formed image formed on the recording medium by the image forming apparatus;
An image processing unit configured to detect the image forming apparatus and the image from the imaging data acquired by the acquisition unit;
In the image data used to form the formed image, a matching image that matches the formed image is specified, and the relative position of the image forming apparatus to the formed image in the imaging data is determined.
Position estimation means for estimating the position of the image forming apparatus in the image data based on the relative position based on the matching image in the image data ;
An image forming unit configured to form the formed image using the position estimated by the position estimating unit. An information processing apparatus capable of communicating with an image forming apparatus that forms an image based on image data while moving a position on a recording medium;
The image forming apparatus and an acquiring unit for acquiring imaging data obtained by imaging the formed image formed on the recording medium by the image forming apparatus;
An image processing unit that detects the image forming apparatus and the formed image from the imaging data;
In the image data used to form the formed image, a matching image that matches the formed image is specified, and the relative position of the image forming apparatus to the formed image in the imaging data is determined.
Position estimation means for estimating the position of the image forming apparatus in the image data based on the relative position based on the matching image in the image data ;
Transmission means for transmitting the position estimated by the position estimation means to the image forming apparatus;
Program to function as.