KR20140044644A - Terminal and method for forming video, apparatus for forming image, driving method thereof, and computer-readable recording medium - Google Patents

Abstract

The present invention relates to a terminal device, an image forming apparatus, a video generating method, a method for driving the image forming apparatus, and a computer-readable recording medium. The terminal device according to an embodiment of the present invention includes a location information sensing unit which obtains location information on the current location of the terminal device, a data processing unit which receives the location information on the image forming apparatus around the current location and calculates the relative location coordinate of the image forming apparatus by using the received location information and the location information on the current location, and an interface unit which overlays an icon on the image forming apparatus on an image obtained by an image pickup unit based on the calculated relative location coordinate and outputs the overlaid result. [Reference numerals] (AA) Communications network

Description

A terminal device, an image forming apparatus, an image generating method, a driving method of the image forming apparatus, and a computer readable recording medium {Terminal and Method for Forming Video, Apparatus for Forming Image, Driving Method Thereof, and Computer-Readable Recording Medium}

The present invention relates to a terminal apparatus, an image forming apparatus, an image generating method, a driving method of an image forming apparatus, and a computer readable recording medium, and more particularly, for example, the position and state information of an image forming apparatus, for which mobility is guaranteed. A terminal apparatus, an image forming apparatus, an image generating method, a driving method of an image forming apparatus, and a computer-readable recording medium, which match and show an image photographed using the terminal apparatus.

The craze of smartphones and tablet PCs, which have swept the world for several years, is changing consumption patterns and lifestyles. Nowadays, IT devices are intimately connected with our daily lives. This is because new technology that has been seen only in movies has been made easy for anyone to use. One of the representative new technologies is 'augmented reality'.

Augmented reality is a very complicated and difficult imaging technology internally, but basically it works with the following principles and procedures. In order to apply augmented reality technology, several things are required: GPS device and gravity sensor for transmitting and receiving geographic and location information, a location information system storing detailed information according to this information, and augmented reality for receiving the detailed information and displaying it on the reality background Application, and finally the IT device that will output it to the display.

First, when a user launches augmented reality application and shines a specific distance or building with a built-in camera such as a smartphone, the GPS receiver can record the latitude and longitude information, tilt and gravity information of the current location temporarily on the smartphone. The GPS information is then sent to a specific location information system via the Internet. This is because it is practically impossible to store all the detailed information of the area or building of the location radius on the smartphone.

The location information system, which receives the GPS information such as the position and the tilt from the user, searches for detailed information of the corresponding area or object in its database and transmits the result back to the smartphone. This includes, of course, the name of the particular building, the telephone number, and so on. Smartphones receiving this data are matched with current map information through augmented reality applications and then displayed in real time. Since the above data transmission and reception phase is continuously maintained and performed, detailed information about the region and the surroundings is sequentially displayed on the screen when the smartphone is carried over the street.

However, the conventional AR application technology simply displays static information and has a disadvantage in that the object is fixed. That is, things are not updated by themselves, and there is a hassle that an administrator can provide the information only by separately inputting data about the thing.

An embodiment of the present invention provides a terminal apparatus, an image forming apparatus, an image generating method, a driving method of an image forming apparatus, for example, displaying and matching the position and state information of an image forming apparatus, which is guaranteed to be mobile, on an image photographed using a terminal apparatus, And a computer readable recording medium.

The terminal device according to an embodiment of the present invention receives a location information sensing unit for obtaining location information of its current location, location information of an image forming apparatus around the current location, and receives the received location. A data processor for calculating the relative position coordinates of the image forming apparatus using information and the position information of the current position, and an icon for the image forming apparatus based on the coordinate values of the relative position coordinates calculated on the acquired image of the imaging unit. It includes an interface unit for outputting the overlay.

The terminal apparatus may further include a searching unit for searching the image forming apparatus, and the data processing unit may receive position information of the searched image forming apparatus.

The terminal device further includes a direction information sensing unit for obtaining direction information of the terminal device, wherein the interface unit overlays the icon by additionally using the direction information when overlaying the icon.

The data processing unit may receive location information stored in the image forming apparatus or receive location information provided upon connection to an external service device storing the location information of the image forming apparatus.

The interface unit may output the state information of the image forming apparatus by including the icon in the icon.

The image forming apparatus according to the embodiment of the present invention is configured to overlay and display an icon for the image forming apparatus on a position information sensing unit for acquiring position information on its current position, and an image obtained by the image capturing unit by the terminal apparatus. When requesting the location information, the control unit for controlling the location information sensing unit to provide the other location information in the request.

The image forming apparatus may further include a state information manager which stores state information of the image forming apparatus and provides the state information when requested by the terminal device.

The state information management unit stores at least one of toner remaining amount, error information, and operation state as state information of the image forming apparatus.

The control unit periodically checks the state of the image forming apparatus and updates the state information stored in the state information management unit.

The location information sensing unit may include a GPS module for performing communication with a GPS, and periodically checks the changing location information of the image forming apparatus through the GPS module.

The location information sensing unit includes a short range communication module for communicating with neighboring access points (APs), and triangulates the APs using the location and signal strength of the APs acquired through the short range communication module. Characterized in that said position information is acquired in a manner.

In an image generating method according to an embodiment of the present invention, in the image generating method of a terminal device, acquiring position information on a current position of the terminal device, position information of an image forming apparatus located around the terminal device; Calculating a relative position coordinate of the image forming apparatus using the position information of the image forming apparatus and the position information of the terminal apparatus, acquiring an image by an image capturing unit of the terminal apparatus, and And overlaying the icon for the image forming apparatus on the acquired image based on the coordinate values of the calculated relative position coordinates.

The image generating method may further include searching the image forming apparatus, and the calculating of the relative position coordinates may include receiving position information of the searched image forming apparatus.

The image generating method may further include acquiring direction information of the terminal device, and the overlaying and outputting the icon may overlay the icon by additionally using the direction information when the icon is overlaid. It is done.

The calculating of the relative position coordinates may include receiving position information stored in the image forming apparatus or accessing an external service apparatus storing position information of the image forming apparatus to receive position information. It is characterized by.

The overlaying and outputting the icon may include outputting state information of the image forming apparatus in the icon.

According to an embodiment of the present invention, a method of driving an image forming apparatus includes: acquiring position information of a current position by a position information sensing unit; and overlaying an icon for the image forming apparatus on an image acquired by an image capturing unit by a terminal apparatus; If the location information is requested for display, the step of controlling the location information sensing unit to provide the location information according to the request.

And storing the state information of the image forming apparatus by the state information storing unit and providing the stored state information at the request of the terminal device.

The state information storage unit stores at least one of toner remaining amount, error information, and operation state as state information of the image forming apparatus.

The control unit periodically checks the state of the image forming apparatus and updates the state information stored in the state information storage unit.

The location information sensing unit communicates with a GPS using a GPS module, and periodically checks the changing location information of the image forming apparatus through the GPS module.

The location information sensing unit communicates with neighboring access points (APs) using a short range communication module, and triangulates the APs using the location and signal strength of the APs acquired through the short range communication module. Characterized in that the position information is acquired.

A computer-readable recording medium according to an embodiment of the present invention includes a program for executing an image generating method of a terminal device, wherein the image generating method is a position with respect to a current position of the terminal device. Acquiring information, receiving position information on an image forming apparatus provided around the terminal apparatus, and using the position information of the image forming apparatus and the position information of the terminal apparatus, relative position coordinates of the image forming apparatus; Calculating an image, acquiring an image using the terminal device, and overlaying and outputting an icon for the image forming apparatus on the acquired image based on the calculated coordinate values of the relative position coordinates.

1 is a view showing an image generating system according to an embodiment of the present invention;
2 is a view for explaining a UI displayed on the terminal device of FIG. 1;
3 is a diagram illustrating a structure of a terminal device of FIG. 1;
4 is a diagram illustrating the structure of the image forming apparatus of FIG. 1;
5 is a diagram illustrating a structure of a service device of FIG. 1;
6 is a view showing an image generating process according to an embodiment of the present invention;
FIG. 7 is a diagram for describing a data flow when the service device of FIG. 1 is used;
8 is a view for explaining an operation process between a terminal device and a service device;
9 is a view for explaining an operation process between a terminal apparatus and an image forming apparatus;
FIG. 10 is a view for explaining a data flow when the service apparatus of FIG. 1 is not used; FIG.
11 is a flowchart illustrating an image generating method according to an embodiment of the present invention;
12 is a flowchart illustrating an image generating method according to another embodiment of the present invention;
13 is a flowchart illustrating a method of driving the image forming apparatus of FIG. 1;
14 is a flowchart illustrating a method of driving a service device of FIG. 1.
FIG. 15 is a diagram illustrating a screen implemented in the terminal device of FIG. 1.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

Before describing in detail, a terminal device according to an embodiment of the present invention may be referred to as a mobile device, and an image forming apparatus may be described assuming a printer and a service device as a printer server, but are not limited thereto.

1 is a diagram illustrating an image generating system according to an exemplary embodiment of the present invention.

As shown in FIG. 1, an image generating system according to an exemplary embodiment of the present invention may be a part of a terminal apparatus 100, an image forming apparatus 110, a communication network 120, a service apparatus 130, and a navigation apparatus 140. Or all of them.

Here, the inclusion of some or all means that some components such as the service device 130 or the navigation device 140 are omitted so that the system may be configured, or some components may be integrated with other components. As it is, in order to help full understanding of the invention, it will be described as including all.

The terminal device 100 includes a mobile communication terminal such as a smartphone, a laptop, a tablet PC, a desktop, a personal digital assistant (PDA), and the like. The terminal device 100 may include, for example, a GPS module for communicating with the navigation device 140 and a short range communication module for performing short range communication such as Wi-Fi. Through this, the location of the current location of the user may be calculated, the calculated location information may be stored in the internal memory, or transmitted to the service device 130. In addition, the terminal device 100 may include a camera, a position sensor, and a directional measurement sensor. The location information acquired through this may be transmitted to the service device 130, and may be provided to the user by performing a data processing process of receiving image formation information from the service device 130 and mapping it to a UI screen.

For example, the terminal device 100 according to an embodiment of the present invention implements an image in which an icon for the image forming apparatus 110 is overlaid on an image acquired through an image capturing unit, for example, a camera. In other words, when the camera is operated to illuminate the surroundings, an icon corresponding to the image forming apparatus 110 recognized in the surroundings is displayed on the screen of the terminal device 100. In this case, the overlaid icon may also display the state information of the image forming apparatus 110. The status information here means toner remaining amount, operation status, and the like. In order to implement such an image, a user of the terminal device 100 may be pre-stored and shipped at the time of manufacture, or may download and install a mobile application (hereinafter, an app) installed from the service device 130.

When executing such an app, the terminal device 100 directly transfers the position information of the image forming apparatus 110 directly from the image forming apparatus 110 in order to overlay an icon on the camera image in the correct position of the image forming apparatus 110. It may be provided, or may be provided from the service device 130 that stores the location information of the image forming apparatus 110. The same applies to the state information of the image forming apparatus 110, of course. This can be set by the system designer or set by the user or administrator of the image forming apparatus 110, which may be referred to as mode setting in an embodiment of the present invention.

The process of obtaining location information by the terminal device 100 is as follows. When the user executes the installed app, the terminal device 100 obtains its own location information. When the current location of the terminal device 100 is transmitted to the service device 130, the service device 130 searches for the image forming device 110 that is close to the user's terminal device 100, and displays the corresponding image forming devices 110. The location information is transmitted to the terminal device 100. In the app that receives and executes the location information of the image forming apparatus 110, the virtual coordinates of the surrounding image forming apparatuses 110 are completed by using the location of the terminal apparatus 100 as a reference point, and further, each image forming apparatus ( 110 may request status information. In this case, if authentication is required, the service device 130 may send only a list of the image forming apparatus 110 authorized to the user as the user information of the app. After receiving the state information of the image forming apparatus 110, the terminal device 100 receives a camera image, and extracts camera angle of view information of the user, location information of the current terminal device 100, and up, down, left, and right directions. Displays an icon and a status information window of the image forming apparatus 110 displayed according to the direction indicated by the terminal device 100 on the screen. In this case, the icon image and the status information window of the image forming apparatus 110 are displayed on the real image by overlaying it on the camera image.

If there is no service device 130, when the user executes the app, the entire list and location information of the image forming apparatus 110 in the same network as the terminal device 100 may be received. For example, one image forming apparatus 110 that acts as a group owner may generate a list of the surrounding image forming apparatus 110 and provide the list to the terminal device 100. The terminal device 100 selects the image forming apparatuses 110 around the terminal apparatus 100 and requests the state information of the corresponding image forming apparatuses 110. After receiving the state information of the image forming apparatus 110, the terminal device 100 receives a camera image, and extracts camera angle of view information of the user, location information of the current terminal device 100, and up, down, left, and right directions. Thereafter, icons and status information windows of the image forming apparatuses 110 displayed in the direction indicated by the terminal device 100 are displayed on the screen. In this case, the icon image and the status information window of the image forming apparatus 110 are displayed on the real image by overlaying it on the camera image.

The terminal device 100 collects information such as an operation state of the image forming apparatus 110, abnormality information, remaining amount of toner, and the like from each module of the set itself. To this end, the terminal device 100 may include a separate search unit. Here, the types of information collected in the set itself are shown in Table 1.

Location information

Status information
Set Information
Latitude, longitude, altitude

Toner level, error information (No paper, Jam, Network error, Device error etc), operation status (in use, waiting)

Set ID, IP Address, MAC, Model Name, Spec (capacity), etc.

More specifically, when the terminal device 100 interworks with the service device 130, various state information of the image forming device 110 is transmitted from the image forming device 110 to the service device 130 in real time. That is, the image forming apparatus 110 periodically transmits the latitude, longitude, altitude, set ID, remaining amount of toner, status, and IP information of the set itself to the service apparatus 130. On the other hand, when not interworking with the service device 130, the terminal device 100 performs discovery while the printing app is executed, and requests the state information from the retrieved image forming apparatus 110, respectively. Then, the image forming apparatus 110 transmits the latitude, longitude, altitude, set ID, remaining amount of toner, status, and IP information of the set itself when the terminal apparatus 100 requests it.

As a result, the state information of the image forming apparatus 110 is displayed on the UI of the terminal apparatus 100 as shown in Fig. 2. In this case, the terminal apparatus 100 is a corresponding image forming apparatus according to the security level of the app user. It is also possible to indicate whether the user has permission to use 110. For example, the reference point is set based on the current position (latitude, longitude, altitude) of the terminal device 100, and then the reference point of the image forming apparatus 110 is set. The section of the currently displayed screen is determined by obtaining angles of view and vertical angles of view, and the positions of the image forming apparatus 110 are displayed thereon, and are entered within a radius according to the distance between the terminal apparatus 100 and the image forming apparatus 110. Only the information of the image forming apparatus 110 is displayed on the screen. The left and right coordinates of the image forming apparatus 110 determine the direction by information extracted from the electronic compass of the terminal apparatus 100, and the vertical coordinates of the image forming apparatus 110 are determined by the terminal apparatus 100. Gyros As information that is extracted from the loop sensor (hereinafter referred to as a gyro sensor) is determined by the slope of the terminal apparatus 100.

When the user clicks the status information box displayed on the UI, the terminal device 100 switches to a screen for printing, copying, scanning, and sending a fax. When the user presses a print button, the command is directly transmitted to the image forming apparatus 110, thereby easily printing the contents in the terminal device 100. Of course, you can also perform other scan, copy, and fax functions.

The image forming apparatus 110 includes, for example, a printer, a copier, a facsimile, a scanner, or a multifunction printer that implements a combination of these functions through one device. The image forming apparatus 110 may operate in two modes. Here, the two mode methods are related to a path for providing the terminal device 100 with its own location information, and further, the status information so that the icon of the image forming apparatus 110 is displayed at the correct location on the image acquired by the camera, for example. There is this.

For example, suppose that the administrator of the image forming apparatus 110 sets the mode to the service apparatus 130. In this case, the image forming apparatus 110 periodically transmits its location information, and further, status information, to the service apparatus 130 in cooperation with the service apparatus 130. Then, the service device 130 stores the position information and the state information of the image forming apparatus 110 transmitted periodically and provides them to the terminal device 100. Here, the reason for the periodic transmission is to accurately inform the user of the terminal apparatus 100 because, for example, the positional change of the image forming apparatus 110 may occur or the remaining amount of the toner may change at any time.

On the other hand, suppose that the administrator of the image forming apparatus 110 sets the information providing path to the terminal device 100. In this case, the image forming apparatus 110 performs a process for obtaining its own location information. The image forming apparatus 110 may obtain location information using approximately three methods according to a spec, that is, a specification and an environment of an installation place. The first method is to use the navigation device 140. A real time kinematic (RTK) method such as a Virtual Reference Station (VRS) and a Flat Korrect Parameter (FKP) may be used to receive location information and to increase accuracy using a GPS module built in the image forming apparatus 110. The image forming apparatus 110 incorporating the GPS module periodically receives its location information and stores the longitude, latitude, and altitude values therein. Secondly, location information is acquired using an indoor access point (AP) or indoor location information using a separate AP and triangulation of a Wi-Fi signal. The image forming apparatus 110 in which the Wi-Fi module is embedded calculates the position of the image forming apparatus 110 itself based on the position and signal strength of the neighboring APs and periodically stores the longitude, latitude, and altitude values therein. Third, there is no separate position receiving sensor, in which case the installer manually inputs the position information to the image forming apparatus 110 at a fixed position. The location information is measured and set in the image forming apparatus 110 by using a mobile phone of an installation manager or a separate measuring device. The administrator can find his current location using his mobile phone and input the longitude, latitude, and altitude values of the current location into the image forming apparatus 110.

In this way, the acquired position information of the image forming apparatus 110, and furthermore, the state information may be periodically transmitted to the service apparatus 130 or may be directly provided to the terminal apparatus 100 according to the setting of the administrator. In this regard, since the terminal device 100 has been described above, further description thereof will be omitted.

The communication network 120 is meant to include both wired and wireless communication networks. Here, the wired network includes an Internet network such as a cable network or a public switched telephone network (PSTN), and the wireless communication network includes means such as CDMA, WCDMA, GSM, Evolved Packet Core (EPC), Long Term Evolution (LTE) to be. Therefore, when the communication network 120 is a wired communication network, the terminal device 100 may access an exchange or the like of a telephone company, but in the case of a wireless communication network, the terminal device 100 may access an SGSN or a Gateway GPRS Support Node (GGSN) operated by a communication company, or may process data. Data can be processed by connecting to various repeaters such as BTS (Base Station Transmission), NodeB, and e-NodeB. That is, the printing app of the terminal device 100 and additional information related thereto are processed.

For example, when the terminal device 100 operates in dependence on the service device 130, when the terminal device 100 transmits the location information to the service device 130, the service device 130 moves to the corresponding location. Based on the information, the image forming apparatus 110 is searched for the terminal forming apparatus 100, and the position information of the found image forming apparatus 110 is transmitted to the terminal apparatus 100. The app of the terminal device 100 receiving the location information completes the virtual coordinates of the peripheral image forming apparatuses 110 based on the position of the terminal device 100, and then requests status information of each image forming apparatus 110. Then, the service device 130 provides status information to the terminal device 100 so that the terminal device 100 displays the icon and the status information of the image forming apparatus 110 on the image acquired by the camera.

The navigation device 140 may communicate with the terminal device 100 and the image forming device 110, for example, as a GPS. Further, if the navigation device 140 includes a GPS module, the navigation device 140 may also communicate with the service device 130. If the terminal device 100 and the image forming apparatus 110 include a GPS module therein and want to acquire their own location information using the GPS module, the terminal device 100 and the image forming apparatus 110 communicate with the navigation device 140 using the internal GPS module. You can find out where you are and save information about it.

3 is a diagram illustrating the structure of the terminal device of FIG.

Referring to FIG. 3 together with FIG. 1, the terminal device 100 according to an exemplary embodiment of the present invention includes an imaging unit 300, a location information sensing unit 310, a direction information sensing unit 320, and a data processing unit 330. And a part or all of the interface unit 340, where the data processor 330 may be used to mean a controller (not shown).

Here, the inclusion of some or all means that some components may be omitted or integrated into other components, and will be described as including all to aid a sufficient understanding of the present invention.

The imaging unit 300 includes a camera. The image acquired through the camera may be provided to the data processor 330.

The location information sensing unit 310 may include a location sensor or may include a GPS module, a Wi-Fi module, etc., so as to acquire its location information using the navigation device 140 or an AP nearby. Alternatively, the device information may be transmitted to the service device 130 to acquire location information.

The direction information sensing unit 320 includes a gyro sensor or the like. As a result, the image sensing unit 300 of the terminal device 100 detects the direction in which the image is reflected, and the icon for the image forming apparatus 110 positioned within the angle of view is overlaid.

The data processor 330 may include a controller. The data processing unit 330 is at least one of the position information and the state information of the image forming apparatus 110 directly from the image forming apparatus 110 via the communication network 120 or indirectly through the service apparatus 130. Receives the information of the and may perform a role of mapping to the image taken by the imaging unit 300. The mapped image may be displayed on the interface unit 340, for example, the display unit.

The interface unit 340 includes a communication interface unit and a user interface unit. Here, the communication interface unit processes a signal or data processed by the data processor 330 and transmits the signal or data to the communication network 120. On the other hand, the user interface unit may include a button unit or a display unit. The button unit may receive a user's command. The display unit may display a button for receiving a user command or may overlay and display an icon and a status information window of the image forming apparatus 110 on an image projected by the imaging unit 300.

4 is a diagram illustrating the structure of the image forming apparatus of FIG. 1.

Referring to FIG. 4 together with FIG. 1, an image forming apparatus 110 according to an exemplary embodiment of the present invention may include a location information sensing unit 400, a state information managing unit 410, an image forming unit 420, 430, and 440. It may include some or all of the control unit 450. Herein, the inclusion of some or all of the above description is omitted.

The location information sensing unit 400 includes a location sensor capable of receiving location information using GPS, location information based on Wi-Fi, or location information manually input by an installer. In this way, the image forming apparatus 110 may acquire position information.

In addition, the state information management unit 410 may perform an operation for acquiring state information of the image forming apparatus 110 itself, for example, the state of the apparatus, the network state, the remaining amount of toner, or the like, and store the acquired information. The state information manager 410 may periodically acquire and store state information.

The image forming units 420, 430, and 440 may include, for example, a printing unit 420, a scanning unit 430, and a fax unit 440. The printing unit 420 prints the input data on the output, the scanning unit 420 copies the input image data, and the fax unit 440 performs a function of transmitting the input data to an external device. This is, for example, a configuration of a multifunction apparatus, and the image forming apparatus 110 according to an embodiment of the present invention includes at least one of them, which will specifically perform an operation when an icon displayed on the terminal device 100 is selected. .

The controller 450 is configured to operate when there is a user's control command, and controls the overall operation of the image forming apparatus 110. In other words, the operations of the location information sensing unit 400 and the state information management unit 410 are controlled. For example, when there is an icon selection from the user, the controller 450 may perform an operation such as printing, scanning, or faxing the image forming apparatus 110 corresponding to the selected icon.

5 is a diagram illustrating the structure of a service device of FIG.

Referring to FIG. 5 together with FIG. 1, the service device 130 according to an exemplary embodiment of the present invention may include some or all of the search unit 500, the receiver 510, the database unit 520, and the transmitter 530. It may further include a separate control unit. Here, the transmitter 530 may include a role of a controller in charge of data processing, or the searcher 500 may include a role of a controller. In addition, the receiver 510 may be referred to as a state information manager.

The search unit 500 receives location information of the terminal device 100 and periodically searches for the image forming apparatus 110 positioned around the terminal device 100 based on the received location information. . For example, the search unit 500 searches for the image forming apparatuses 110 located in close proximity based on latitude, longitude, and altitude values, and transmits the current state of the image forming apparatuses 110 to the terminal apparatus 100. . The method of finding the adjacent image forming apparatus 110 is as follows. Assuming that the proximity distance is 10 meters based on the latitude, longitude, and altitude of the terminal device 100, an image in which the difference in the linear distances considering the latitude, longitude, and altitude values of each image forming apparatus 110 is within 10 meters. The forming apparatus 110 is searched. Therefore, the search unit 500 may include a separate comparison unit.

The receiver 510 requests the state information under the control of the image forming apparatus 110, more precisely under the control of the controller, and receives the requested state information.

The database unit 520 may match and store various pieces of information such as device information, location information, and state information corresponding to the device information of the image forming apparatus 110. In response to a request from the searcher 500 and the receiver 510, corresponding information may be output.

The transmitter 530 transmits the result retrieved by the searcher 500 to the terminal device 100, or further, provides the terminal device with status information provided by the receiver 510 or after searching the database unit 520. 100 may be provided.

6 is a diagram illustrating an image generation process according to an exemplary embodiment of the present invention, and FIG. 7 is a diagram schematically illustrating the process of FIG. 6.

6 and 7 together with FIG. 1, the image forming apparatus 110 acquires its own location information and state information and transmits it to the service device 130 (S600 and S610). To this end, the image forming apparatus 110 communicates with the GPS satellites, or performs short-range communication with nearby APs to acquire location information using a triangular measurement method using signal strength, etc., and the controller checks each set to provide status information. Can be obtained.

Subsequently, the terminal device 100 acquires its own location information and transmits it to the service device 130 (S620), and the service device 130 searches for the surrounding image forming device 110 based on the corresponding location information, for example. The location information about them may be provided or list information on the location information may be transmitted (S630 and S640).

Then, the terminal device 100 calculates a relative position based on itself using the position information of the image forming apparatus 110 (S650).

In operation S660, the terminal device 100 displays an icon and status information of the image forming apparatus 110 at the position of the relative coordinate calculated from the image acquired through the camera.

In addition, the terminal device 100 additionally performs various functions such as switching to a screen for printing, copying, scanning, and sending a fax by clicking a status information box on the UI (S670, S680).

In this process, as shown in FIG. 7, when the terminal apparatus 100 requests information about the image forming apparatus 110, the service apparatus 130 may provide the service apparatus 130 after performing a separate authentication procedure. The embodiment of the present invention will not be particularly limited to such.

FIG. 8 is a diagram for describing an operation process between a terminal device and a service device. The terminal device is a mobile device such as a mobile phone, and the image forming device is a printer.

Referring to FIG. 8 together with FIG. 1, the terminal device 100 may execute what is called a mobile app in order to implement AR technology for the surrounding image forming apparatus 110 (S801).

Accordingly, the terminal device 100 senses the location information of the mobile device, for example, transmits the sensed location information to the service device 130 (S805), and then requests the location information of the image forming apparatus 110, that is, the printer. (S807).

Then, the service device 130 searches for a nearby printer based on the location information of the mobile device, and provides the result to the terminal device 100 (S809 and S811).

Accordingly, the terminal device 100 completes the relative position of the printer based on its position (S813).

In addition, the terminal device 100 requests the state information on the printer from the service device 130 again, and receives the requested state information (S815 and S817).

Subsequently, when the terminal device 100 acquires an image through the camera, the terminal device 100 maps and displays the location and state information of the printer on the acquired image (S819 and S821).

FIG. 9 is a view for explaining an operation process between a terminal apparatus and an image forming apparatus. The terminal apparatus is a mobile apparatus such as a mobile phone, and the image forming apparatus is a printer. FIG. 10 is a service apparatus of FIG. Is a diagram for explaining the data flow when not used.

9 and 10 together with FIG. 1, for example, when the terminal device 100 does not interwork with a separate service device 130 to implement AR technology, the terminal device 100 executes a mobile app. Communication with the image forming apparatus 110 that can operate as a neighboring group owner can be performed (S901 and S903).

In this manner, the terminal device 100 may receive location information or list information about the surrounding image forming apparatuses 110 from the image forming apparatus 110 (S905).

 Subsequently, the terminal device 100, which is a mobile device, acquires location information by sensing its location, and selects adjacent printers based on the acquired location information (S907 and S909).

In this case, for example, if the printer is set to select printers within 10 meters, relative position coordinates are completed only for printers within 10 meters (S911).

Thereafter, the terminal device 100 receives the requested state information by requesting the state information of the adjacent printers (S913 and S915).

In addition, the terminal device 100 maps and displays the icon and the status information on the printer on the image acquired through the camera (S917 and S919).

Of course, even in such a step, the user of the terminal device 100 may further select a desired printer as shown in FIG. 10 to further perform functions such as copying and fax scanning.

11 is a flowchart illustrating an image generating method according to an exemplary embodiment of the present invention.

Referring to FIG. 11 together with FIG. 1, the terminal device 100 according to an embodiment of the present invention acquires location information about its current location (S1100). To this end, when the terminal device 100 executes a so-called mobile app, the terminal device 100 may acquire location information by operating a location sensor. Since this has been described in detail in the foregoing, further explanation is omitted.

Subsequently, the terminal apparatus 100 sets relative position coordinates of the image forming apparatus 110 based on the terminal apparatus 100 using the positional information of the image forming apparatus 110 and the positional information of the terminal apparatus 100. (S1120). That is, the relative distance can be calculated.

Thereafter, the terminal device 100 overlays and outputs an icon for the image forming apparatus 110 on the image acquired by the imaging unit, for example, the camera, based on the coordinate value, that is, the distance.

12 is a flowchart illustrating a method of generating an image according to another exemplary embodiment of the present invention.

Referring to FIG. 12 together with FIG. 1, as a mobile device, the terminal device 100 acquires the location information of the printer from a peripheral printer or a separate server, and also acquires its own location information (S1201 and S1203).

The relative position coordinates of the printers based on the position of the terminal apparatus 100 are set (or calculated) using the acquired two position information (S1205).

Thereafter, the terminal device 100 acquires an image through a camera and overlays the icon on the acquired image based on the printer position and displays the image on the screen (S1207 and S1209).

In this process, when the terminal device 100 also acquires direction information, the terminal device 100 displays the location of the printer on the screen according to the direction (S1211 and S1213).

The terminal device 100 acquires the printer status information, updates the status information periodically, and completes and displays the final UI for the user (S1215 and S1217). Here, as described above, the state information may be obtained from a printer when directly communicating with a nearby printer, and may be obtained from a server when interworking with a separate server.

13 is a flowchart illustrating a method of driving the image forming apparatus of FIG. 1;

Referring to FIG. 13 together with FIG. 1, when the image forming apparatus 110 according to an embodiment of the present invention interworks with each other by executing a mobile app in the terminal device 100, location information on its current location. A process of acquiring the data may be performed (S1300).

Subsequently, when the location information is requested from the terminal device 100 directly or bypassed from the separate service device 130, the acquired location information may be provided to the terminal device 100 (S1310). This allows the terminal device 100 to display an icon for the image forming apparatus 110 on the image acquired by the camera.

In this process, when the image forming apparatus 100 requests status information directly from the terminal apparatus 100 or bypasses the service apparatus 130, the image forming apparatus 100 also provides the corresponding status information.

14 is a flowchart illustrating a method of driving a service device of FIG. 1.

Referring to FIG. 14 together with FIG. 1, the service device 130 according to an embodiment of the present invention receives the location information provided by the terminal device 100 and based on the received location information of the terminal device 100. The image forming apparatuses 110 located at the periphery are periodically searched (S1400). For example, since the image forming apparatus 110 is mobile, retrieval may be necessary at regular intervals.

Subsequently, when the terminal device 100 requests location information of the image forming apparatus 110, the service device 130 may provide the retrieved location information.

Of course, the service device 130 may perform various operations. For example, the relative coordinate value of the image forming apparatus 110 is calculated based on the position information of the terminal device 100, and the image is generated so that an icon of the image forming apparatus 110 appears by mapping to the calculated distance. The terminal device 100 may be provided to the terminal device 100, and the terminal device 100 may perform only an operation of receiving and displaying a corresponding image. Therefore, embodiments of the present invention will not be specifically limited to the above.

FIG. 15 is a diagram illustrating a screen implemented in the terminal device of FIG. 1.

Referring to FIG. 15 together with FIG. 1, when a user of the terminal apparatus 100 drives a camera to illuminate the surroundings, a certain range of images is acquired, and an icon for the image forming apparatus 110 within that range is obtained. Overlays and appears on the screen.

In this case, the terminal device 100 displays the icon and the state information of the image forming apparatus 110, for example, the remaining amount of the toner or the operation state.

And when the user clicks the icon, it performs additional operations such as printing, faxing, and scanning.

On the other hand, even if all the components constituting the embodiment of the present invention is described as being combined or operated in combination, the present invention is not necessarily limited to these embodiments. That is, within the scope of the present invention, all of the components may be selectively coupled to one or more of them. In addition, although all of the components may be implemented in one independent hardware, each or some of the components of the components are selectively combined to perform some or all functions combined in one or a plurality of hardware. It may be implemented as a computer program having a. The codes and code segments constituting the computer program may be easily deduced by those skilled in the art. Such a computer program may be stored in a computer readable media and read and executed by a computer, thereby implementing embodiments of the present invention. As the storage medium of the computer program, a magnetic recording medium, an optical recording medium, a carrier wave medium, or the like may be included.

While the invention has been described in connection with what is presently considered to be practical exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present invention.

100: terminal device 110: image forming apparatus
120: communication network 130: service device
140: navigation device 300: imaging unit
310, 400: location information sensing unit 320: direction information sensing unit
330: data processing unit 340: interface unit
410: state information management unit 450: control unit
500: search unit 510: receiver
520: database unit 530: transmitter

Claims (23)

A location information sensing unit for obtaining location information on its current location;
A data processor configured to receive positional information on the surrounding image forming apparatus based on the current position, and calculate relative position coordinates of the image forming apparatus using the received positional information and the positional information of the current position; And
An interface unit for overlaying and outputting an icon for the image forming apparatus based on the calculated coordinate values of the relative position coordinates on the acquired image of the imaging unit;
Terminal device. The method of claim 1,
A searcher for searching the image forming apparatus;
And the data processing unit receives position information of the searched image forming apparatus. The method of claim 1,
Further comprising: a direction information sensing unit for obtaining direction information of the terminal device,
And the interface unit overlays the icon by additionally using the direction information when overlaying the icon. The method of claim 1,
The data processing unit receives location information stored in the image forming apparatus or receives terminal location information provided when connecting to an external service device storing the location information of the image forming apparatus. . The method of claim 1,
And the interface unit outputs the state information of the image forming apparatus by including the icon in the icon. A location information sensing unit for obtaining location information on its current location; And
A control unit for controlling the location information sensing unit to provide the other location information to the request when the terminal device requests the location information to display an icon for the image forming apparatus overlaid on the image acquired by the imaging unit;
The image forming apparatus comprising: The method according to claim 6,
And a state information management unit which stores state information of the image forming apparatus and provides the state information when requested by the terminal device. 8. The method of claim 7,
And the state information management unit stores at least one of toner remaining amount, error information, and operation state as state information of the image forming apparatus. 8. The method of claim 7,
And the control unit periodically checks the state of the image forming apparatus and updates the state information stored in the state information managing unit. The method according to claim 6,
The location information sensing unit includes a GPS module for communicating with a satellite navigation device (GPS),
And periodically checking the changing position information of the image forming apparatus through the GPS module. The method according to claim 6,
The location information sensing unit includes a short range communication module for communicating with neighboring access points (AP),
And obtaining the position information by triangulating the APs using the positions and signal strengths of the APs acquired through the short range communication module. In the image generating method of the terminal device,
Acquiring location information on the current location of the terminal device;
Receiving position information of an image forming apparatus positioned around the terminal apparatus, and calculating relative position coordinates of the image forming apparatus using the position information of the image forming apparatus and the position information of the terminal apparatus;
Acquiring an image by an image capturing unit of the terminal device; And
Overlaying and outputting an icon for the image forming apparatus on the obtained image based on the coordinate values of the calculated relative position coordinates;
Image generation method comprising. The method of claim 12,
Searching for the image forming apparatus;
The calculating of the relative position coordinates may include receiving position information of the searched image forming apparatus. The method of claim 12,
Acquiring direction information of the terminal device;
The overlaying and outputting the icon may include overlaying the icon using the direction information when overlaying the icon. The method of claim 12,
The calculating of the relative position coordinates may include receiving position information stored in the image forming apparatus or accessing an external service apparatus storing position information of the image forming apparatus to receive position information. Image generating method characterized in that. The method of claim 12,
The outputting of the icon by overlaying the image may include outputting the state information of the image forming apparatus in the icon. Acquiring, by the location information sensing unit, location information of its current location; And
Controlling the location information sensing unit to provide the location information according to the request when the terminal device requests the location information to display an icon for the image forming apparatus overlaid on the image acquired by the imaging unit;
And a driving method of the image forming apparatus. 18. The method of claim 17,
And a state information storing unit storing state information of the image forming apparatus, and providing the stored state information at the request of the terminal device. 19. The method of claim 18,
And the state information storage section stores at least one of toner remaining amount, error information, and operation state as state information of the image forming apparatus. 19. The method of claim 18,
And the control unit periodically checks the state of the image forming apparatus and updates the state information stored in the state information storage unit. 18. The method of claim 17,
The location information sensing unit communicates with a GPS using a GPS module.
And periodically checking the changing position information of the image forming apparatus through the GPS module. 18. The method of claim 17,
The location information sensing unit communicates with nearby access points (APs) using a short range communication module,
And obtaining the position information by triangulating the APs using the positions and signal strengths of the APs acquired through the short range communication module. A computer readable recording medium comprising a program for executing an image generating method of a terminal device.
The image generation method includes:
Acquiring location information on the current location of the terminal device;
Receiving position information of an image forming apparatus provided around the terminal apparatus, and calculating relative position coordinates of the image forming apparatus using the position information of the image forming apparatus and the position information of the terminal apparatus;
Acquiring an image using the terminal device; And
Overlaying and outputting an icon for the image forming apparatus on the obtained image based on the coordinate values of the calculated relative position coordinates;
A computer readable medium having computer readable program code embodied thereon.

Images