KR20230060039A - Extended display using mobile device - Google Patents

Abstract

The present disclosure discloses embodiments of extending a display of an image forming apparatus using a mobile terminal. The disclosed image forming apparatus includes a communication module forming a network connection with a mobile terminal, a user interface device, a processor, and a memory storing instructions executable by the processor, wherein the processor executes the instructions to display the user interface Identifying a visual object to be displayed on a device, determining a device on which the visual object is to be displayed among the image forming device and the mobile terminal based on the type of the identified visual object, and displaying the visual object on the mobile device In response to the determination, the communication module may be controlled to generate a screen to be displayed on the mobile terminal with the visual object and to transmit control data for the created screen to the mobile terminal.

Description

Display extension using mobile terminal {EXTENDED DISPLAY USING MOBILE DEVICE}

The image forming apparatus includes a control panel that controls functions of the image forming apparatus. Existing control panels include a small-sized screen with low resolution sufficient to control the image forming apparatus for limited purposes such as printing, copying, and scanning, and to briefly display the status of the image forming apparatus.

1 is a conceptual diagram illustrating an example of a method of extending a display of an image forming apparatus using a mobile terminal.
2 is a block diagram for explaining configurations and operations of an image forming apparatus and a mobile terminal.
3 is a flowchart of operations of extending a display by an image forming apparatus.
4 is a flowchart of operations for forming a network between an image forming apparatus and a mobile terminal.
5 is a diagram for explaining a two-way communication network formed between an image forming apparatus and a mobile terminal.
6 is a diagram for explaining a method of activating a display extension function in an image forming apparatus.
7 is a diagram for explaining objects displayed on an image forming apparatus and a mobile terminal by a display extension function.
8 is a diagram for explaining objects displayed on an image forming apparatus and a mobile terminal by a display extension function.
9 is a diagram for explaining an object displayed on an image forming apparatus and a mobile terminal by a display extension function.
10 is a diagram for explaining objects displayed on an image forming apparatus and a mobile terminal by a display extension function.
11 is a diagram for explaining objects displayed on an image forming apparatus and a mobile terminal by a display extension function.
12 is a diagram for explaining an example of a method of displaying an object on an image forming device and a mobile terminal by a display extension function.
13 is a diagram for explaining an example of a method of displaying an object on an image forming device and a mobile terminal by a display extension function.

As the functions provided by the image forming apparatus become more diverse, it is difficult to display the functions provided by the image forming apparatus on the screen of a conventional control panel. As the function of the image forming apparatus diversifies, a control panel having a larger screen is required. However, as the screen of the control panel becomes larger, the manufacturing cost of the image forming apparatus increases. Therefore, there is a need for a method of expanding the display of the image forming apparatus by using the display of the mobile terminal including the display having a higher resolution than the control panel.

The present disclosure discloses embodiments of extending a display of an image forming apparatus using a mobile terminal. The disclosed image forming apparatus includes a communication module forming a network connection with a mobile terminal, a user interface device, a processor, and a memory storing instructions executable by the processor, wherein the processor executes the instructions to display the user interface Identifying a visual object to be displayed on a device, determining a device on which the visual object is to be displayed among the image forming device and the mobile terminal based on the type of the identified visual object, and displaying the visual object on the mobile device In response to the determination, the communication module may be controlled to generate a screen to be displayed on the mobile terminal with the visual object and to transmit control data for the created screen to the mobile terminal.

According to the disclosed embodiments, functions of an image forming apparatus, which are difficult to provide through a screen of a control panel of an image forming apparatus, can be easily provided to a user using a display of a mobile terminal.

Hereinafter, with reference to the accompanying drawings, embodiments of the present disclosure will be described in detail so that those skilled in the art can easily carry out the present disclosure. However, the present disclosure may be implemented in many different forms and is not limited to the embodiments described herein.

In the present disclosure, an image forming apparatus collectively refers to an apparatus capable of performing an image forming operation, such as a printer, a copier, a multifunction machine, a fax machine, and the like. An image forming job may mean various tasks related to images, such as printing, copying, scanning, faxing, or functions related thereto, and a series of processes necessary to perform an image forming job. It may mean including.

Mobile terminals of the present disclosure collectively refer to electronic devices carried by users, and may include computing devices such as PCs (Personal Computers), smartphones, tablets, and notebooks that form a network with an image forming device to transmit and receive data. there is. The mobile terminal may serve as a host device that transmits output data generated using content to be an image forming task to the image forming device.

In the present disclosure, output data refers to a printing command describing an image generated by a host device using standard functions in response to a print request from a user. The output data is also called a print job. The output data may include Page Description Language (PDL) data in which the host device encodes a printing command in a description language. The output data may include a Printer Job Language (PJL) related to settings of the image forming apparatus. The output data may include pixel information of the image (eg, information indicating whether a pixel corresponds to an image, graphic, or text). The image forming apparatus may generate image information by performing rendering using output data.

In the present disclosure, a visual object refers to a screen component to be displayed in relation to a function provided by an image forming apparatus. For example, the visual object may be a setting screen of the image forming device, icons and widgets provided by the image forming device, a text box, a preview of an output to be output from the image forming device, a scanned image acquired by the image forming device, and image formation. It may include applications provided by the device. One or more visual objects may be displayed on at least one of a screen of a control panel of the image forming apparatus and a display of a mobile terminal connected to the image forming apparatus.

In the present disclosure, the extension function refers to a function of extending a display to use a display of a mobile terminal as a secondary display of an image forming apparatus. A mobile terminal connected to the image forming apparatus may display an object on a display of the mobile terminal based on control data received from the image forming apparatus. The mobile terminal may transmit data related to input received from the user to the image forming apparatus.

In the present disclosure, sensitive information refers to information that should not be leaked to others or to the outside. For example, sensitive information may include information requiring security, such as trade secrets, personal information, and a URL for accessing personal information and information requiring security.

1 is a conceptual diagram illustrating an example of a method of extending a display of an image forming apparatus using a mobile terminal.

Referring to FIG. 1 , the image forming apparatus 10 may include a control panel 13 . The control panel 13 may include a screen displaying one or more visual objects.

The image forming apparatus 10 may receive a user's input for controlling the image forming apparatus 10 through the control panel 13 . For example, the image forming apparatus 10 may receive a user's input for selecting one of the functions provided by the image forming apparatus 10 through the touch screen of the control panel 13 . The image forming apparatus 10 may display a visual object related to a selected function in response to a user's input.

The image forming apparatus 10 may form a network with the mobile terminal 20 . For example, the image forming apparatus 10 may operate as a web server so that the mobile terminal 20 can access it using a web browser. For another example, the image forming apparatus 10 may form a network for exchanging data with the mobile terminal 20 using a short-distance communication protocol (eg, Bluetooth, UWB, WIFI, etc.). The image forming apparatus 10 may transmit/receive data with the mobile terminal 20 through a network. For example, the image forming apparatus 10 may receive control data for controlling the image forming apparatus 10 from the mobile terminal 20 . The image forming apparatus 10 may receive output data from the mobile terminal 20 .

The image forming apparatus 10 may form a two-way communication network with the mobile terminal 20 . The bidirectional communication network is a communication network in which the image forming apparatus 10 and the mobile terminal 20 generate and transmit/receive data controlled by each other. For example, in the interactive communication network, the image forming apparatus 10 transmits first control data so that the mobile terminal 20 displays the first visual object, and the image forming apparatus 10 transmits the first control data to the screen of the control panel 13. This is a communication network through which the mobile terminal 20 transmits second control data to the image forming apparatus 10 so as to display the second visual object.

The image forming apparatus 10 may provide an extended function using the display of the mobile terminal 20 as a secondary display of the image forming apparatus 10 . The image forming apparatus 10 may generate control data for controlling the mobile terminal 20 and transmit the control data to the mobile terminal 20 in response to a user's input for activating the extended function.

As the extension function is activated, the image forming apparatus 10 may determine a position where the visual object is to be displayed. That is, the image forming apparatus 10 may determine whether to display a visual object on the screen of the control panel 13 of the image forming apparatus 10 or the display of the mobile terminal 20 .

The image forming apparatus 10 may determine a location where the visual object is to be displayed based on the type of the visual object. For example, the image forming apparatus 10 may determine a display location of a visual object related to sensitive information as the display of the mobile terminal 20 .

The image forming apparatus 10 may determine a position to display a visual object after the extension function is activated based on an arrangement of at least one visual object displayed on the screen of the control panel 13 before the extension function is activated. there is. For example, the image forming apparatus 10 determines the display position of each of the plurality of visual objects based on horizontal and/or vertical arrangement of the plurality of visual objects displayed on the screen of the control panel 13. and arrangement can be determined.

The image forming apparatus 10 may generate frame data including a visual object based on a position to be displayed. The screen data may include control data for controlling the image forming apparatus 10 or the mobile terminal 20 to display a screen including a visual object. For example, the image forming apparatus 10 may generate screen data of a first screen including a first visual object to be displayed on the screen of the control panel 13 . The image forming apparatus 10 may generate screen data of a second screen including a second visual object to be displayed on the display of the mobile terminal 20 . The image forming apparatus 10 generates screen data of a first screen and screen data of a second screen based on a language (eg, xml) used to generate a user interface (UI) of the image forming apparatus 10 can do. The image forming apparatus 10 may generate screen data of the second screen based on a language (eg, html) used to create a web page.

The image forming apparatus 10 may transmit screen data of the second screen to the mobile terminal 20 so that the mobile terminal 20 displays the second screen. For example, the mobile terminal 20 may display the second screen using a dedicated application. For another example, the mobile terminal 20 may display a second screen in the form of a web page by accessing the image forming apparatus 10 operating as a web server using a web browser.

The image forming apparatus 10 may swap the screen displayed on the screen of the control panel 13 and the frame displayed on the display of the mobile terminal 20 based on a user's input. For example, the image forming apparatus 10 may transmit screen data of the first screen to the mobile terminal 20 so that the display of the mobile terminal 20 displays the first screen. The image forming apparatus 10 may control the screen of the control panel 13 to display the second screen.

The image forming apparatus 10 may convert screen data to exchange the screen displayed on the screen of the control panel 13 and the screen displayed on the display of the mobile terminal 20 . For example, the image forming apparatus 10 may convert screen data of the second screen into a language (eg, xml) used to generate a user interface (UI) of the image forming apparatus 10 . The image forming apparatus 10 may convert screen data of the first screen into a language (eg, html) used to create a web page.

2 is a block diagram for explaining configurations and operations of an image forming apparatus and a mobile terminal.

Referring to FIG. 2 , the image forming apparatus 10 and the mobile terminal 20 may transmit and receive data.

The image forming apparatus 10 may include a scan module 11 , an output module 12 , a control panel 13 , a communication module 15 , a memory 17 , and a processor 19 . However, not all of the components of the image forming apparatus 10 shown in FIG. 2 are essential components of the image forming apparatus 10 . The image forming apparatus 10 may be implemented with more elements than the elements of the image forming apparatus 10 shown in FIG. 2, and fewer elements than the elements of the image forming apparatus 10 shown in FIG. The image forming apparatus 10 can be implemented by.

The scan module 11 scans an image recorded on a document by irradiating light onto a document and receiving light reflected from the document.

The output module 12 may form and output an image on a printing medium through various printing methods such as an electrophotographic method and an inkjet method. The output module 12 may include a cartridge containing toner or ink for forming an image on a print medium.

The control panel 13 refers to a user interface device that receives a user's input for controlling the image forming apparatus 10 and outputs information of the image forming apparatus. The control panel 13 may receive a user's input for controlling the image forming apparatus 10 . Also, the control panel 13 may include a screen for outputting information about the image forming apparatus 10 .

The communication module 15 may transmit/receive data with external devices or networks such as the mobile terminal 20 and the server 30 . For example, the communication module 15 may receive output data from an external device. The communication module 15 may perform wired communication of standards such as Ethernet, Universal Serial Bus (USB), and High Definition Multimedia Interface (HDMI). The communication module 15 may perform wireless communication of standards such as Wi-Fi (Wireless Fidelity), Wi-Fi Direct, Bluetooth, UWB (Ultra Wide Band), and NFC (Near Field Communication). The communication module 15 performs a fax function by transmitting the output image obtained by the scan module 11 to an external fax device or receiving output data from an external fax device so that the output module 12 outputs the output image. can

The memory 17 may store various types of data such as programs and files such as applications. The memory 17 may store at least one command (instruction) and at least one program for processing and control of the processor 19 . The memory 17 can store information about the states of the image forming apparatus 10 .

The memory 17 includes a memory temporarily storing data such as random access memory (RAM) and static random access memory (SRAM), a flash memory type, and a read-only memory (ROM). It may include at least one type of storage medium among data storages for non-temporarily storing data. Memory 17 may consist of one or more memory chips.

The processor 19 may include at least one processing unit such as a CPU or the like. Also, the processor 19 may include at least one specialized processing unit corresponding to each function.

The processor 19, by executing at least one instruction (instruction) stored in the memory 17, the scan module 11, the output module to provide the embodiments described with reference to FIGS. 1, 3 to 13 (12), control panel 13, communication module 15, memory 17, etc. can be controlled overall.

The processor 19 may access and use data stored in the memory 17 or may store new data in the memory 17 . The processor 19 may execute a program installed in the memory 17. The processor 19 may store files or programs received from the outside through the communication module 15 in the memory 17 .

The mobile terminal 20 may include a display 21 , a user interface 26 , a communication module 25 , a memory 27 and a processor 29 . However, not all of the components of the mobile terminal 20 shown in FIG. 2 are essential components of the mobile terminal 20 . The mobile terminal 20 may be implemented by more components than the components of the mobile terminal 20 shown in FIG. 2, and by fewer components than the components of the mobile terminal 20 shown in FIG. Terminal 20 may be implemented.

The communication module 25 allows the mobile terminal 20 to transmit/receive data with an external device or network such as the image forming apparatus 10 and an external server. For example, the communication module 25 may form a network with the image forming apparatus 10 . The communication module 25 may perform wired communication of standards such as Ethernet, Universal Serial Bus (USB), and High Definition Multimedia Interface (HDMI). The communication module 25 may perform wireless communication of standards such as Wi-Fi (Wireless Fidelity), Wi-Fi Direct, Bluetooth, UWB (Ultra Wide Band), and NFC (Near Field Communication).

User interface 26 is a hardware module and/or device that receives user input and outputs information. For example, the user interface 26 may include a combination of an output device such as a monitor and an input device such as a keyboard, touch pad, or mouse, or a touch screen. Also, the user interface 26 may include a display 26 - 1 that displays a screen including a visual object using screen data received from the image forming apparatus 10 .

The memory 27 may store various types of data such as an operating system (OS), programs such as applications, and files. The memory 27 may store at least one command (instruction) and at least one program for processing and control of the processor 23 .

The memory 27 includes a memory temporarily storing data such as RAM (Random Access Memory) SRAM (Static Random Access Memory), a flash memory type, and a ROM (Read-Only Memory). It may include at least one type of storage medium among data storages for non-temporarily storing data.

The processor 29 may include at least one processing unit such as a CPU or the like. Also, the processor 29 may include at least one specialized processing unit corresponding to each function.

The processor 29, by executing at least one command (instruction) stored in the memory 27, to provide the embodiments described with reference to Figures 1, 3 to 13 below, the user interface 26, The communication module 25, the memory 27, and the like can be generally controlled.

The processor 29 may generate control data for controlling the operation of the image forming apparatus 10 . The processor 29 may generate output data so that the image forming apparatus 10 may perform an image forming task. For example, the processor 29 may execute instructions stored in the memory 27 to generate a page description file of a document to be printed according to a print request through an application application.

3 is a flowchart of operations of extending a display by an image forming apparatus.

Each step of FIG. 3 may be implemented by instructions executable by a processor. Also, instructions may be stored in a storage medium readable by a computer including a processor.

Referring to step S310, the image forming apparatus may display a first screen.

The image forming apparatus may display a first screen including one or more visual objects on the screen of the control panel. For example, the image forming apparatus may include a setting screen of the image forming apparatus, icons and widgets provided by the image forming apparatus, a text box, a preview of an output to be output from the image forming apparatus, and scanned images and images obtained by the image forming apparatus. A first screen including at least one visual object such as an application provided by the forming device may be displayed.

The image forming apparatus may display a first screen on which a plurality of visual objects are arranged on the screen of the control panel. For example, the image forming apparatus may arrange each of a plurality of visual objects to be positioned on at least a part of the first screen. The image forming apparatus may arrange a plurality of visual objects to divide at least a portion of the first screen horizontally and/or vertically. The image forming apparatus may generate first screen data in which a plurality of visual objects are arranged. The image forming apparatus may control the control panel to display the first screen data on the screen of the control panel.

Referring to step S320, the image forming apparatus may form a network with the mobile terminal. For example, the image forming apparatus may form a network using a mobile terminal and a short-distance communication protocol (eg, Bluetooth, UWB, WIFI, etc.). For another example, the image forming apparatus may operate as a web server so that a mobile terminal can access the image using a web browser. The image forming apparatus may form, with the mobile terminal, a bidirectional communication network in which data that the image forming apparatus and the mobile terminal mutually control is generated and transmitted/received.

Referring to step S330, the image forming apparatus may activate the extended function.

The image forming apparatus may activate an extension function for extending the display to use the display of the mobile terminal as a secondary display of the image forming apparatus.

The image forming apparatus may activate the extended function based on a result of identifying the connection of the mobile terminal. For example, the image forming apparatus may identify a mobile terminal connected to a predetermined web address generated by the image forming apparatus. Alternatively, the image forming apparatus may identify a mobile terminal connected using a short-range communication protocol (eg, Bluetooth, UWB, WIFI, etc.). The image forming apparatus may automatically activate the extended function in response to access by the mobile terminal. Alternatively, the image forming apparatus may activate the extended function in response to a user's input to a UI (eg, a toggle button) for activating the extended function.

The image forming apparatus may generate control data for controlling the mobile terminal in response to activation of the extended function. For example, the image forming apparatus may generate control data for controlling the mobile terminal to display screen data including one or more visual objects.

Referring to step S340, the image forming apparatus may determine a position where the visual object is to be displayed.

As the extension function is activated in step S330, the image forming apparatus may determine a position where the visual object is to be displayed. That is, the image forming apparatus 10 may determine whether to display a visual object on the screen of the control panel 13 of the image forming apparatus 10 or the display of the mobile terminal 20 . Also, the image forming apparatus may determine an arrangement of visual objects at positions where the visual objects are to be displayed.

The image forming apparatus may determine a display position of the visual object based on the priority of the visual object. The priority of visual objects indicates weights of visual objects determined to be desirable to be displayed on the mobile terminal for security of sensitive information and user convenience. The image forming apparatus may determine the location of the visual object so that the visual object having a higher priority is displayed on the mobile terminal. The priority of the visual object may be determined based on the type of the visual object and the ratio of the visual object. For example, the priority of a visual object receiving an input related to sensitive information from a user may be set high. In addition, the priority of an image visual object such as a thumbnail of a printout or a scanned image may be set high. In addition, the priority of a text box receiving an input from a user through a virtual keyboard may be set high. In addition, the priority of a visual object having a height longer than a width may be set higher.

In operation S310, the image forming apparatus may determine a position where the visual object is to be displayed based on the arrangement of the visual object displayed on the screen of the control panel. For example, the image forming apparatus may determine the display position and arrangement of a plurality of visual objects based on horizontal and/or vertical arrangement of the plurality of visual objects displayed on the screen of the control panel.

Referring to step S350, the image forming apparatus may create a second screen including visual objects.

The image forming apparatus may create a screen including the visual object according to the location where the visual object determined in step S340 is to be displayed. The image forming apparatus may generate a first screen including the first visual object determined to be displayed on the screen of the control panel and create a second screen including the second visual object determined to be displayed on the display of the mobile terminal.

The image forming apparatus may determine a language used to generate screen data based on the location where the visual object determined in step S340 is to be displayed. The screen data may include control data for controlling the image forming apparatus 10 or the mobile terminal 20 to display a screen including a visual object. For example, the image forming apparatus may generate first screen data used to display the first screen based on a language (eg, xml) used to create a user interface (UI) of the image forming apparatus. there is. The image forming apparatus may generate second screen data used to display the second screen based on a language (eg, html) used to create a web page.

Referring to step S360, the image forming apparatus may transmit the second screen to the mobile terminal.

The image forming apparatus may transmit the second screen data to the mobile terminal so that the mobile terminal displays the second screen generated in step S350. For example, the image forming apparatus may transmit second screen data using a network formed through a dedicated application of a mobile terminal. For another example, the image forming device operating as a web server may transmit second screen data generated as a web page to a mobile terminal accessed through a web browser.

The mobile terminal may display the received second screen data. For example, the mobile terminal may display the second screen using a dedicated application. For another example, the mobile terminal may display the second screen in the form of a web page using a web browser.

4 is a flowchart of operations for forming a network between an image forming apparatus and a mobile terminal.

Each step of FIG. 4 may be implemented by instructions executable by a processor. Also, instructions may be stored in a storage medium readable by a computer including a processor.

In step S410, the mobile terminal 20 may transmit connection request data to the image forming apparatus 10.

The mobile terminal 20 may transmit connection request data to the image forming apparatus 10 using a short-range communication protocol (eg, Bluetooth, UWB, WIFI, etc.). For example, the mobile terminal 20 may transmit connection request data conforming to Bluetooth standard specifications to the image forming apparatus 10 .

Alternatively, the mobile terminal 20 may transmit connection request data to the image forming apparatus 10 connected to the Internet. For example, the mobile terminal 20 may transmit connection request data to an IP address assigned to the image forming apparatus 10 .

In step S420, the image forming apparatus 10 may create a network connection with the mobile terminal 20.

The image forming apparatus 10 may create a network connection with the mobile terminal 20 in response to the connection request data received in step S410. The image forming apparatus 10 may transmit data for forming a network to the mobile terminal 20 through a short-range communication protocol (eg, Bluetooth, UWB, WIFI, etc.) used to receive connection request data. For example, the image forming apparatus 10 may transmit connection data conforming to Bluetooth standard specifications to the mobile terminal 20 . Alternatively, the image forming apparatus 10 may transmit data for forming a network to the mobile terminal 20 accessed through the IP address assigned to the image forming apparatus 10 .

In step S430, the image forming device 10 and the mobile terminal 20 may perform a Websocket handshake.

The image forming apparatus 10 and the mobile terminal 20 may use a websocket to establish bi-directional connection communication. The image forming apparatus 10 and the mobile terminal 20 may perform websocket handshake for websocket connection.

The image forming device 10 and/or the mobile terminal 20 may transmit Websocket connection request data to the other party. For example, the mobile terminal 20 may transmit Websocket connection request data including a key value for verifying a handshake response to the image forming device 10 through a “Sec-WebSocket-Key” header. In addition, the mobile terminal 20 may transmit to the image forming device 10 Websocket connection request data including additional information, such as protocol information to be used as an aid in a WebSocket connection, in a header. Websocket connection request data may include headers indicating Websocket request data, such as "Connection:Upgrade" and "Upgrade:websocket".

The image forming apparatus 10 may generate Websocket connection data in response to Websocket connection request data received from the mobile terminal 20 . The image forming apparatus 10 may hash SHA-1 by adding a specific value to the key value received through the “Sec-WebSocket-Key” header of the Websocket connection request data. The image forming apparatus 10 may obtain a key value for verifying a handshake response by encoding the hashed value into base64. The image forming apparatus 10 may generate Websocket connection data including a “Sec-WebSocket-Accept” header including a key value. The image forming apparatus 10 may transmit Websocket connection data to the mobile terminal 20 .

The mobile terminal 20 may verify the Websocket handshake by using Websocket connection data received from the image forming device 10 . After the Websocket handshake is completed, the mobile terminal 20 and the image forming device 10 may form a network for bidirectional communication using the Websocket protocol.

In step S440, the image forming device 10 may transmit the access token to the mobile terminal 20.

In step S430, the image forming device 10 may transmit an access token to the mobile terminal 20 through a bidirectional communication network formed by performing a Websocket handshake between the image forming apparatus 10 and the mobile terminal 20. Access Token is a token used to authenticate the mobile terminal 20 .

The image forming apparatus 10 may generate a token including an access token based on information about the mobile terminal 20 received from the mobile terminal 20 . For example, the image forming apparatus 10 may include identification information of the mobile terminal 20 , identification information of a user using the mobile terminal 20 , and identification of an application used by the mobile terminal to access the image forming apparatus 10 . Token can be created using one or more of the information. Specifically, the image forming apparatus 10 includes identification information of the mobile terminal 20 , identification information of a user using the mobile terminal 20 , and identification information of an application used by the mobile terminal to access the image forming apparatus 10 . By hashing one or more of them, a Token containing the obtained string can be created.

In step S450, the image forming device 10 and the mobile terminal 20 may activate extended functions.

The image forming apparatus 10 and/or the mobile terminal 20 may activate an extension function that expands the display to use the display of the mobile terminal 20 as a secondary display of the image forming apparatus 10 . The image forming apparatus 10 and/or the mobile terminal 20 may transmit and receive control data for activating extended functions. Since step S450 is similar to step S330 of FIG. 3, overlapping details will be omitted.

In step S460, the image forming apparatus 10 may provide the URL to the mobile terminal 20.

The image forming apparatus 10 may transmit information about a URL to be accessed by the mobile terminal 20 to the image forming apparatus 10 in order to provide extended functions. The image forming apparatus 10 may transmit URLs allocated differently for each user to the mobile terminal 20 .

The image forming apparatus 10 includes at least one of identification information of the mobile terminal 20, identification information of a user using the mobile terminal 20, and identification information of an application used by the mobile terminal to access the image forming apparatus 10. Based on the string obtained by hashing, a URL that is differently assigned to each user may be generated.

The image forming apparatus 10 may transmit a URL to the mobile terminal 20 using Websocket. The image forming apparatus 10 may transmit a URL including access address, port, and protocol information to the mobile terminal.

In step S470, the mobile terminal 20 can access the URL received from the image forming device 10.

The mobile terminal 20 can access the URL received in step S460. The mobile terminal 20 may access the image forming apparatus 10 through URLs allocated differently for each user. When the mobile terminal 20 accesses the URL, it can be authenticated by the image forming device using the access token obtained in step S440.

The image forming apparatus 10 may check the validity of an Access Token received from a mobile terminal accessing through a URL. When the access token is valid, the image forming apparatus 10 may transmit second screen data generated as a web page to the mobile terminal 20 .

When the connection of the mobile terminal 20 is terminated, the image forming apparatus 10 may release the network established with the mobile terminal 20 . For example, the image forming apparatus 10 may release the websocket connection formed in step S430.

5 is a diagram for explaining a two-way communication network formed between an image forming apparatus and a mobile terminal.

Referring to FIG. 5 , the image forming apparatus may display a first screen on the screen 510 of the control panel 13 . For example, the image forming apparatus may display output settings on the screen 510 of the control panel 13 . The mobile terminal 20 may display the second screen on the display 520 . For example, the mobile terminal 20 may display a preview of output on the display 520 .

The image forming apparatus 10 may form a two-way communication network with the mobile terminal 20 . The image forming apparatus 10 may perform a websocket handshake with the mobile terminal 20 to form a two-way communication network using a websocket protocol.

The bidirectional communication network is a communication network in which the image forming apparatus 10 and the mobile terminal 20 generate and transmit/receive data controlled by each other. For example, in a bidirectional communication network, an image forming apparatus transmits first control data for updating a second screen to a mobile terminal based on a first user's input for a first screen, and the mobile terminal displays the second screen. Based on the second user's input for the first screen, second control data for updating the first screen may be transmitted to the image forming apparatus. The first control data may include screen data obtained by regenerating the second screen.

According to the user's input for selecting color or black and white for the output setting screen displayed on the screen 510 of the image formation device, the color of the preview of the output displayed on the display 520 of the mobile terminal 20 changes to color or black and white. can be changed to

In addition, according to the user's input for modifying the preview of the output displayed on the display 520 of the mobile terminal 20 (for example, the user's input for masking sensitive information), the screen 510 of the Mars forming device Modifications (eg, sensitive information is masked) may be displayed on the displayed output setting screen. The mobile terminal 20 may modify the output image received from the image forming apparatus based on the user's input for modifying the preview, encode the modified output image into base64, and transmit the modified output image to the image forming apparatus. The Mars Forming Device may display the correction items of the output on the screen 510 .

6 is a diagram for explaining a method of activating a display extension function in an image forming apparatus.

Referring to FIG. 6 , the image forming apparatus may activate an extension function for extending the display to use the display of the mobile terminal as a secondary display of the image forming apparatus.

The image forming apparatus may activate the extension function in response to a user's input to the UI for activating the extension function. For example, the image forming apparatus may activate an extended function in response to a user's input 620 to the toggle button 610 .

The image forming apparatus may activate the extended function based on a result of identifying the connection of the mobile terminal. For example, the image forming apparatus may activate the toggle button 610 so that the user input 620 is received only when connection of the mobile terminal is identified. Alternatively, the image forming apparatus may automatically activate the extension function in response to access by the mobile terminal.

The image forming apparatus may generate control data for controlling the mobile terminal in response to activation of the extended function. For example, the image forming apparatus may generate control data for controlling the mobile terminal to display screen data including one or more visual objects.

7 to 9 are diagrams for explaining visual objects displayed on an image forming apparatus and a mobile terminal by a display extension function.

7 to 9 , the image forming apparatus may determine a position of a visual object to be displayed after the extension function is activated, based on an arrangement of visual objects displayed on the control panel 13a before the extension function is activated. there is. For example, the image forming apparatus has a plurality of visual objects to be displayed on the control panel 13b and the mobile terminal 20 based on horizontal and/or vertical arrangement of the visual objects displayed on the control panel 13a. , and the arrangement of visual objects to be displayed on the control panel 13b and the mobile terminal 20 may be determined.

Referring to FIG. 7 , the control panel 13a of the image forming apparatus may display the first visual object 710a and the second visual object 720a in a horizontal arrangement. For example, the first visual object 710a may be an output setting screen. The second visual object 720a may be a preview of the output. The aspect ratios of the first visual object 710a and the second visual object 720a may be different.

The image forming apparatus may determine positions where the first visual object 710a and the second visual object 720a are to be displayed in response to activation of the extension function. The image forming apparatus displays the first visual object 710b on the control panel 13b based on the horizontal arrangement of the first visual object 710a and the second visual object 720a displayed on the control panel 13a. and it may be determined to display the second visual object 720b on the mobile terminal 20 . For example, the image forming apparatus may determine to display an output setting screen on the control panel 13b and display a preview of the output on the mobile terminal 20 .

Referring to FIG. 8 , the image forming apparatus may display a first visual object 810a, a second visual object 820a, and a third visual object 830a by dividing the control panel 13a into three parts. For example, the first visual object 810a may be a print list. The second visual object 820a may be an output setting screen. The third visual object 830a may be a preview of an output. The control panel 13a may vertically arrange and display the first visual object 810a with respect to the horizontally arranged second and third visual objects 820a and 830a.

The image forming apparatus may determine positions where the first visual object 810a, the second visual object 820a, and the third visual object 830a are to be displayed in response to activation of the extension function. The image forming apparatus controls the control panel 13b based on the arrangement of the first visual object 810a, the second visual object 820a, and the third visual object 830a within the three-part control panel 13a. The first visual object 810b may be determined to be displayed, and the second visual object 820b and the third visual object 830b may be determined to be displayed on the mobile terminal 20 . For example, the image forming apparatus may determine to display a print list on the control panel 13b and display an output setting screen and a preview of output on the mobile terminal 20 .

The image forming apparatus may determine the arrangement of visual objects to be displayed on the control panel 13b and the mobile terminal 20 . The image forming apparatus controls the control panel 13b and the mobile terminal ( 20) may determine an arrangement of visual objects to be displayed. For example, the image forming apparatus may display the second visual object 820a and the third visual object 830a horizontally on the control panel 13a based on the display of the second visual object 820a and the third visual object 830a. The object 830a may be determined to be displayed on the mobile terminal 20 in a horizontal arrangement.

Referring to FIG. 9 , the image forming apparatus may display a first visual object 910a, a second visual object 920a, and a third visual object 930a by dividing the control panel 13a into three parts. For example, the first visual object 910a may be a print list. The second visual object 920a may be an output setting screen. The third visual object 930a may be a preview of output. The control panel 13a may horizontally arrange and display the first visual object 910a with respect to the vertically arranged second and third visual objects 920a and 930a.

The image forming apparatus may determine positions where the first visual object 910a, the second visual object 920a, and the third visual object 930a are to be displayed in response to activation of the extension function. The image forming apparatus controls the control panel 13b based on the arrangement of the first visual object 910a, the second visual object 920a, and the third visual object 930a within the three-divided control panel 13a. The first visual object 910b may be determined to be displayed, and the second visual object 920b and the third visual object 930b may be determined to be displayed on the mobile terminal 20 . For example, the image forming apparatus may determine to display a print list on the control panel 13b and display an output setting screen and a preview of output on the mobile terminal 20 .

The image forming apparatus may determine the arrangement of visual objects to be displayed on the control panel 13b and the mobile terminal 20 . The image forming apparatus controls the control panel 13b and the mobile terminal ( 20) may determine an arrangement of visual objects to be displayed. For example, the image forming apparatus displays the second visual object 920a and the third visual object 920a and the third visual object 930a based on the display of the second visual object 920a and the third visual object 930a in a vertical arrangement on the control panel 13a. The object 930a may be determined to be displayed on the mobile terminal 20 in a vertical arrangement.

10 and 11 are diagrams for explaining visual objects displayed on an image forming apparatus and a mobile terminal by a display extension function.

Referring to FIGS. 10 and 11 , the image forming apparatus may determine a display location of a visual object based on the type of the visual object to be displayed. The image forming apparatus may determine a display position of the visual object by using the priority of the visual object determined based on the type of the visual object. The priority of visual objects is the weight of visual objects that are predetermined to be displayed on the mobile terminal for security enhancement and user convenience.

For example, the priority of a visual object receiving an input on sensitive information from a user may be determined as the first priority. In addition, the priority of an image visual object such as a thumbnail of a printout or a scanned image may be determined as second priority. Also, the priority of a text box receiving an input from a user through a virtual keyboard may be determined as the third priority. Also, the priority of the output setting screen may be determined as 4th priority. The image forming apparatus may determine the location of the visual object so that the visual object having a higher priority is displayed on the mobile terminal 20 .

Referring to FIG. 10 , the image forming apparatus may receive a user input. The image forming apparatus may determine a position where a text box for receiving a user's text input is to be displayed. For example, the image forming apparatus may determine a position where a text box for receiving sensitive information is to be displayed among the screen 1010 of the control panel 13 or the display 1020 of the mobile terminal 20 . The image forming apparatus may determine a position where the text box is displayed on the display 1020 of the mobile terminal 20 based on the priority of text boxes receiving sensitive information.

The image forming apparatus determines screen data of a first screen to be displayed on the screen 1010 of the control panel 13 and a second screen to be displayed on the display 1020 of the mobile terminal 20 based on the determined position of the text box. Screen data can be created. For example, the image forming apparatus may use the mobile terminal 20 to generate screen data of the first screen including a notification message to log in. Also, the image forming apparatus may generate screen data of a second screen including a text box for receiving user's personal information. The image forming apparatus may control the screen 1010 of the control panel 13 to display the first screen. The image forming apparatus may transmit screen data of the second screen to the mobile terminal so that the second screen is displayed on the display 1020 of the mobile terminal 20 .

Referring to FIG. 11 , the image forming apparatus may display a plurality of visual objects. The image forming apparatus may determine a display position of each of the plurality of visual objects based on the priority of the visual object to be displayed. The image forming apparatus may determine a location where the output setting screen and the output preview are to be displayed in response to activation of the extended function.

For example, the image forming apparatus selects an output setting screen and an output preview from among the screen 1110 of the control panel 13 or the display 1120 of the mobile terminal 20 based on the priorities of the output setting screen and the output preview. You can determine where each will be displayed. The image forming apparatus determines the location where the output setting screen will be displayed on the screen 1110 based on the fact that the priority of the output preview is higher than the priority of the output setting screen, and determines the location where the output preview will be displayed on the mobile terminal 20 It can be determined by the display 1120 of.

The image forming apparatus displays screen data of the first screen to be displayed on the screen 1110 of the control panel 13 and the display 1120 of the mobile terminal 20 based on the determined positions of the output setting screen and the output preview. Screen data of the second screen may be generated. For example, the image forming apparatus may generate screen data of a first screen including an output setting screen. Also, the image forming apparatus may generate screen data of the second screen including the output preview. The image forming apparatus may control the screen 1110 of the control panel 13 to display the first screen. The image forming apparatus may transmit screen data of the second screen to the mobile terminal so that the second screen is displayed on the display 1120 of the mobile terminal 20 .

12 is a diagram for explaining an example of a method of displaying a visual object on an image forming device and a mobile terminal by a display extension function.

Referring to FIG. 12 , the image forming apparatus with the extended function activated generates first screen data 1220 and second screen data 1240 in response to a user's input to the screen 1210 of the control panel 13a. can create The image forming apparatus may control the control panel 13b so that the control panel 13b displays the first screen 1230 using the first screen data 1220 . The image forming apparatus may control the mobile terminal 20 to display the second screen 1260 by using the second screen data 1240 . For example, in the image forming apparatus, in response to a user's input for selecting one from a list of documents, the first screen data ( 1220), and the mobile terminal 20 may generate second screen data 1240 used to display the second screen 1260 including the output preview.

The image forming apparatus may generate the first screen data 1220 and the second screen data 1240 using different languages. For example, the image forming apparatus may generate the first screen data 1220 based on a language (eg, xml) used to generate a user interface (UI) of the image forming apparatus. In addition, the image forming apparatus operating as a web server may use a language (eg, html) used to generate a web page so that a mobile terminal accessing the image forming apparatus through a web browser can display the second screen 1260. ), the second screen data 1240 may be generated.

The image forming apparatus may transmit, to the mobile terminal 20 , a URL 1250 to be accessed by the mobile terminal 20 to receive the second screen data 1240 . The URL 1250 may include information about an address, port, and protocol to be accessed by the mobile terminal 20 . The mobile terminal 20 can receive the second screen data 1240 by accessing the image forming device using the transmitted URL 1250 . The mobile terminal 20 may display the second screen 1260 using the received second screen data 1240 .

13 is a diagram for explaining an example of a method of displaying a visual object on an image forming device and a mobile terminal by a display extension function.

Referring to FIG. 13 , the image forming apparatus with the extended function activated corresponds to a user's input, with a first screen 1310a displayed on the control panel 13a and a second screen 1310b displayed on the mobile terminal 20a. It is possible to control the control panel 13a and the mobile terminal 20a to exchange and display ). For example, in the image forming apparatus, a first screen 1310a including an output setting screen displayed on the control panel 13a is displayed on the mobile terminal 20a, and a preview of the output displayed on the mobile terminal 20a is displayed. The control panel 13a and the mobile terminal 20a may be controlled so that the second screen 1310b is displayed on the control panel 13a.

The image forming apparatus may convert screen data generated based on the first language into screen data generated based on the second language. For example, an image forming device operating as a web server generates a user interface (UI) of the image forming device so that a mobile terminal accessing the image forming device through a web browser can display the first screen 1350. The first screen data 1320 generated based on the language used (eg xml) is the first screen data 1330 generated based on the language used to create the web page (eg html) can be converted to In addition, the image forming apparatus converts the second screen data 1360 generated based on a language (eg, html) used to generate a web page into a language (eg, html) used to create a user interface (UI) of the image forming apparatus. For example, it may be converted into second screen data 1370 generated based on xml.

The image forming apparatus may control the control panel 13b to display the second screen 1380 by using the second screen data 1370 . The image forming apparatus may control the mobile terminal 20b to display the first screen 1350 by using the first screen data 1330 .

The image forming apparatus may transmit, to the mobile terminal 20b, a URL 1340 to be accessed by the mobile terminal 20b to receive the first screen data 1330. The URL 1340 may include information about an address, port, and protocol to be accessed by the mobile terminal 20b. The mobile terminal 20b can receive the first screen data 1330 by accessing the image forming device using the transmitted URL 1340 . The mobile terminal 20b may display the second screen 1350 using the received first screen data 1330 .

Meanwhile, according to the disclosed embodiments, the present disclosure may promote user convenience by providing a preview image to a user through a high-resolution display of a mobile terminal. In addition, according to the present disclosure, since a user's input for sensitive information can be received through a mobile terminal, security for sensitive information can be enhanced. In addition, the present disclosure may promote user convenience by receiving text input through a virtual keyboard of a mobile terminal familiar to the user. In addition, the present disclosure may promote user convenience by performing image editing (eg, masking sensitive information) through a mobile terminal. In addition, according to the present disclosure, the display extension function can be provided regardless of the manufacturer or type of the mobile terminal by providing a web page to the mobile terminal by the image forming apparatus.

Meanwhile, the above-described operating methods of the image forming apparatus 10 may be implemented in the form of a computer readable storage medium storing instructions or data executable by a computer or processor. It can be written as a program that can be executed on a computer, and can be implemented in a general-purpose digital computer that operates such a program using a computer-readable storage medium. Such computer-readable storage media include read-only memory (ROM), random-access memory (RAM), flash memory, CD-ROMs, CD-Rs, CD+Rs, CD-RWs, CD+RWs, DVD-ROMs , DVD-Rs, DVD+Rs, DVD-RWs, DVD+RWs, DVD-RAMs, BD-ROMs, BD-Rs, BD-R LTHs, BD-REs, magnetic tapes, floppy disks, magneto-optical data storage devices, An optical data storage device, hard disk, solid-state disk (SSD), and may store instructions or software, related data, data files, and data structures, and may cause the processor or computer to execute instructions or It may be any device capable of providing software, related data, data files, and data structures.

As described above, although the embodiments have been described with limited examples and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Claims (15)

In the image forming apparatus,
A communication module for forming a network connection with a mobile terminal;
user interface devices;
processor; and
a memory for storing instructions executable by the processor;
The processor, by executing the instructions,
identify a visual object to be displayed on the user interface device;
Based on the type of the identified visual object, determining a device on which the visual object is to be displayed among the image forming device and the mobile terminal;
In response to the determination that the visual object is displayed on the mobile device, a screen on which the visual object is to be displayed on the mobile terminal is generated;
An image forming apparatus that controls the communication module to transmit control data for the generated screen to the mobile terminal.
According to claim 1,
The processor, by executing the instructions,
Determine that a visual object related to sensitive information is displayed on the mobile terminal;
Generating the screen including the visual object related to the sensitive information,
image forming device.
According to claim 2,
The sensitive information
including one or more of personal information, information requiring security, and a URL for accessing personal information and information requiring security;
image forming device.
According to claim 1,
The processor, by executing the commands, identifies a plurality of visual objects displayed on the user interface device, and based on the arrangement of the identified plurality of visual objects, to generate a screen to be displayed on the mobile terminal,
image forming device.
According to claim 1,
The image forming apparatus operates as a web server,
The processor, by executing the instructions,
Creating a screen to be displayed on the mobile terminal in the form of a web page,
image forming device.
According to claim 1,
The processor, by executing the instructions,
generating control data for swapping screens displayed on the mobile terminal and the user interface device;
Controlling the communication module to transmit control data for switching the screen to the mobile terminal;
image forming device.
According to claim 1,
The processor, by executing the instructions,
Regenerating a screen to be displayed on the mobile terminal based on an input received by the user interface device and transmitting the screen to the mobile terminal;
image forming device.
A computer-readable storage medium in which instructions executable by a processor of a computer are stored, the computer-readable storage medium comprising:
a command for identifying a visual object to be displayed on a user interface device of an image forming apparatus;
instructions for determining a device on which the visual object will be displayed among the image forming device and a mobile terminal, based on the type of the identified visual object;
instructions for generating a screen on which the visual object is to be displayed on the mobile terminal in response to the determination of displaying the visual object on the mobile device; and
a command for transmitting control data for the generated screen to the mobile terminal; including,
A computer-readable storage medium.
A method of using a mobile terminal as an auxiliary display by an image forming apparatus, the method comprising:
identifying a visual object to be displayed on a user interface device of the image forming apparatus;
determining a device on which the visual object is to be displayed among the image forming device and the mobile terminal, based on the type of the identified visual object;
generating a screen on which the visual object is to be displayed on the mobile terminal in response to the determination of displaying the visual object on the mobile device; and
transmitting control data for the generated screen to the mobile terminal; including,
method.
According to claim 9,
The step of determining the device on which the visual object will be displayed,
determining a visual object related to sensitive information to be displayed on the mobile terminal; and
Generating the screen including the visual object related to the sensitive information; including,
method.
According to claim 10,
The sensitive information
including one or more of personal information, information requiring security, and a URL for accessing personal information and information requiring security;
method.
According to claim 9,
The step of determining the device on which the visual object will be displayed,
identifying a plurality of visual objects displayed on the user interface device; and
Based on the arrangement of the identified plurality of visual objects, generating a screen to be displayed on the mobile terminal,
method.
According to claim 9,
The image forming apparatus operates as a web server,
Creating a screen to be displayed on the mobile terminal,
Creating a screen to be displayed on the mobile terminal in the form of a web page,
method.
According to claim 9,
The method,
generating control data for exchanging screens displayed on the mobile terminal and the user interface device; and
Further comprising transmitting control data for switching the screen to the mobile terminal,
method.
According to claim 9,
The method,
Further comprising regenerating a screen to be displayed on the mobile terminal based on an input received by the user interface device and transmitting the screen to the mobile terminal.
method.

Images