JP2019140601A - Image processing apparatus and program - Google Patents

Abstract

To provide an image processing apparatus to which a job can be submitted, which uses a resource, such as a camera in an external terminal, while the same operability as that of an operation panel of a main body is ensured with a remote control from an external terminal.SOLUTION: A web UI unit 21 of an image processing apparatus 10 displays the same operation screen 50 on an operation panel 17 of the apparatus and a browser of an external terminal 40. When a Scan To button 53 is operated on the operation panel 17, the web UI unit reads a document by using a scanner unit 14 on the image processing apparatus 10 side, and when the Scan To button 53 is operated on the external terminal 40, controls to photograph a document by using a camera 41 of the external terminal 40 in place of the scanner unit 14, and executes a job.SELECTED DRAWING: Figure 4

Description

  The present invention relates to an image processing apparatus that can be remotely operated from an external terminal and a program thereof.

  When scanning or copying a document on a multifunction device, place the document on the platen of the document reader of the multifunction device, and perform job setting operations and start instruction input operations from the multifunction device's control panel. Is called.

  On the other hand, in recent years, work forms such as transferring an image taken with an external terminal such as a portable terminal to a multifunction peripheral, printing out the image with the multifunction peripheral, and converting the image into a PDF format file are increasing.

However, in order to transfer an image taken with a mobile terminal to a multifunction device and print it out,
(1) Taking an image with a mobile terminal,
(2) Select an image captured and saved and send it to the MFP.
Because it is necessary to perform a plurality of individual operations such as the above, it takes time and labor to print with the multifunction machine.

  Patent Document 1 below proposes an image processing system that reduces the above trouble. In this image system, a dedicated application having functions of an imaging control unit for controlling a camera and a remote control unit for remotely operating an image processing apparatus is operated on a mobile terminal, and the dedicated control unit performs image processing by the remote control unit. When the device is in a cooperative state for remote control, the camera operates to send an image captured by the camera to the image processing device, and when the device is in a cooperative state, the image processing device sets the portable terminal as an input source of image data. It is like that. Therefore, if the cooperative state is set, an image captured by the portable terminal is automatically transferred to the image processing apparatus and input.

  In addition, there is a method called Web UI as one of methods for remotely operating a multifunction peripheral or the like. This is because a device such as a multifunction device having a Web server function transmits display data for the operation screen of the device itself to an external terminal such as a tablet in HTML format, and the external terminal that has received this displays the operation screen in a browser. Then, the operation information received from the user is returned from the external terminal to the Web server of the multi-function peripheral, thereby receiving a job setting operation.

JP 2014-64151 A

  In the image processing system disclosed in Patent Document 1, it is necessary to install and operate a dedicated application on a mobile terminal. In addition, since the input source is switched to the portable terminal after setting to the cooperative state, once the cooperative state is established, the operation panel of the image processing apparatus body is operated to input an image from the scanner of the image processing apparatus body. Can not be. That is, the operation panel of the main body and the operation from the mobile terminal cannot be performed simultaneously. Furthermore, it supports only when the camera of the portable terminal is used as an input device, and cannot cope with other cases. In the linked state, the operation on the mobile terminal is a remote operation of the image processing apparatus, so the operation such as changing the shooting conditions of the camera on the mobile terminal side cannot be performed. To perform such an operation, another application is required. Had to start.

  By the way, the conventional Web UI function normally displays an operation screen having the same contents and the same display mode as the operation screen displayed on the operation panel of the main body on the mobile terminal. Therefore, the remote operation from the mobile terminal is the same as the main body. Operability can be ensured. However, since an operation screen with the same contents and the same display mode as the operation screen displayed on the operation panel of the main body is displayed on the mobile terminal, jobs other than jobs that can be accepted on the operation panel of the main body are submitted from the mobile terminal. It is not possible. For example, a scan job that can be submitted from the operation screen of the mobile terminal is a scan job that reads a document with a scanner of the main body, and a job that uses a camera on the mobile terminal side as an input device cannot be submitted.

  The present invention is intended to solve the above problem, and submits a job using resources such as the camera on the external terminal side while ensuring the same operability as the operation panel of the main body in remote operation from the external terminal. An object of the present invention is to provide an image processing apparatus and a program that can be used.

  The gist of the present invention for achieving the object lies in the inventions of the following items.

[1] a first display control unit that transmits and displays display data for an operation screen including a first operation button on the operation panel of the device;
A second display control unit for transmitting and displaying display data of an operation screen including the first operation button having the same display mode as that displayed on the operation panel to an external terminal;
When an operation on the first operation button is received on the operation panel, the first function corresponding to the first operation button is executed using the first resource of the own device, and the operation on the first operation button is performed. When the external terminal receives the function, the function execution unit executes the first function using the second resource of the external terminal instead of the first resource;
An image processing apparatus comprising:

  In the above invention and the invention described in [14] below, the operation screen including the first operation button having the same display mode is displayed on the operation panel and the external terminal, but the first operation button is operated on the operation panel. Uses the first resource on the image processing apparatus side for executing the function corresponding to the first operation button. When the first operation button is operated on the external terminal, the external terminal executes the function corresponding to the first operation button. Switch to use the second resource on the side.

[2] A distance recognition unit for recognizing a distance between the image processing apparatus and the external terminal;
The function execution unit has a function corresponding to the first operation button when the external terminal receives an operation on the first operation button in a state where the distance recognized by the distance recognition unit is less than a predetermined value. The image processing apparatus according to [1], wherein the image processing apparatus is executed using the first resource.

  In the above invention and the invention described in [15] below, even when the first operation button is operated by an external device, if the distance between the image processing device and the external terminal is short (less than a predetermined value), The first resource on the image processing apparatus side is used to execute the function corresponding to one operation button.

[3] For each operation button, the image processing apparatus side resource used when the operation button is operated on the operation panel to execute a function corresponding to the operation button, and the operation button is operated on the external terminal The image processing apparatus according to [1] or [2], wherein a resource on the external terminal side used when executing a function corresponding to the operation button is assigned in advance.

  In the above invention and the invention described in [16] below, when there are a plurality of operation buttons such as function keys, for each operation button, the operation button is operated on the operation panel and the resources used when the operation button is operated on the external terminal. The resources to be used when the operation buttons are assigned are allocated in advance, and the resources to be used when each operation button is operated are switched accordingly.

[4] The function corresponding to the first operation button is a function of optically reading a document and acquiring image data.
The first resource is a scanner included in the image processing apparatus;
The second resource is a camera of the external terminal;
The function execution unit, when executing the function corresponding to the first operation button using the camera, causes the external terminal to transmit an image captured by the camera to the own device. The image processing apparatus according to [1] or [2].

  In the above invention and the invention described in [17] below, when the first operation button is operated on the external terminal, the camera of the external terminal is used instead of the scanner of the own apparatus, and data of an image photographed by the camera is used. The external terminal is controlled so as to be transferred to its own device.

[5] A third display control unit for transmitting display data of a setting screen for setting the reading conditions of the scanner to the operation panel and displaying the data.
A fourth display control unit for transmitting and displaying the display data of the setting screen having the same display content as the setting screen to the external terminal;
A shooting condition changing unit that changes the shooting condition of the camera in accordance with the reading condition set on the setting screen displayed on the external terminal;
The image processing apparatus according to [4], further comprising:

  In the above invention and the invention described in [18] below, a setting screen having the same display content as the setting screen for changing the reading conditions of the scanner displayed on the operation panel is displayed on the external terminal. When the setting is changed on the external terminal, the shooting condition of the camera of the external terminal, not the scanner, is changed.

[6] The display data transmitted by the second display control unit to the external terminal is for displaying an operation screen further including a second operation button,
When the function executing unit executes the function corresponding to the first operation button using the camera, the function executing unit adds the first tag information to the image captured by the camera to the external terminal. When the function corresponding to the second operation button is executed, the external terminal adds the second tag information to the image captured by the camera and transmits the image to the own apparatus. 4] or [5].

  In the above invention and the invention described in [19] below, the first tag information is added to an image photographed by the camera based on the operation of the first operation button, and the image photographed by the camera based on the operation of the second operation button. A second information tag is added to the image.

[7] The image processing apparatus may further include an image deforming unit that performs a deformation process on the image when a predetermined condition including that the first tag information is added to the image received from the external terminal is satisfied. The image processing device according to [6].

  In the above invention and the invention described in [20] below, when the first tag information is added to the image, that is, for an image taken by the camera of the external terminal based on the operation of the first operation button, Image deformation processing is performed.

[8] An image analysis unit that analyzes an image received from the external terminal,
The image processing apparatus according to [7], wherein the predetermined condition includes that the image received from the external terminal is determined to be a photograph of a document by analysis of the image analysis unit. .

  In the above invention and the invention described in [21] below, when the first tag information is added and it is determined by image analysis that the subject is a document, the image is subjected to deformation processing.

[9] An image analysis unit that analyzes an image received from the external terminal;
An image deformation unit that performs a deformation process on the image when it is determined by the analysis of the image analysis unit that the image received from the external terminal is a photograph of a document;
The image processing apparatus according to [4] or [5], further comprising:

  In the above invention and the invention described in [22] below, when the subject is determined to be a document by image analysis regardless of the presence or absence of the first tag information, the image is subjected to deformation processing.

[10] When the subject in the image has a quadrangular boundary and a character image is inside the quadrangular boundary, the image analysis unit determines that the image is a photograph of a document. The image processing apparatus according to [8] or [9], which is characterized.

[11] The camera can photograph a subject in three dimensions,
The image processing apparatus according to [8] or [9], wherein the image analysis unit determines that the image is a photograph of a document when a subject in the image is a flat object. .

[12] The image processing apparatus includes a Web server that functions as the first display control unit and the second display control unit.
Each of the operation panel and the external terminal is a Web client that requests operation screen display data from the Web server.
The image processing apparatus according to any one of [1] to [11], wherein:

  In the above invention and the invention described in [25] below, the first and second display control units of the image processing apparatus have a relationship with a Web server, and the operation panel and the external terminal have a relationship with a Web client.

[13] The image processing apparatus according to any one of [1] to [12], wherein display data is transmitted by an HTTP protocol.

[14] A program executed by the control unit of the image processing apparatus,
A first display step of transmitting and displaying operation screen display data including a first operation button on the operation panel of the device;
A second display step of transmitting and displaying display data of an operation screen including the first operation button having the same display mode as that displayed on the operation panel to an external terminal;
When an operation on the first operation button is received on the operation panel, the first function corresponding to the first operation button is executed using the first resource of the own device, and the operation on the first operation button is performed. When the external terminal receives the first function, a function execution step of executing the first function using a second resource of the external terminal instead of the first resource;
The program characterized by having.

[15] The method further comprises a distance recognition step of recognizing a distance between the image processing apparatus and the external terminal.
In the function execution step, when the external terminal receives an operation on the first operation button in a state where the distance recognized in the distance recognition step is less than a predetermined value, a function corresponding to the first operation button is The program according to [14], wherein the program is executed using the first resource.

[16] For each operation button, the image processing apparatus side resource used when the operation button is operated on the operation panel to execute the function corresponding to the operation button, and the operation button is operated on the external terminal The program according to [14] or [15], wherein the resource on the external terminal side used when executing the function corresponding to the operation button is assigned in advance.

[17] The function corresponding to the first operation button is a function of optically reading a document and acquiring image data.
The first resource is a scanner included in the image processing apparatus;
The second resource is a camera of the external terminal;
In the function execution step, when the function corresponding to the first operation button is executed using the camera, the external terminal is caused to transmit an image captured by the camera to the image processing apparatus. The program according to [14] or [15].

[18] A third display step of transmitting display data of a setting screen for setting the reading conditions of the scanner to the operation panel and displaying the data.
A fourth display step of transmitting display data of the setting screen having the same display content as the setting screen to the external terminal for display;
A shooting condition changing step of changing the shooting condition of the camera in correspondence with the reading condition set on the setting screen displayed on the external terminal in the fourth display step;
The program according to [17], further comprising:

[19] The display data to be transmitted to the external terminal in the second display step is to display an operation screen further including a second operation button on the external terminal,
In the function execution step, when the function corresponding to the first operation button is executed using the camera, the first tag information is added to the image captured by the camera to the external terminal and the image When transmitting to the processing device and executing the function corresponding to the second operation button, adding the second tag information to the image captured by the camera and transmitting the image to the image processing device to the external terminal. The program according to [17] or [18], which is characterized.

[20] The image processing method may further include an image deformation step of performing a deformation process on the image when a predetermined condition including that the first tag information is added to the image received from the external terminal is satisfied. The program according to [19].

[21] An image analysis step of analyzing an image received from the external terminal,
The program according to [20], wherein the predetermined condition includes that the image received from the external terminal is determined to be a photograph of a document by analysis of the image analysis step.

[22] An image analysis step of analyzing an image received from the external terminal;
When the analysis determines that the image received from the external terminal is a photograph of a document, an image deformation step for performing a deformation process on the image;
The program according to [17] or [18], further comprising:

[23] In the image analysis step, if the subject in the image has a quadrangular boundary and a character image is present inside the quadrangular boundary, it is determined that the image is a photograph of a document. The program according to [21] or [22], which is characterized.

[24] The camera can photograph a subject in three dimensions,
The program according to [21] or [22], wherein, in the image analysis step, when the subject in the image is a flat object, it is determined that the image is a photograph of a document.

[25] The control unit of the image processing apparatus functions as a Web server,
Each of the operation panel and the external terminal is a Web client that requests operation screen display data from the Web server.
The program according to any one of [14] to [24].

[26] The program according to any one of [14] to [25], wherein display data is transmitted using an HTTP protocol.

[27] In the first display step, display data in which an instruction to operate the operation panel is embedded so as to notify the control unit of acceptance of the operation when the first operation button is operated. Send to the operation panel,
In the function execution step, when the notification is received from the operation panel, the first function is executed using the first resource of the image processing apparatus,
In the second display step, when the first operation button is operated, the external resource is used so that the second resource of the external terminal is used instead of the first resource for executing the first function. Display data embedded with an instruction to operate the terminal is transmitted to the external terminal, and the second display step is performed when the external terminal receives an operation on the first operation button in the function execution step. The program according to [14], wherein the first function is performed by using a second resource of the external terminal instead of the first resource.

  In the above invention, the resource to be used is switched by changing the program embedded in the display data to be transmitted in response to the request from the operation panel and the request from the external terminal.

  According to the image processing apparatus and program of the present invention, it is possible to submit a job that uses resources such as the camera on the external terminal side while ensuring the same operability as the operation panel of the main body in remote operation from the external terminal. it can.

1 is a diagram illustrating an example of an image processing system including an image processing apparatus according to an embodiment of the present invention. 1 is a block diagram illustrating a schematic configuration of an image processing apparatus according to an embodiment of the present invention. It is a figure which shows an example of the operation screen displayed on the operation panel of an image processing apparatus, or the browser of an external terminal with the same display mode by the WebUI function which the image processing apparatus which concerns on embodiment of this invention has. FIG. 10 is a diagram showing an operation when a ScanTo button is pressed on the operation panel and an operation when the ScanTo button is pressed on the external terminal 40. FIG. 4 is a diagram illustrating an example of a document photographed by a camera of an external terminal and a corrected document obtained by correcting the document by image deformation processing. It is a figure which shows an example in the case of switching the resource used based on the distance of an image processing apparatus and an external terminal. It is a figure which shows the operation | movement when a ScanTo button is pressed with an external terminal, and the operation when a PhotoTo button is pressed. It is a flowchart which shows the process (detail of step S306 of FIG. 7) which judges whether the to-be-photographed object in an image is a document by image analysis. It is a figure which shows the information flow in the case of receiving user operation on the operation screen displayed on the operation panel of the image processing apparatus. It is a figure which shows the information flow at the time of receiving user operation on the operation screen displayed on the browser of the external terminal. 6 is a flowchart illustrating processing performed by a Web UI unit of the image processing apparatus. It is a figure which shows the resource used for execution of the function corresponding to the function key for every function key when this function key is operated with an operation panel, and when it is operated with an external terminal.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 shows an example of an image processing system 3 including an image processing apparatus 10 according to an embodiment of the present invention. The image processing system 3 includes an image processing apparatus 10, an external terminal 40 such as a portable terminal used by a user, a cloud server 5, and the like. The external terminal 40 is a portable information processing apparatus having a camera 41. The external terminal 40 has a CPU (Central Processing Unit) and the like, and operates according to an OS program or an application program.

  The image processing apparatus 10 has a copy function for printing an image obtained by reading a document with a scanner on paper, and a scan function for sending an image obtained by reading with a scanner to a file and transmitting it to the outside or storing it in an internal storage. This is a so-called multi-function machine having a facsimile function for transmitting / receiving an image according to a facsimile communication procedure, a print function for printing an image based on received print data on a sheet, and the like. Note that the image processing apparatus 10 is not limited to a multifunction peripheral, and may be any apparatus as long as it has any of the above functions or performs some kind of image processing.

  The image processing apparatus 10 has a function as a Web server that provides a Web UI function. The Web UI function transmits operation screen display data (HTML format) to the external terminal 40, and the external terminal 40 that receives the data displays the operation screen with a browser and receives information on operations received from the user from the external terminal 40. This function receives a job setting operation or the like from a remote external terminal 40 by sending a reply to the Web server of the image processing apparatus 10. In the Web UI function, the image processing apparatus 10 is a Web server, and the external terminal 40 is a Web client. Transmission of operation screen display data is performed using the HTTP protocol.

  The operation screen provided to the external terminal 40 by the Web UI function has the same display content as the operation screen displayed on the operation panel 17 of the own apparatus (the screen configuration, the operation buttons to be arranged, and the display mode thereof are the same). Thereby, the user can perform a remote operation on the same operation screen as when operating with the familiar operation panel of the image processing apparatus 10.

  FIG. 2 is a block diagram illustrating a schematic configuration of the image processing apparatus 10. The image processing apparatus 10 includes a CPU 11 as a control unit that comprehensively controls the operation of the image processing apparatus 10. A program memory 12, a RAM (Random Access Memory) 13, a scanner unit 14, a printer unit 15, a network communication unit 16, an operation panel 17, a facsimile communication unit 18, a distance recognition unit 19, and the like are connected to the CPU 11 through a bus. .

  The CPU 11 is based on the OS program, and executes middleware, application programs, and the like. Various programs are stored in the program memory 12, and each function of the image processing apparatus 10 is realized by the CPU 11 executing various processes according to these programs.

  The RAM 13 is used as a work memory for temporarily storing various data when the CPU 11 executes processing based on a program, an image memory for storing image data, and the like.

  The scanner unit 14 performs a function of optically reading a document and acquiring image data. The scanner unit 14 sequentially moves, for example, a light source that irradiates light on a document, a line image sensor that receives the reflected light for one line in the width direction, and a line-by-line reading position in the length direction of the document. An optical path composed of a moving unit, a lens or a mirror that guides the reflected light from the document to the line image sensor and forms an image, a conversion unit that converts the analog image signal output from the line image sensor into digital image data, etc. It is configured with.

  The scanner unit 14 reads an original set on the platen glass and is provided with an automatic document conveying device. The document set on the document table of the automatic document conveying device is automatically automatically one by one from the top. It is also possible to read out a plurality of originals continuously.

  The printer unit 15 has a function of forming (printing) an image corresponding to image data on a sheet. Here, it has a paper conveying device, a photosensitive drum, a charging device, a laser unit, a developing device, a transfer separation device, a cleaning device, and a fixing device, and forms an image by an electrophotographic process. It is configured as a so-called laser printer. Other methods may be used for image formation.

  The network communication unit 16 has a function of communicating with external devices such as the external terminal 40 and the cloud server 5 via a network.

  The facsimile communication unit 18 functions to transmit / receive image data to / from an external facsimile apparatus via a telephone line.

  The operation panel 17 includes an operation unit 17a and a display unit 17b. The display unit 17b functions to display various operation screens, setting screens, and the like, and includes a liquid crystal display. The operation unit 17a includes various hard keys such as a start button and a numeric keypad that receive various operations from the user, and a touch screen provided on the display surface of the display unit 17b.

  The image processing apparatus 10 handles the operation panel 17 like the external terminal 40, and the operation panel 17 also displays an operation screen and accepts an operation using the Web UI function. That is, the image processing apparatus 10 serves as a Web server in the Web UI function, and the operation panel 17 operates as a client that makes a request such as a screen request to the Web server, as with the external terminal 40.

  The distance recognition unit 19 recognizes the distance between the image processing apparatus 10 and the external terminal 40. For example, the distance between the external device 40 connected to the own device (image processing device 10) and the own device, the position information sent from the terminal, and the image processing device 10 and the external terminal 40 are connected by wireless communication. If it is, it is recognized based on the radio field intensity.

  The CPU 11 of the image processing apparatus 10 functions as a Web UI unit 21, a main body control unit 22, an image analysis unit 23, an image deformation unit 24, and the like by executing a program.

  The web UI unit 21 performs processing related to the web UI function. The Web UI unit 21 includes functions such as a request source determination unit 31, a display control unit 32, a function execution control unit 33, and an imaging condition change unit 34.

  When receiving a request such as a screen request or other information, the request source determination unit 31 determines whether the transmission source (request source) of the request or information is the operation panel 17 or the external terminal 40. The display control unit 32 performs a function of generating display data for the operation screen and the setting screen in response to the screen request and transmitting the data to the request source. In the display data, a program that is executed on a terminal (or an operation panel) that is displaying the display data when an operation button in the operation screen is operated is embedded.

  The function execution control unit 33 performs control for executing a function corresponding to the operation button received on the operation screen. The function execution control unit 33 uses the same operation button as the operation button when operated on the operation screen displayed on the operation panel 17 and when operated on the operation screen displayed on the external terminal 40. Switch the resource used when executing the corresponding function. Here, when the resources on the image processing apparatus 10 side and the resources on the external terminal 40 side can be substituted for each other, these resources are the targets of switching.

  For example, when the ScanTo function button is operated on the operation screen displayed on the operation panel 17, the scanner unit 14 of the image processing apparatus 10 is used for reading a document, and the ScanTo is displayed on the operation screen displayed on the external terminal 40. When the function button is operated, resources are switched such that the camera 41 of the external terminal 40 is used for reading the document.

  The photographing condition changing unit 34 has a function of allowing the external terminal 40 to change the setting of the photographing condition of the camera 41 of the external terminal 40 while the screen is displayed on the external terminal 40 by the Web UI function. Here, if the setting screen for operating the setting screen for setting the reading condition of the scanner unit 14 is displayed on the external terminal 40, the camera 41 of the external terminal 40 is not the scanner unit 14 of the image processing apparatus 10. The shooting conditions are changed.

  The image analysis unit 23 has a function of determining whether or not the subject of the image captured by the camera 41 of the external terminal 40 is a document by image analysis.

  The image deformation unit 24 functions to deform an image. For example, when a document (originally a rectangle) in an image photographed by the camera 41 of the external terminal 40 is distorted into a trapezoid or the like, the image is deformed so as to be a rectangle and corrected. The image deformation unit 24 may transfer an image to the external cloud server 5 and request the cloud server 5 to perform image deformation processing.

  The main body control unit 22 controls the operation of each unit of the image processing apparatus 10 and the execution of jobs. For example, the operation of each unit is controlled when executing a copy job or a scan job.

  FIG. 3 shows an example of an operation screen 50 displayed on the operation panel 17 of the image processing apparatus 10 or the browser of the external terminal 40 by the Web UI function. Basically, the operation screen is displayed on the operation panel 17 and the external terminal 40 in the same display content / display mode.

  On the operation screen 50, a function button for selecting a function to be executed, a power button, a registration button for registering an arbitrary series of operations, a start button 51 for instructing start of job execution, an instruction for stopping a job being executed, etc. A stop button for resetting, a reset button for resetting the current setting contents and the like to default values and the like are arranged.

  The function buttons include a Copy button 52 for selecting a copy function, a ScanTo button 53 for selecting a scan function, a Fax button 54 for selecting a facsimile function, a PhotoTo button 55 for selecting a subject other than a document with the camera 41, There are a Box button 56 for selecting a box function and a Tool button 57 for selecting various tools.

  Next, an operation in which the Web UI unit 21 of the image processing apparatus 10 switches resources used for executing a function related to the pressed operation button in the operation screen will be described.

  4 illustrates a case where the ScanTo button 53 is pressed on the operation screen 50 displayed on the operation panel 17 of the image processing apparatus 10 and a case where the ScanTo button 53 is pressed on the operation screen 50 displayed on the external terminal 40. The operation is shown in contrast. When the ScanTo button 53 is operated on the operation panel 17, the scanner unit 14 of the image processing apparatus 10 reads the document (step S101), and the main body control unit 22 controls the storage of the read image and the printing on the paper. And executed (step S102).

  On the other hand, when the ScanTo button 53 is pressed on the operation screen being displayed on the external terminal 40, the camera 41 of the external terminal 40 is activated, and the camera 41 captures a document (step S111). The external terminal 40 transmits an image captured by the camera 41 to the image processing apparatus 10 (step S112), and the image processing apparatus 10 that has received the image performs image transformation processing as necessary (step S113). Thereafter, image storage and printing on paper are executed under the control of the main body control unit 22 (step S114).

  FIG. 5 shows an example of a document 61 photographed by the camera 41 of the external terminal 40 and a corrected document 62 after this is corrected by image deformation processing. If a photograph is taken with the optical axis of the camera 41 of the external terminal 40 perpendicular to the document, a rectangular document can be photographed into a rectangle. However, for example, when an original placed on a flat desk is photographed from an oblique direction, the optical axis of the camera 41 is not perpendicular to the original, and a rectangular original is distorted into a trapezoid or a rhombus. Taken at. Therefore, the image is corrected by performing an image transformation process so that the document distorted in this way becomes an original rectangle.

  Next, a case where the use of resources of the external terminal 40 is limited according to the distance between the image processing apparatus 10 and the external terminal 40 will be described with reference to FIG. When the ScanTo button 53 is pressed on the operation screen 50 displayed on the external terminal 40, it is determined whether or not the distance between the image processing apparatus 10 and the external terminal 40 is a certain distance (specified value) or more (step S201).

  When the distance between the image processing apparatus 10 and the external terminal 40 is less than a certain distance, it is estimated that the user performs a job input operation for reading a document with the scanner unit 14 of the image processing apparatus 10 on the external terminal 40. The Therefore, even if the ScanTo button 53 is pressed on the operation screen 50 displayed on the external terminal 40, if the distance between the image processing apparatus 10 and the external terminal 40 is less than a certain distance (step S201; No), the image The original is read by the scanner unit 14 of the processing apparatus 10 (step S202). Thereafter, image storage and printing on paper are executed under the control of the main body control unit 22 (step S203).

  On the other hand, when the distance between the image processing apparatus 10 and the external terminal 40 is a certain distance or more, it is estimated that the user prints the document read by the camera 41 of the external terminal 40 with the image processing apparatus 10. Therefore, when the distance between the image processing apparatus 10 and the external terminal 40 is equal to or greater than a certain distance (step S201; Yes), the camera 41 of the external terminal 40 is activated and a document is photographed by the camera 41 (step S204). The captured image is transmitted from the external terminal 40 to the image processing apparatus 10 (step S205), and the image processing apparatus 10 that has received the image performs image transformation processing as necessary (step S206). Thereafter, image storage and printing on paper are executed under the control of the main body control unit 22 (step S207).

  Next, operations when the ScanTo button 53 is operated on the external terminal 40 and when the PhotoTo button 55 is operated will be described. Here, the ScanTo button 53 is used when reading a document, and the PhotoTo button 55 is used when shooting a person other than the document, for example, a person.

  As shown in FIG. 7, when the ScanTo button 53 is pressed on the operation screen 50 displayed on the external terminal 40, the camera 41 of the external terminal 40 is activated and an image is taken with the camera 41 (step S301). . The external terminal 40 adds to the image data a first tag indicating that the image is captured by operating the ScanTo button 53, and transmits the image tag to the image processing apparatus 10 (step S302).

  On the other hand, when the PhotoTo button 55 is pressed on the operation screen 50 displayed on the external terminal 40, the camera 41 of the external terminal 40 is activated and an image is taken with the camera 41 (step S303). The external terminal 40 adds a second tag indicating that the image is captured by operating the PhotoTo button 55 to the image data, and transmits the image data to the image processing apparatus 10 (step S304).

  The image processing apparatus 10 that has received the image from the external terminal 40 checks whether or not the first tag is added to the image (step S305), and if the first tag is added (step S305; Yes). Then, it is determined that the subject in the image is a document, image transformation processing is performed on the image (step S306), and the process proceeds to step S307.

  On the other hand, when the first tag is not added to the image (when the second tag is added, or when neither the first tag nor the second tag is added) (step S305; No), the image is transformed into the image. The process proceeds to step S307 without performing the process. In this case, image processing for a photograph may be performed on the image.

  In step S307, image storage and printing on paper are executed under the control of the main body control unit 22.

  Next, a case will be described in which it is determined whether or not the subject in the image is a document by image analysis, and whether or not to perform image deformation processing is switched based on the determination result.

  FIG. 8 shows details of step S306 in FIG. 7 when it is determined by image analysis whether or not the subject in the image is a document. This process is executed when it is determined in step S305 in FIG. 7 that the first tag has been added to the image.

  The image processing apparatus 10 performs image analysis to determine whether or not the subject in the image is a document (step S401). For example, if the subject has a square boundary and a character image is inside the square boundary, the subject is determined to be a document. If the camera 41 is three-dimensional (3D) compatible, it is checked whether or not the subject in the image is a flat object. You may make it judge that it is not. The document may be determined to be a document on the condition that it is flat and has a square outer shape and a character image inside.

  As a result of the image analysis, if it is determined that the subject in the image is a document (step S402; Yes), an image transformation process is performed (step S405), and this process ends.

  As a result of the image analysis, if it is determined that the subject in the image is not a document (step S402; No), the user is warned and inquired whether or not to perform image transformation processing using this image as a document (step S403). ). When a selection for performing the image deformation process is received from the user (step S404; Yes), the image deformation process is performed (step S405), and this process ends. When the user has received a selection that the image deformation process is not to be performed (step S404; No), the present process is terminated without performing the image deformation process.

  Note that the processing of FIG. 8 may be executed regardless of whether or not the first tag is added to the image. For example, the process of FIG. 8 may be performed in a configuration in which the process of adding the first tag and the second tag is not performed on the external terminal 40 side.

  Next, an information flow between each unit of the image processing apparatus 10 and each unit of the external terminal 40 when operating in response to a user operation will be described with reference to FIGS. 9 and 10.

  FIG. 9 shows an information flow when a user operation is received from the operation panel 17 of the image processing apparatus 10. The operation panel 17 of the image processing apparatus 10 transmits the screen request 1 to the Web UI unit 21 (P1). Receiving this, the Web UI unit 21 determines the request source of the screen request 1, and returns the operation screen display data (screen UI1) corresponding to the request source (here, the operation panel 17) to the request source (P2). ).

  The operation panel 17 displays an operation screen based on the received screen UI1. When the operation panel 17 receives an operation (in this case, an operation of a function button) from the user on the operation screen (P3), the operation panel 17 transmits key information 1 corresponding to the operated function button to the Web UI unit 21 (P4). Screen request 2 is transmitted to the Web UI unit 21 (P5).

  The Web UI unit 21 instructs the main body control unit 22 to execute processing corresponding to the received key information 1 (P6). The main body control unit 22 transmits a response such as OK to the received instruction to the Web UI unit 21 (P7) and executes the instructed process (P8). For example, when the key information 1 indicates an operation of the ScanTo button 53, the main body control unit 22 performs processing such as activation of the scanner unit 14 and setting of reading conditions.

  Further, the Web UI unit 21 identifies the transition destination screen based on the key information 1 received in P4, and generates display data (screen UI2) for the identified screen in response to the screen request 2 received in P5. Transmit (P9). For example, when the ScanTo button 53 is operated on the operation screen 50 on which various function buttons are arranged, a setting screen for setting and changing the reading condition of the scanner unit 14 is specified as a screen after the transition, and the screen is displayed. Send data. Upon receiving an operation on this setting screen, the Web UI unit 21 instructs the main body control unit 22 to change the setting of the scanner unit 14 in response to the operation, and the main body control unit 22 changes the setting of the scanner unit 14 according to the instruction. .

  Thereafter, the operation panel 17 receives an operation of the start button 51 from the user (P10), and transmits the corresponding key information 2 to the Web UI unit 21 (P11). Further, the screen request 3 of the next screen is transmitted to the Web UI unit 21 (P12).

  The Web UI unit 21 instructs the main body control unit 22 to execute processing corresponding to the received key information 2 (P13). For example, after the start button 51 is pressed, the main body control unit 22 is instructed to start job execution. The main body control unit 22 transmits a response such as OK to the received instruction to the Web UI unit 21 (P14) and executes the instructed process (P15). For example, the main body control unit 22 controls the scanner unit 14 to execute a document reading operation and the like.

  Further, the Web UI unit 21 identifies the transition destination screen based on the key information 2 received at P11, and generates display data (screen UI3) for the identified screen in response to the screen request 3 received at P12. Transmit (P16). For example, an operation screen indicating that a document is being read is transmitted.

  FIG. 10 shows an information flow when the image processing apparatus 10 is remotely operated in response to a user operation on the external terminal 40. The browser of the external terminal 40 transmits the screen request 1 to the Web UI unit 21 of the image processing apparatus 10 (P21). Receiving this, the Web UI unit 21 determines the request source of the screen request 1, and returns the operation screen display data (screen UI1) corresponding to the request source (here, the external terminal 40) to the external terminal 40 ( P22). In the operation screen display data to be transmitted to the external terminal 40, a program to be executed by the external terminal 40 when a function button in the screen is operated is embedded in advance.

  The browser of the external terminal 40 displays an operation screen based on the received display data, and when an operation (in this case, operation of a function button) is received from the user (P23), the program embedded in the function button is executed. (P24). For example, a program for starting the camera 41 of the external terminal 40 is embedded in the ScanTo button 53, and the external terminal 40 executes this program and starts the camera 41 when the ScanTo button 53 is operated (P24). .

  Furthermore, the browser of the external terminal 40 transmits device information 1 indicating that the device (camera 41) has been activated to the Web UI unit 21 and transmits the next screen request 2 to the Web UI unit 21 (P26). The information on the activated device is sequentially transmitted from the browser of the external terminal 40 to the Web UI unit 21.

  The Web UI unit 21 specifies a transition destination screen based on the device information 1 received in P25, generates display data (screen UI2) for the specified screen for the screen request 2 received in P26, and generates an external terminal. 40 (P27). For example, the display data for displaying on the browser the setting screen having the same display content and display mode as the setting screen for changing the setting of the reading condition of the scanner unit 14 is transmitted. The display contents and display mode of this setting screen are the same as those displayed on the operation panel 17, but a program to be executed when an operation is received is embedded in each operation button. When an operation to change is received, processing for changing the exposure of the camera 41 is executed by the program. Items to be set include light on / off, resolution, scaling, photo / video, and the like.

  Thereafter, when the browser of the external terminal 40 receives an operation of the start button 51 from the user on the operation screen being displayed (P28), an image output from the camera 41 is displayed by executing a program embedded in the operation button. After capturing, the camera 41 is stopped (P29). Then, the device information 2 including the image data of the captured image is transmitted to the Web UI unit 21 (P30), and further the screen request 3 of the next screen is transmitted to the Web UI unit 21 (P31).

  The Web UI unit 21 instructs the main body control unit 22 to execute processing corresponding to the received device information 2 (P32). The main body control unit 22 transmits a response such as OK to the received instruction to the Web UI unit 21 (P33) and executes the instructed process (P34). For example, the Web UI unit 21 sends received image data and a print instruction to the main body control unit 22, and the main body control unit 22 that receives the image data controls the execution of a job for printing the received image.

  Further, the Web UI unit 21 specifies the transition destination screen based on the device information 2 received at P30, and generates display data (screen UI3) for the specified screen in response to the screen request 3 received at P31. Transmit (P35). For example, a screen during printing is transmitted.

  As described above, the display content and display mode of the operation screen displayed on the operation panel 17 and the external terminal 40 are the same, but when the operation button is operated on the operation screen display data transmitted to the external terminal 40. Since the program to be executed on the external terminal 40 is embedded, the scanner unit 14 is activated when the ScanTo button 53 in the operation screen is operated on the operation panel 17, and when operated on the external terminal 40, A job can be executed by automatically switching a device (resource) to be used, such as starting the camera 41.

  FIG. 11 is a flowchart showing processing performed by the Web UI unit 21. The Web UI unit 21 monitors the transmission of information such as a screen request and key information (or device information) from the client (operation panel 17 or external terminal 40) (steps S501; No, S506;; No ). When the screen request is received (step S501; Yes), it is determined whether the request source is the operation panel 17 or the external terminal 40 (step S502).

  If the request source is the operation panel 17 (step S502; operation panel), operation screen display data (screen UI) in which an instruction to return key information to the Web UI unit 21 when the operation button is pressed is generated. (Step S503), this is transmitted to the operation panel 17 of the request source (Step S505).

  On the other hand, if the request source is the external terminal 40 (step S502; external terminal), operation screen display data in which an instruction (program) for starting the device on the external terminal 40 side is embedded when the operation button is pressed. (Screen UI) is generated (step S504), and this is transmitted to the operation panel 17 of the request source (step S505).

  If key information or device information has been received (step S506; Yes), it is determined whether the information is information received from an external terminal (step S507), and information received from the operation panel 17 (key information) In the case of (step S507; No), an instruction for processing corresponding to the operation is transmitted to the main body control unit 22 (steps S509 and S510).

  When device information is received from the external terminal 40 (step S507; Yes), processing corresponding to the received information, for example, image transformation processing is executed (step S508), and printing of the processed image is performed on the main body. The controller 22 is instructed (step S510).

<Others>
When the scanner unit 14 of the image processing apparatus 10 reads and copies a plurality of documents while sequentially replacing them on the platen glass (when the automatic document feeder is not used), one document is read. Every time, the operation panel 17 accepts a selection operation of whether to read the next original further or to finish the reading and start the copy printing operation.

  Even when a plurality of documents are read using the camera 41 of the external terminal 40, the processing can be performed by the same user operation as described above. That is, when the operation of the Copy button 52 is received on the operation screen displayed on the external terminal 40 with the setting not to use the automatic document feeder, one document is shot (captured) by the camera 41 and then the next document When the selection of “with next document” is received, the camera 41 proceeds to the process of shooting the next document. When the selection of “end document” is received, the camera 41 is stopped. The operation may be performed such that an image is transmitted to the Web UI unit 21 of the image processing apparatus 10 to instruct printing.

  On the other hand, when the operation of the Copy button 52 is received on the operation screen of the external terminal 40 at the time of setting to use the automatic document feeder, a plurality of documents are shot in the video mode (Video) by the camera 41 of the external terminal 40. Then, the moving image is transferred to the image processing apparatus 10, and the Web UI unit 21 of the image processing apparatus 10 that has received the moving image analyzes the moving image to extract an image of each document, and prints the extracted plurality of documents or images. It only has to operate to save as a file.

  In the description so far, the case where the resource to be switched depending on whether the request source is the operation panel 17 or the external terminal 40 is the scanner unit 14 on the image processing apparatus 10 side and the camera 41 on the external terminal 40 side is an example. As described above, the resource to be switched is not limited to this. For example, as shown in FIG. 12, resources on the image processing apparatus 10 side and resources on the external terminal 40 side are allocated in advance for each function key, and an operation is received from the operation panel 17 or an operation is received from the external terminal 40. Depending on the resource, the resource used when executing the function corresponding to the function key may be switched according to the assignment in FIG. For example, when the “address book” selection button is pressed on the operation screen of the external terminal 40, the address book held by the external terminal 40 is activated.

  The embodiment of the present invention has been described with reference to the drawings. However, the specific configuration is not limited to that shown in the embodiment, and there are changes and additions within the scope of the present invention. Are also included in the present invention.

  The operation screen 50 shown in the embodiment is an example, and the present invention is not limited to this. Further, the operation screen displayed on the operation panel 17 and the operation screen displayed on the external terminal 40 may not be completely the same, and the number and arrangement of the displayed function keys and operation buttons are partially different. It doesn't matter. The display mode of at least one function button or operation button in the screen may be the same.

  In the description of FIG. 9 to FIG. 11, an example in which the device on the external terminal 40 side is activated and used by embedding a program executed on the external terminal 40 when the operation button is operated in the operation button will be described. However, it may be configured such that the Web UI unit 21 that receives the information of the operation button performs control for starting and operating the device on the external terminal 40 side. For example, when the ScanTo button 53 is operated on the operation screen 50 being displayed on the external terminal 40, the key information is notified from the external terminal 40 to the Web UI unit 21, and the Web UI unit 21 that receives the key information activates the camera 41. An instruction or an instruction to transfer a captured image may be transmitted to the external terminal 40.

DESCRIPTION OF SYMBOLS 3 ... Image processing system 5 ... Cloud server 10 ... Image processing apparatus 11 ... CPU
12 ... Program memory 13 ... RAM
DESCRIPTION OF SYMBOLS 14 ... Scanner part 15 ... Printer part 16 ... Network communication part 17 ... Operation panel 17a ... Operation part 17b ... Display part 18 ... Facsimile communication part 19 ... Distance recognition part 21 ... Web UI part 22 ... Main body control part 23 ... Image analysis part 24 ... Image transformation part 31 ... Request source judgment part 32 ... Display control part 33 ... Function execution control part 34 ... Shooting condition change part 40 ... External terminal 41 ... Camera 50 ... Operation screen 51 ... Start button 52 ... Copy button 53 ... ScanTo button 54 ... Fax button 55 ... PhotoTo button 56 ... Box button 57 ... Tool button 61 ... Original 62 ... Corrected original

Claims (27)

A first display control unit for transmitting and displaying operation screen display data including a first operation button on the operation panel of the device;
A second display control unit for transmitting and displaying display data of an operation screen including the first operation button having the same display mode as that displayed on the operation panel to an external terminal;
When an operation on the first operation button is received on the operation panel, the first function corresponding to the first operation button is executed using the first resource of the own device, and the operation on the first operation button is performed. When the external terminal receives the function, the function execution unit executes the first function using the second resource of the external terminal instead of the first resource;
An image processing apparatus comprising: A distance recognition unit for recognizing a distance between the image processing apparatus and the external terminal;
The function execution unit has a function corresponding to the first operation button when the external terminal receives an operation on the first operation button in a state where the distance recognized by the distance recognition unit is less than a predetermined value. The image processing apparatus according to claim 1, wherein the image processing apparatus is executed using the first resource. For each operation button, when the operation button is operated on the operation panel and the function corresponding to the operation button is executed, the resource on the image processing apparatus side and the operation button are operated on the external terminal. The image processing apparatus according to claim 1, wherein a resource on an external terminal side used when executing a function corresponding to the operation button is allocated in advance. The function corresponding to the first operation button is a function for optically reading a document and acquiring image data.
The first resource is a scanner included in the image processing apparatus;
The second resource is a camera of the external terminal;
The function execution unit, when executing the function corresponding to the first operation button using the camera, causes the external terminal to transmit an image captured by the camera to the own apparatus. Item 3. The image processing apparatus according to Item 1 or 2. A third display control unit for transmitting and displaying display data of a setting screen for setting the reading condition of the scanner to the operation panel;
A fourth display control unit for transmitting and displaying the display data of the setting screen having the same display content as the setting screen to the external terminal;
A shooting condition changing unit that changes the shooting condition of the camera in accordance with the reading condition set on the setting screen displayed on the external terminal;
The image processing apparatus according to claim 4, further comprising: The display data transmitted by the second display control unit to the external terminal is for displaying an operation screen further including a second operation button,
When the function executing unit executes the function corresponding to the first operation button using the camera, the function executing unit adds the first tag information to the image captured by the camera to the external terminal. When the function corresponding to the second operation button is executed, the external terminal adds the second tag information to the image photographed by the camera and transmits the image to the own apparatus. Item 6. The image processing apparatus according to Item 4 or 5. The image processing apparatus according to claim 6, further comprising an image deforming unit configured to perform a deformation process on the image when a predetermined condition including that the first tag information is added to the image received from the external terminal is satisfied. An image processing apparatus according to 1. An image analysis unit for analyzing an image received from the external terminal;
The image processing apparatus according to claim 7, wherein the predetermined condition includes that an image received from the external terminal is determined to be a photograph of a document by analysis of the image analysis unit. . An image analysis unit for analyzing an image received from the external terminal;
An image deformation unit that performs a deformation process on the image when it is determined by the analysis of the image analysis unit that the image received from the external terminal is a photograph of a document;
The image processing apparatus according to claim 4, further comprising: The image analysis unit determines that the image is a photograph of a document when the subject in the image has a quadrangular boundary and a character image is present inside the quadrangular boundary. The image processing apparatus according to claim 8 or 9. The camera can photograph a subject in three dimensions,
The image processing apparatus according to claim 8, wherein the image analysis unit determines that the image is a photograph of a document when a subject in the image is a flat object. The image processing apparatus includes a Web server that functions as the first display control unit and the second display control unit,
Each of the operation panel and the external terminal is a Web client that requests operation screen display data from the Web server.
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. The image processing apparatus according to any one of claims 1 to 12, wherein display data is transmitted by an HTTP protocol. A program executed by the control unit of the image processing apparatus,
A first display step of transmitting and displaying operation screen display data including a first operation button on the operation panel of the device;
A second display step of transmitting and displaying display data of an operation screen including the first operation button having the same display mode as that displayed on the operation panel to an external terminal;
When an operation on the first operation button is received on the operation panel, the first function corresponding to the first operation button is executed using the first resource of the own device, and the operation on the first operation button is performed. When the external terminal receives the first function, a function execution step of executing the first function using a second resource of the external terminal instead of the first resource;
The program characterized by having. A distance recognition step of recognizing a distance between the image processing apparatus and the external terminal;
In the function execution step, when the external terminal receives an operation on the first operation button in a state where the distance recognized in the distance recognition step is less than a predetermined value, a function corresponding to the first operation button is The program according to claim 14, wherein the program is executed using the first resource. For each operation button, when the operation button is operated on the operation panel and the function corresponding to the operation button is executed, the resource on the image processing apparatus side and the operation button are operated on the external terminal. The program according to claim 14 or 15, wherein resources on the external terminal side used when executing a function corresponding to the operation button are allocated in advance. The function corresponding to the first operation button is a function for optically reading a document and acquiring image data.
The first resource is a scanner included in the image processing apparatus;
The second resource is a camera of the external terminal;
In the function execution step, when the function corresponding to the first operation button is executed using the camera, the external terminal is caused to transmit an image captured by the camera to the image processing apparatus. The program according to claim 14 or 15. A third display step of transmitting display data of a setting screen for setting the reading condition of the scanner to the operation panel and displaying the data;
A fourth display step of transmitting display data of the setting screen having the same display content as the setting screen to the external terminal for display;
A shooting condition changing step of changing the shooting condition of the camera in correspondence with the reading condition set on the setting screen displayed on the external terminal in the fourth display step;
The program according to claim 17, further comprising: The display data to be transmitted to the external terminal in the second display step is to display an operation screen further including a second operation button on the external terminal,
In the function execution step, when the function corresponding to the first operation button is executed using the camera, the first tag information is added to the image captured by the camera to the external terminal and the image When transmitting to the processing device and executing the function corresponding to the second operation button, adding the second tag information to the image captured by the camera and transmitting the image to the image processing device to the external terminal. The program according to claim 17 or 18, characterized in that The image modification step of performing a deformation process on the image when a predetermined condition including that the first tag information is added to the image received from the external terminal is satisfied. The program described in. An image analysis step of analyzing an image received from the external terminal;
21. The program according to claim 20, wherein the predetermined condition includes that an image received from the external terminal is determined to be a photograph of a document by analysis of the image analysis step. An image analysis step of analyzing an image received from the external terminal;
When the analysis determines that the image received from the external terminal is a photograph of a document, an image deformation step for performing a deformation process on the image;
The program according to claim 17 or 18, further comprising: In the image analysis step, when a subject in the image has a quadrangular boundary and a character image is present inside the quadrangular boundary, it is determined that the image is a photograph of a document. The program according to claim 21 or 22. The camera can photograph a subject in three dimensions,
The program according to claim 21 or 22, wherein, in the image analysis step, when the subject in the image is a flat object, it is determined that the image is a photograph of a document. The control unit of the image processing apparatus functions as a Web server,
Each of the operation panel and the external terminal is a Web client that requests operation screen display data from the Web server.
25. The program according to any one of claims 14 to 24, wherein: The program according to any one of claims 14 to 25, wherein the display data is transmitted by an HTTP protocol. In the first display step, display data in which an instruction to operate the operation panel is embedded to the operation panel so as to notify the control unit of acceptance of the operation when the first operation button is operated. Send
In the function execution step, when the notification is received from the operation panel, the first function is executed using the first resource of the image processing apparatus,
In the second display step, when the first operation button is operated, the external resource is used so that the second resource of the external terminal is used instead of the first resource for executing the first function. Display data embedded with an instruction to operate the terminal is transmitted to the external terminal, and the second display step is performed when the external terminal receives an operation on the first operation button in the function execution step. The program according to claim 14, wherein the program fulfills the function of a part that executes the first function using a second resource of the external terminal instead of the first resource.