JP2012137525A - Image forming apparatus, and display device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image forming apparatus that guides a user to operate the apparatus in an appropriate procedure in a state where operation by a user is required.SOLUTION: When an image forming apparatus 100 determines that a target place is included in a picked-up image, a display device 400 controls an operation display part 401 to display an operation content indicated in an operation step to be superimposed on the picked-up image. When the image forming apparatus 100 determines that a target place is not included in the picked-up image, the display device 400 controls the operation display part 401 to display a content instructing a direction of movement for allowing a camera 304 to pick up an image of the target place, to be superimposed on the picked-up image.

Description

  The present invention relates to an image forming apparatus and a display apparatus, and more particularly to an image forming apparatus and a display apparatus that are operated by a user.

  Conventionally, when an original or paper jam occurs in an image forming apparatus such as a copying machine or a printer, the user is notified of the occurrence of the paper jam and at the same time the image forming operation is stopped. Then, a paper jam processing procedure is displayed on the display device in order to remove the jammed paper. The user removes the jammed paper according to the display, but there is a problem that it is difficult for a user unfamiliar with the operation to determine where the paper is jammed and to properly remove the jammed paper.

  On the other hand, a video object indicating a paper is generated and a video object display unit for displaying the video object is provided. The video object corresponding to the paper remaining in the machine and the additional video necessary for processing the jammed paper Display the object. An image forming apparatus has been proposed in which a user grasps a location to be processed and an internal structure using these video objects (see, for example, Patent Document 1).

JP 2008-201101 A

  According to the operation guidance display device described in Patent Document 1 described above, the video object for the paper is displayed in accordance with the direction in which the user is viewing, but all parts of the paper transport path are touched by the user. However, when a transport path is constituted by a plurality of transport units, the paper often remains across the plurality of transport units. For this reason, various processes corresponding to the remaining position of the paper are required, such as rotating the transport roller by hand before pulling out the transport unit.

  Accordingly, the additional video object corresponding to the paper jam removal process is displayed to the user, which may cause a new problem.

  Specifically, in order to handle a paper jam correctly, it is necessary to remove the paper in an appropriate order according to the remaining position of the paper, but the user who showed the paper tried to remove the paper in an incorrect procedure, for example, Inconveniences such as breaking the paper and causing a new paper jam occurred.

  In image forming apparatuses, there are many operations that require an appropriate procedure, such as toner replacement, as well as a paper jam.

  However, the conventional technology has a problem in that it is not possible to guide the user to perform an operation in an appropriate procedure when the user needs to perform an operation.

  An object of the present invention is to provide an image forming apparatus and a display device that can guide a user to perform an operation in an appropriate procedure when the user needs to perform an operation.

  In order to achieve the above object, an image forming apparatus according to a first aspect of the present invention is an image forming apparatus that communicates with a display device that includes an imaging unit and a display unit that displays an image. A state detecting unit for detecting that a state requiring an operation, a setting unit for setting an operation step indicating an operation content and an operation procedure of the user corresponding to the state detected by the state detecting unit, A receiving unit that receives a captured image captured by the imaging unit from the display device, and a captured image that is received by the receiving unit as a target location for an operation indicated in the operation procedure set by the setting unit. When the determination unit determines that the target location is included in the captured image, the determination unit determines whether the target part is included in the captured image. When the operation content is transmitted to the display device, and it is determined that the target location is not included in the captured image, the display device is used to instruct the moving direction of the imaging unit so that the target location can be imaged. And a transmission means for transmitting to.

  ADVANTAGE OF THE INVENTION According to this invention, when it will be in the state which requires a user's operation, a user can be guide | induced to operate in an appropriate procedure.

1 is a diagram illustrating an electrical configuration of an image forming apparatus according to an embodiment of the present invention. 1 is a diagram illustrating a mechanical configuration of an image forming apparatus according to an embodiment of the present invention. It is a figure which shows the electrical constitution of the display apparatus in FIG. It is a figure which shows the external appearance of the display apparatus in FIG. 1, (A) shows the perspective view seen from the front, (B) has shown the perspective view seen from the back surface. FIG. 2 is a diagram illustrating an area directly accessible by a user and a conveyance path in the image forming apparatus of FIG. 1. FIG. 6 is a diagram illustrating a state where three sheets of paper are jammed as an example of a paper jam. 3 is a flowchart illustrating a procedure of guidance processing executed by the image forming apparatus in FIG. 1. An image to be superimposed on the captured image is shown, (A) shows the target location and the state of the paper, and (B) shows the moving direction. It is a figure which shows the image which displayed the object location by emphasizing the position of the cover which should be opened by blinking. It is a figure which shows the image which displayed only the arrow, without displaying a residual paper. FIG. 1 is a diagram illustrating a head mounted display that can be used as a display device.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a diagram showing an electrical configuration of an image forming apparatus 100 according to an embodiment of the present invention.

  In FIG. 1, an image forming apparatus 100 includes a CPU 101, a ROM 102, a RAM 103, an operation unit 104, a document reading unit 105, an external interface 106, an image processing unit 107, an image forming unit 108, a fixing unit 109, a paper feeding unit 110, and a conveyance unit. 111, a terminal communication unit 112, a captured image processing unit 113, and a computer graphics image generation unit 114.

  The CPU 101 controls the entire image forming apparatus 100. The ROM 102 is a storage device that stores control contents to be executed by the CPU 101. A RAM 103 is a storage device used as a work area necessary for the CPU 101 to control the image forming apparatus 100. In addition to the work area of the CPU 101, the RAM 103 has an image obtained by the original reading unit 105 reading the original or a computer graphics image generated by the computer graphics image generation unit 114 (hereinafter simply referred to as “image”). Is stored). The operation unit 104 is for setting a copy job that the user wants to execute for the image forming apparatus 100. In addition to the operation unit 104, the operation unit 104 can execute image forming processing via the external interface 106. is there.

  The document reading unit 105 reads the image of the document placed on the document table of the image forming apparatus 100 according to the setting from the operation unit 104, digitizes it, and stores it in the RAM 103. The external interface 106 is connected to a network such as TCP / IP, and receives an instruction to execute an image forming process from a computer connected to the network, or notifies the computer of information in the image forming apparatus 100. Can do.

  The image processing unit 107 should perform the necessary image processing on the digital image in accordance with the copy job setting such as single-sided / double-sided printing setting from the operation unit 104 and the content of the read image from the document reading unit 105. A digital image is stored in the RAM 103.

  The image forming unit 108 forms a toner image from an image stored in the RAM 103 and transfers the toner image onto paper to form an image. The fixing unit 109 permanently fixes the toner image on the paper by heating and pressing the toner transferred on the paper in the image forming unit 108 and the paper. The paper stored in the image forming apparatus 100 is fed by the paper feeding unit 110 and subsequently conveyed by the conveying unit 111 to the image forming unit 108 and the fixing unit 109 to transfer and fix the toner image. The paper is discharged as it is, or is transported again toward the image forming unit 108 and the fixing unit 109 by the transport unit 111 in order to form a second image.

  The terminal communication unit 112 is for performing communication with the display device 400, and performs communication in a configuration that does not use a wired connection such as a wireless LAN, but may be wired. The captured image processing unit 113 processes an image captured by the camera in the display device 400 received by the terminal communication unit 112. The computer graphics image generation unit 114 generates an image according to the content of the image processed by the captured image processing unit 113 and displays the image on the display unit of the display device 400 via the terminal communication unit 112.

  FIG. 2 is a diagram showing a mechanical configuration of the image forming apparatus 100 according to the embodiment of the present invention.

  In FIG. 2, an automatic document feeder (ADF) 280 automatically conveys a document. The scanner 202 includes a document illumination lamp 203, a scanning mirror 204, and the like. The scanner 202 is reciprocally scanned in a predetermined direction by a motor (not shown), and the reflected light from the document on the platen glass 201 is transmitted through the lens 207 via the scanning mirrors 204 to 206, and reaches the CCD sensor in the image sensor unit 208. To form an image.

  The exposure control unit 210 is constituted by a laser, a polygon scanner, or the like. In the exposure control unit 210, the image of the document is converted into an image signal by the image sensor unit 208, and the photosensitive drum 211 is irradiated with laser light 219 modulated based on the image signal on which predetermined image processing has been performed. Around the photosensitive drum 211, a primary charger 212, a developing device 213, a transfer roller 216, a pre-exposure lamp 214, and a cleaning device 215 are provided. After the photosensitive drum 211 is charged to a desired potential by the primary charger 212, the laser beam 219 from the exposure control unit 210 is irradiated to form an electrostatic latent image. The electrostatic latent image formed on the photosensitive drum 211 rotated in the direction of the arrow shown in the figure by a motor (not shown) is developed by the developing device 213 and visualized as a toner image. In the present embodiment, forming an image on paper may be simply expressed as “printing”.

  Paper fed from the right cassette deck 221, the left cassette deck 222, the upper cassette 223 or the lower cassette 224 by the pickup rollers 225, 226, 227, and 228 is sent to the conveyance path by the paper feeding rollers 229, 230, 231, and 232. It is done. Subsequently, the toner image is sent to the transfer unit by the registration roller 233 and the toner image formed on the photosensitive drum 211 is transferred onto the paper by the transfer roller 216. After the transfer, the photosensitive drum 211 is cleaned of residual toner by the cleaning device 215, and the residual charge is erased by the pre-exposure lamp 214.

  The transferred paper is separated from the photosensitive drum 211 by the separation charger 217 and sent to the fixing device 235 as it is by the transport belt 234. The paper sent to the fixing device 235 is fixed with toner by pressure and heating, and is discharged out of the image forming apparatus 100 by the inner discharge roller 236 and the discharge roller 244.

  The image forming apparatus 100 is equipped with a deck 250 that can store 3500 sheets of paper. The lifter 251 of the deck 250 ascends according to the amount of paper so that the paper always abuts against the pickup roller 252, and the paper is sent to the transport path in the image forming apparatus 100 by the paper feed roller 253.

  Further, the image forming apparatus 100 is equipped with a multi-manual feed 254 that can accommodate 50 sheets of paper, and this is also transported to the registration roller 233 by a paper feed roller 255 and a transport roller 256.

  The paper discharge flapper 237 switches the path between the paper discharge unit transport path 238 side and the discharge path 243 side. The lower transport path 240 guides the paper fed from the inner paper discharge roller 236 upside down through the reverse path 239 and leads it to the refeed path 241. The paper fed from the left cassette deck 222 by the paper feed roller 230 is also guided to the refeed path 241. The refeed roller 242 refeeds the paper to the image forming unit 108. The discharge roller 244 is disposed in the vicinity of the paper discharge flapper 237 and discharges the paper switched to the discharge path 243 side by the paper discharge flapper 237 to the outside of the apparatus. During double-sided recording (double-sided copying), the paper discharge flapper 237 is raised upward, and the copied paper is guided to the refeeding path 241 through the reverse path 239 and the lower transport path 240.

  At this time, the reversing roller 245 pulls the reversing path 239 to the rear end of the paper, and reverses the reversing roller 245 to send it to the lower transport path 240. When the paper is reversed and discharged from the main body, the paper discharge flapper 237 is raised upward, the reverse roller 245 leaves a part of the rear end of the paper and is drawn into the reverse path 239, and the reverse roller 245 is reversely rotated. Is turned over and sent to the discharge roller 244 side.

  The paper discharge processing apparatus 290 aligns and closes the paper discharged from the image forming apparatus 100, and stacks the paper discharged for each sheet on the processing tray 294 to align them. When the discharge due to partial image formation is completed, the paper bundle is stapled and discharged to the paper discharge tray 292 or 293 as a bundle.

  The paper discharge tray 293 is controlled to move up and down by a motor (not shown), moves to the position of the processing tray 294 before starting the image forming process, and then the height of the paper surface is reached when discharged paper is stacked. Moves to the position of the processing tray 294. Further, the sheet tray 291 stacks separator sheets to be inserted between the discharged sheets, and the Z folding machine 295 performs Z folding on the discharged sheets. Further, the bookbinding machine 296 performs bookbinding by collectively folding a part of the discharged paper and performing stapling, and the bound paper bundle is discharged to a discharge tray 297.

  FIG. 3 is a diagram showing an electrical configuration of the display device 400 in FIG.

  The display device 400 includes a CPU 301, ROM 302, RAM 303, camera 402, external communication control unit 305, and operation display unit 401.

  In FIG. 3, the CPU 301 controls the entire display device 400. The ROM 302 is a storage device that stores control contents to be executed by the CPU 301. The RAM 303 is a storage device used as a work area necessary for the CPU 301 to control the display device 400. The RAM 303 stores not only the work area of the CPU 301 but also image information read by the camera 402, This is used for combining the image received via the external communication control unit 305 and the image information captured by the camera 402.

  The camera 402 is used to capture an image around the display device 400 and can capture a moving image. The external communication control unit 305 performs wireless communication with the terminal communication unit 112 in order to communicate with the image forming apparatus 100. The operation display unit 401 is a touch panel capable of displaying images, and displays image information combined with the image information captured by the camera 402 and the image generated by the computer graphics image generation unit 114.

  4A and 4B are views showing the appearance of the display device in FIG. 1, wherein FIG. 4A shows a perspective view seen from the front, and FIG. 4B shows a perspective view seen from the back.

  In FIG. 4A, the display device 400 includes an operation display unit 401. The operation display unit 401 is a touch panel as described above, and the user can switch the display image by touching the surface according to the displayed image. The operation display unit 401 displays information regarding the image forming apparatus 100 as shown in FIG.

  In FIG. 4B, the display device 400 includes a camera 402 and a flash 403. The imaging direction of the camera 402 is made to coincide with the front direction of the operation display unit 401, and the captured image is displayed on the operation display unit and also to the image forming apparatus 100 via a communication antenna (not shown). Has been sent.

  FIG. 5 is a diagram illustrating an area that can be directly accessed by the user and a conveyance path in the image forming apparatus 100 of FIG.

  In FIG. 5, the directly accessible areas are a multi-manual feed area 601, a fixing transport area 602, a lower transport area 603, a left deck area 604, an upper cassette area 605, a lower cassette area 606, a lower right cover area 607, and a right deck area. 608.

  The paper jam processing position will be described with reference to FIG.

  The paper remaining in the image forming apparatus 100 is taken out by the user directly accessing the transport path in the apparatus by operating the cover that can be opened and closed and the detachable unit of the image forming apparatus 100. A right deck area 608, a left deck area 604, an upper cassette area 605, and a lower cassette area 606 are areas that can be accessed by pulling out a paper feed tray storing paper in the front direction of the drawing.

  A multi manual feed area 601 and a lower right cover area 607 are areas that allow access to the transport path by opening the right cover of the main body. A multi-manual feed 254, a paper feed roller 255, and a transport roller 256 are attached to the right cover of the image forming apparatus 100, and are connected to the main body by a hinge (not shown) on the back side. As a result, by moving the front side of the right cover in the rightward direction, the hinge can be opened and closed with the pivoting operation as a fulcrum.

  Similarly, by opening the lower right cover with a hinge (not shown) on the back side, the right cassette deck 221, the upper cassette 223, the lower cassette 224, and the transport path from the refeed path 241 to the registration roller 233 are accessed. Enable.

  The fixing conveyance area 602 and the lower conveyance area 603 allow access to the conveyance path by opening a front door (not shown) and further pulling out units provided with rails on both sides.

  The fixing conveyance unit includes a registration roller 233, a conveyance belt 234, a fixing device 235, an internal paper discharge roller 236, a paper discharge roller 244, and the like, and constitutes a fixing conveyance area 602. In the lower transport unit, a paper discharge unit transport path 238, a reverse path 239, a lower transport path 240, a re-feed path 241, a re-feed roller 242 and the like are placed, and a lower transport area 603 is configured. A guide cover (not shown) is provided in the lower transport unit, and the reversing path 239 is raised upward by raising the guide cover to allow access to the lower transport path 240 from above.

  FIG. 6 is a diagram illustrating a state where three sheets of paper are jammed as an example of a paper jam.

  In FIG. 6, the stop positions and processing methods for the three remaining sheets are as follows. The first sheet spans the fixing conveyance area 602, the lower right cover area 607, and the lower conveyance area 603.

  The fixing conveyance area 602 and the lower conveyance area 603 at the leading edge and trailing edge of the paper can be accessed by pulling them forward. Since the lower right cover area 607 in the center is an area that can be accessed by releasing the right side, the paper processing method is to pull out the leading edge and the trailing edge of the paper from the lower right cover area 607. It can be removed without breaking the paper.

  Since all the second sheets are included in the lower transfer area 603, they can be removed by pulling out the lower transfer area 603.

  The third sheet spans the lower transfer area 603 and the left deck area 604 and an area where direct access between them is not possible. The fixing conveyance area 602 and the left deck area 604 at the leading edge of the paper can be accessed by pulling them forward, and the area at the center is an area where the drawer action cannot be performed. However, with respect to the left deck area 604, since no paper is sandwiched and only the rear end of the paper is on top of the paper stored in the left cassette deck, the left cassette deck is pulled out, By removing the paper remaining from the back in the left direction, it can be removed without breaking the paper.

Therefore, the user operation steps for a paper jam are as follows.
1. Open the lower right cover. Pull out the 2.1st sheet. 4. Open the left cassette deck. 4. Pull out the third sheet. Pull out the lower transport unit. 6. Take out the second sheet of paper. 7. Replace the lower transport unit. 8. Close the left cassette deck. The image forming apparatus 100 that closes the lower right cover guides the user to perform this operation.

  FIG. 7 is a flowchart showing a procedure of guidance processing executed by the image forming apparatus 100 of FIG.

  The guidance process in FIG. 7 is executed by the CPU 101 of the image forming apparatus 100 and the CPU 301 of the display apparatus 400.

  Note that the guidance processing in FIG. 7 will be described using the paper jam described in FIG. 6 as an example. Specifically, the paper jam in FIG. 6 occurred between the re-feed roller 242 and the registration roller 233 during the single-side continuous copying operation using the paper stored in the left cassette deck 222. Is.

  In FIG. 7, a paper jam state is first detected (step S700) (state detection means), and the paper before the paper jam occurs is printed normally (step S701). In this case, when the paper jam occurs, printing is performed as it is on the paper whose rear end has already passed the registration roller 233. On the other hand, for the paper that is moving on the transport path from the left cassette deck 222 to the registration roller 233, the driving of the refeed roller 242, the feed roller 230, and the refeed path 241 between them is stopped. This is left in the image forming apparatus 100 (step S702).

  An example of the paper stop position at this time is as shown in FIG. After stopping the driving of the roller, in order to notify the user of the occurrence of a paper jam, a warning message of the occurrence of a paper jam is displayed on both the operation unit 104 and the operation display unit 401 of the display device 400 (step S703). Next, an operation step indicating the operation content and operation procedure of the user corresponding to the state detected in step S700 is set (step S704) (setting means). Examples of operation steps are as described in FIG. Next, a target location is determined (step S705). In this case, since “1. Open the lower right cover” (hereinafter referred to as “1”), which is the head of the order, the target location is determined as the lower right cover.

  Next, the display device 400 transmits an image captured by the camera in the display device 400 to the image forming device 100 (a transmission unit in the display device 400), and receives a captured image from the display device 400 (in the image forming device 100). Receiving means), the captured image is processed by the captured image processing unit 113, and an image captured by the camera in the display device 400 is taken out (step S706). Then, it is determined whether or not the target portion is included in the image (step S707) (determination means). In step S707, it may be further determined whether or not the image is taken from a predetermined direction corresponding to the target portion. The predetermined direction is, for example, the lower right cover is a target portion in this case, and therefore the normal direction when the lower right cover is regarded as a plane (normal line toward the outside rather than the inside of the image forming apparatus 100). Direction), and a direction entering an interval where the angle formed with the normal line is about 0 ° to 45 °.

  As a result of the determination in step S707, when the target location is included, or when the target location is included and the image is taken from a predetermined direction corresponding to the target location (YES in step S707), the display device A predetermined image is transmitted to 400 (step S708) (transmission means in the image forming apparatus 100). In this case, the predetermined image is an image showing the position or shape of the jammed paper. Specifically, as shown in FIG. 8A, an image in which the paper remains inside the lower right cover. It is. At this time, the size of the image to be transmitted is adjusted to the size of the remaining paper, and the first paper is the cause of the paper jam. When you see, you can avoid the feeling of strangeness. In the case of “1”, if the lower right cover of the image forming apparatus 100 is imaged at a sufficient distance from the right side of the image forming apparatus 100, an affirmative determination may be made in step S707.

  Next, the display device 400 receives the predetermined image (receiving means in the display device 400), and controls the operation display unit 401 of the display device 400 to display the operation content superimposed on the captured image (step). S709) (control means), the process proceeds to step S713.

  As a result of the determination in step S707, when the target portion is not included in the image, or even when a portion is included in the image, the image is not picked up from a predetermined direction corresponding to the target portion (step S707). NO), the taken image is analyzed, and it is determined which direction around the image forming apparatus 100 the camera of the display device 400 is imaging (step S710). Then, in order to guide the operation of “1”, the content indicating the moving direction of the camera as shown in FIG. 8B showing the direction from this position to the lower right cover is transmitted to the display device 400 as an image. (Step S711) (transmission means in the image forming apparatus 100). In step S711, a predetermined image, that is, an image indicating the position or shape of the jammed paper is not transmitted to the display device 400. The display device 400 receives the content instructing the moving direction of the camera (receiving means in the display device 400), and superimposes the content instructing the moving direction of the camera on the captured image on the operation display unit 401 of the display device 400. The display is controlled (step S712) (control means), and the process returns to step S707. As described above, when the content indicating the moving direction is received, the image indicating the position or shape of the jammed paper is not received. The operation display unit 401 is controlled not to display an image indicating the position or shape. This is because, when an image indicating the position or shape of the paper is displayed in a state where the portion to be operated is not imaged, a user may take a wrong procedure to remove the paper indicated in the displayed image (for example, another This is because the paper may be torn if the operation is started (with the cover open).

  In accordance with this guidance, when the user moves so that the lower right cover to be operated is viewed from the right side of the image forming apparatus 100, an affirmative determination is made in step S707, and an image image in which paper remains is displayed on the display apparatus 400. You will be notified. It should be noted that a method for determining whether or not a desired image is picked up at a desired distance and direction from among the images, and for generating a residual image of paper or the like in the computer graphics image generating unit 114 are general image processing. Therefore, detailed explanation is omitted.

  In addition, the image transmitted from the image forming apparatus 100 to the display device 400 in step S708 and step S711 is combined with the captured image captured by the camera 402 in the display device 400 and displayed on the operation display unit 401. The The image transmitted in S708 is synthesized according to the position of the lower right door of the main body, and as shown in FIG. 9, the target location is displayed by highlighting the position of the cover to be opened by blinking. The position of the cover to be opened and the position of the paper remaining in the machine can be notified. The image transmitted in step S711 does not display the remaining paper as shown in FIG. 10 because the cover to be opened is not visible from the front regardless of the position of the paper remaining in the image forming apparatus 100. Only arrows are displayed.

  Next, it is determined whether or not a desired operation has been performed (step S713). In this case, since the desired operation is “1”, it is determined that the desired operation has been performed when the user opens the lower right cover. As a result of the determination in step S713, when it is determined that the desired operation has not been performed, the process returns to step S707.

  As a result of the determination in step S713, when the camera 402 of the display device 400 captures that the user has opened the lower right cover, and the captured image processing unit 113 of the image forming apparatus 100 detects this (YES in step S713). It is determined whether or not there is a next operation step (step S714).

  As a result of the determination in step S714, when there is a next operation step (NO in step S714), the operation step is advanced (step S715), and the process returns to step S705. In this case, assuming that the operation “1” is completed, the operation step is incremented by one, and the new operation step is changed to “2.1 pull out the second sheet”.

  If the result of determination in step S714 is that there is no next operation step (YES in step S714), the paper jam has been cleared, so a recovery process from the paper jam is executed (step S716), and the printing operation is resumed (step S716). Step S717), the process is terminated.

  In step S715, after the new operation is advanced to “draw the 2.1st sheet”, a target location corresponding to “draw the 2.1st sheet” is determined (step S715). S705). Specifically, the target location is determined as the first sheet of paper that should be visible with the lower right cover open. As processing thereafter, an image indicating an arrow indicating the paper drawing direction is generated, and when it is detected that the paper is pulled out from the transport path and no paper remains, “2.1 sheets” is detected. The operation step of “drawing out the eye paper” is terminated. As described above, by performing the operation steps 1 and 2, the residual image of the paper can be displayed only when the user is guided in the paper drawing direction. Then, the user can correctly remove the first sheet.

  Thereafter, by repeating the processes from S705 to S715 for the above-described nine operation steps, the user can operate in an appropriate procedure.

  According to the processing in FIG. 7, when the image forming apparatus 100 determines that the target portion is included in the captured image (YES in step S707), the display device 400 superimposes the operation content indicated in the operation step on the captured image. When the image forming apparatus 100 determines that the target location is not included in the captured image (NO in step S707), the display apparatus 400 Since the operation display unit 401 is controlled so as to superimpose and display the content instructing the moving direction of the camera 304 on the captured image so that the location can be imaged (step S712), the user needs to perform an operation. The user can be guided to operate in an appropriate procedure.

  Accordingly, by switching to an image generated by the computer graphics image generation unit 114 based on an image captured by the display device 400, the user can be guided to an appropriate procedure in the paper jam processing. . As a result, the jammed paper can be easily removed.

  FIG. 11 is a diagram showing a head mounted display 1300 that can be used as the display device 400 in FIG.

  In FIG. 11, a head mounted display 1300 includes a video display unit 1301, an external camera 1302, ear pads 1303 and 1304, a line-of-sight detection camera 1305 (line-of-sight detection means), and an antenna (not shown).

  The video display unit 1301 is translucent. When the user wears the head-mounted display 1300 from the head, the video that is seen through the video display unit 1301 and the computer graphics image generation unit 114 sent from the image forming apparatus 100 are displayed. It is possible to see a composite of the images generated by.

  Further, the line-of-sight detection camera 1305 faces in the opposite direction to the external camera 1302, and the user's line-of-sight direction can be detected by detecting the position of the user's pupil.

  Therefore, by transmitting the image captured by the external camera 1302 from the head mounted display 1300 to the image forming apparatus 100 and the position of the pupil detected by the line-of-sight detection camera, the image forming apparatus 100 allows the user to use the external camera. It is determined which direction in the image from 1302 is being viewed.

  As a result, it is possible to extract an image necessary for determining the paper jam processing more accurately than when the display device 400 shown in FIG. 4 is used.

  Specifically, in step S707 in FIG. 7, it is determined whether or not the target portion is included in the line-of-sight direction. In step S707, it may be further determined whether or not the line-of-sight direction is a predetermined direction. The predetermined direction is, for example, the lower right cover is a target portion in this case, and therefore the normal direction when the lower right cover is regarded as a plane (normal line toward the outside rather than the inside of the image forming apparatus 100). Direction), and a direction entering an interval where the angle formed with the normal line is about 0 ° to 45 °.

In the present exemplary embodiment, the operation corresponding to the state where the image forming apparatus 100 is jammed is taken as an example of the state requiring the user's operation. However, the toner bottle of the image forming apparatus 100 needs to be replaced. It may be a state. In this case, user operation steps are as follows, for example.
1. Open the front cover 2. 2. Turn the knob of the toner container to be replaced to the right. 3. Hold the handle and pull out the toner container. 4. Hold a new toner container and shake it lightly so that it is upside down. 5. Push the toner container all the way in. 6. Close with front cover Confirming a message displayed on the touch panel display In the above operation step, an image showing a procedure necessary for replacement or a position of a cover necessary for replacement of the toner bottle may be formed.

  Furthermore, an image corresponding to the replacement work of the image forming apparatus may be displayed that is mainly intended for a service person who performs maintenance on the image forming apparatus as well as the target for the user.

  Further, in the present embodiment, the target to be jammed in the image forming apparatus 100 is paper, but the present embodiment can be applied to any recording medium that passes through the conveyance path in the image forming apparatus 100 without being limited to paper. is there.

(Other embodiments)
The present invention is also realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program code. It is a process to be executed. In this case, the program and the storage medium storing the program constitute the present invention.

101,301 CPU
102,302 ROM
103,303 RAM
304, 1302 Camera 400 Display device 401 Operation display unit 1300 Head mounted display 1301 Video display unit 1305 Eye-gaze detection camera

Claims (16)

An image forming apparatus that communicates with a display device that includes an imaging unit and a display unit that displays an image,
State detecting means for detecting that the image forming apparatus is in a state requiring an operation by a user;
Setting means for setting an operation step indicating the operation content and operation procedure of the user corresponding to the state detected by the state detection means;
Receiving means for receiving a picked-up image picked up by the image pickup means from the display device;
A discriminating unit that discriminates whether or not a target location that is a target of an operation indicated in the operation procedure set by the setting unit is included in the captured image received by the receiving unit;
When the determination unit determines that the target location is included in the captured image, the operation content indicated in the operation step is transmitted to the display device, and it is determined that the target location is not included in the captured image. An image forming apparatus comprising: a transmission unit configured to transmit to the display device a content for instructing a moving direction of the imaging unit so that the target portion can be imaged. An image forming apparatus that communicates with a display device that includes a line-of-sight detection unit that detects a user's line-of-sight direction and a display unit that displays an image.
State detecting means for detecting that the image forming apparatus is in a state requiring an operation by the user;
Setting means for setting an operation step indicating the operation content and operation procedure of the user corresponding to the state detected by the state detection means;
Receiving means for receiving the line-of-sight direction detected by the line-of-sight detection means from the display device;
A discriminating unit that discriminates whether or not the target location that is the target of the operation indicated in the operation procedure set by the setting unit is included in the line-of-sight direction received by the receiving unit;
When the determination unit determines that the target location is included in the detected line-of-sight direction, the operation content indicated in the operation step is transmitted to the display device, and the target location is detected in the line-of-sight direction. Transmission means for transmitting, to the display device, contents for instructing a moving direction of the line-of-sight direction so that the target portion is included in the line-of-sight direction when it is determined that the target part is included in the line-of-sight direction. Image forming apparatus.   The image forming apparatus according to claim 1, wherein the transmission unit transmits the target portion in addition to the operation content indicated in the operation step.   4. The image forming apparatus according to claim 1, wherein the state requiring an operation by the user is a state in which the image forming apparatus is jammed. 5.   The image forming apparatus according to claim 4, wherein the transmission unit transmits an image indicating a position or shape of a paper jam in addition to the operation content indicated in the operation step.   The image forming apparatus according to claim 1, wherein the state that requires an operation by the user is a state that requires replacement of a toner bottle of the image forming apparatus. A display device comprising an imaging means and a display means for displaying an image, and communicating with an image forming apparatus,
A transmission unit that transmits a captured image captured by the imaging unit to the image forming apparatus;
The imaging unit is configured so that the user's operation content corresponding to a state in which the image forming apparatus requires an operation by the user and a target portion that is an operation target indicated in the user operation procedure can be imaged. Receiving means for receiving any one of the contents instructing the moving direction;
And a control means for controlling the display means so as to superimpose and display the content received by the receiving means on the captured image. A display device that includes a line-of-sight detection unit that detects a user's line-of-sight direction and a display unit that displays an image, and communicates with the image forming apparatus,
Transmitting means for transmitting the line-of-sight direction detected by the line-of-sight detection means to the image forming apparatus;
The user's operation content corresponding to a state in which the image forming apparatus requires an operation by the user, and the target portion that is the target of the operation indicated in the user's operation procedure is included in the line-of-sight direction. Receiving means for receiving any one of the contents instructing the movement direction of the line-of-sight direction;
And a control means for controlling the display means so as to display the content received by the receiving means.   The display device according to claim 7, wherein the receiving unit receives the target portion in addition to the operation content.   The display device according to claim 7, wherein the state that requires an operation by the user is a state in which the image forming apparatus is jammed.   The display device according to claim 10, wherein the receiving unit transmits an image indicating a position or a shape of a jammed paper in addition to the operation content.   12. The display device according to claim 7, wherein the state that requires an operation by the user is a state that requires replacement of a toner bottle of the image forming apparatus. An image forming apparatus that communicates with a display device that includes an imaging unit and a display unit that displays an image,
State detecting means for detecting that the image forming apparatus is in a jammed state;
Setting means for setting a user operation procedure corresponding to the paper jam detected by the state detection means;
Receiving means for receiving a picked-up image picked up by the image pickup means from the display device;
The target location that is the target of the operation indicated in the operation procedure set by the setting means is included in the captured image received by the receiving means, and is imaged from a predetermined direction corresponding to the target location. Determining means for determining whether or not
When the determination unit determines that the target location is included in the captured image and is captured from a predetermined direction corresponding to the target location, the position or shape of the jammed paper is indicated. When the image is transmitted to the display device and the target location is included in the captured image and it is determined that the image is not captured from a predetermined direction corresponding to the target location, the jammed paper An image forming apparatus comprising: a transmission unit that does not transmit an image indicating the position or shape of the image to the display device. An image forming apparatus that communicates with a display device that includes a line-of-sight detection unit that detects a user's line-of-sight direction and a display unit that displays an image.
State detecting means for detecting that the image forming apparatus is in a jammed state;
Setting means for setting an operation procedure of the user corresponding to a paper jam detected by the state detection means;
Receiving means for receiving the line-of-sight direction detected by the line-of-sight detection means from the display device;
The target location that is the target of the operation indicated in the operation procedure set by the setting means is included in the line-of-sight direction received by the receiving means, and the line-of-sight direction is predetermined corresponding to the target location. Discriminating means for discriminating whether or not the direction,
When the determining unit determines that the target portion is included in the line-of-sight direction detected by the line-of-sight detecting unit and the line-of-sight direction is a predetermined direction corresponding to the target portion, the paper jam An image showing the position or shape of the paper is transmitted to the display device, and even if the target location is included in the gaze direction detected by the gaze detection means, the gaze direction corresponds to the target location in advance. An image forming apparatus comprising: a transmission unit that does not transmit an image indicating a position or shape of the jammed paper to the display device when it is determined that the direction is not the specified direction. A display device comprising an imaging unit and a display unit for displaying an image, wherein the display device communicates with the image forming apparatus according to claim 13,
A transmission unit that transmits a captured image captured by the imaging unit to the image forming apparatus;
Receiving means for receiving an image indicating the position or shape of the jammed paper;
And a control means for controlling the display means so as to superimpose and display the content received by the receiving means on the captured image. A display device comprising: a line-of-sight detection unit that detects a user's line-of-sight direction;
Transmitting means for transmitting the line-of-sight direction detected by the line-of-sight detection means to the image forming apparatus;
Receiving means for receiving an image indicating the position or shape of the jammed paper;
And a control means for controlling the display means so as to display the content received by the receiving means.

Images