JP2015075832A - Ar display system, ar display device, information processor, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an AR display system, an AR display device, an information processor, and a program for enabling a user to easily find out target materials or components from among a plurality of articles placed in a preservation place by using an AR technology.SOLUTION: The information processor is configured to photograph the same scene (real space) as a scene which a user is viewing by a camera, and to determine whether or not an article exists in the scene, and to specify the position of the article. Then, when the article is an article lacking in a predetermined device, the AR display device displays an extended real space in which a predetermined mark is overlapped and synthesized with the article.

Description

  The present invention relates to an AR display system, an AR display device, an information processing device, and a program that provide information related to materials and parts to a user using AR technology.

  There is a method for realizing guidance or the like by superimposing an image on a field of view viewed by a user by an apparatus such as AR glasses using a so-called AR (Augmented Reality) technology or a head mounted display.

  For example, Patent Document 1 manages the inspection and replacement timing information of consumables in the vehicle based on the travel information of the vehicle, and uses the augmented reality (AR) technology to check and replace the consumables in the vehicle. A method for providing such information is disclosed (see Patent Document 1).

  Patent Document 2 discloses a technique for displaying an appropriate process and whether or not a part selected by an operator is an appropriate part for the process using a head-mounted display when an assembly operation is performed in a plurality of steps. It is disclosed (see Patent Document 2).

JP 2012-043396 A JP 2010-2111623 A

  By the way, when a multifunction machine (MFP (Multi Function Peripheral)) such as an image forming apparatus having functions such as a printer, a scanner, a facsimile machine, etc., it is generally necessary to replenish materials or replace parts. Go to pick up the materials and parts at the material storage.

  However, the user may not be able to determine which material or part is the target in the material storage area. For example, even if the user goes to the material storage area for the purpose of replenishing A4 paper to the MFP, the user forgets the size of the paper to be replenished before reaching the material storage area, or there are various types of paper in the material storage area. A case may be considered where it is not clear which paper is A4 paper.

  In the technique described in Patent Document 1, when the user looks at the apparatus, it can be known where the material is insufficient, but the detailed type of the insufficient material cannot be grasped. For example, when a user looks at a vehicle with a decrease in engine oil, it is displayed that the engine oil is insufficient due to the AR technology. However, when a plurality of engine oils are prepared in the material storage area, the user must determine which engine oil is suitable for his vehicle.

  In the technique described in Patent Document 2, it is displayed whether or not the part being watched by the user is an appropriate part in the current level of assembly work. Therefore, when this technology is applied to a scene where a target material is searched for from among a large number of materials in the material storage area, if the user does not pay close attention to each material, whether the material is the corresponding material. Since it is not known whether or not, it takes a great deal of time to search for the target material from among a large number of materials.

  The present invention is intended to solve the above-described problem, and a user can easily find a target material or part from a large number of items placed in a storage place by using AR technology. An object of the present invention is to provide an AR display system, an AR display device, an information processing device, and a program.

  The gist of the present invention for achieving the object lies in the inventions of the following items.

[1] An AR display device including an AR display unit that displays an augmented reality space obtained by synthesizing an image with a real space viewed by the user, and an imaging unit that captures the real space viewed by the user, and the AR display An AR display system comprising an information processing device capable of communicating with a device,
The AR display device transmits photographing data obtained by photographing by the photographing unit to the information processing device,
The information processing apparatus acquires management information indicating at least one item of a material that needs replenishment and a part that needs to be replaced in a predetermined device, and the information is received in the imaging data received from the AR display device If the item indicated by the management information exists, data for specifying the position of the existing item and causing the AR display device to display an augmented reality space in which an image of a predetermined mark is superimposed on the existing item and synthesized. Is generated and transmitted to the AR display device,
The AR display system displays an augmented reality space obtained by superimposing the image of the mark on the existing item based on the data.

  In the inventions described in [1] and [11] below, when the user sees a material storage area that is a storage place for items such as materials to be replenished to a predetermined device or parts to be replaced, the AR display device An augmented reality space is displayed in which an image of a predetermined mark is superimposed on a product (material or part) that is lacking in a predetermined device. As a result, the user can easily find out a shortage of items with a predetermined device from among a large number of items.

  As a method of determining whether or not the item indicated by the management information exists in the shooting data and specifying the position when the item exists, for example, image analysis is performed on the image indicated by the shooting data. A method or a method of registering the position of an item to be replenished or exchanged with a predetermined device in advance, specifying the position indicated by the photographing data with an orientation sensor or a GPS (Global Positioning System) function, and comparing the positions of both. is there.

[2] The information processing apparatus can acquire the management information from a plurality of apparatuses, acquires the identification information of the apparatus related to the management information together with the management information, and stores the information on the existing item as the data. Generates data for displaying on the AR display device an augmented reality space in which the identification information of the device corresponding to the management information indicating the existing item is further synthesized in the augmented reality space synthesized by superimposing the mark images The AR display system according to [1], characterized in that:

  In the inventions described in [2] and [12] below, management information indicating items (materials or parts) that are lacking in a predetermined device is acquired from a plurality of devices. Then, the AR display device displays the augmented reality space in which the identification information of the device lacking the article on which the image of the predetermined mark is superimposed is combined with the predetermined mark. As a result, the user can grasp which device is lacking the product on which the image of the predetermined mark is superimposed.

[3] The information processing apparatus uses the AR display device as an augmented reality space in which, as the data, the mark is made different for each device, and a mark corresponding to one device is associated with identification information indicating the device. The AR display system according to [2], wherein data to be displayed on the screen is generated.

  In the inventions described in [3] and [13] below, the display form of the predetermined mark is changed for each device, and the identification information of the device that lacks the item on which the mark is superimposed is associated. indicate. For example, the mark color is changed for each device and displayed, and the color indicating the identification information of each device is the same color as the mark corresponding to the device. Thereby, it can be grasped by which apparatus each item on which the predetermined mark is superimposed is insufficient.

[4] The management information includes urgency information indicating urgency when the item is needed.
The AR processing system according to [1], wherein the information processing apparatus changes a display form of the mark according to the urgency of an item corresponding to the mark.

  In the invention described in the above [4] and [14] below, display of a predetermined mark to be combined so as to overlap the item according to the urgency of the time when the item such as material or part is required in the predetermined device Different forms. Thereby, the user can grasp | ascertain the urgency of the time when the goods on which the image of the predetermined | prescribed mark was piled up are needed with a predetermined | prescribed apparatus. For example, it is possible to grasp whether the product is already in a shortage or shortly soon.

[5] When the item indicated by the management information exists outside the real space corresponding to the captured data received from the AR display device, the information processing device guides the position where the item exists. The AR display system according to any one of [1] to [4], wherein an augmented reality space obtained by combining information with a real space is displayed on the AR display device.

  In the inventions described in [5] and [15] below, if there is an item that is deficient in a predetermined device outside the range of the real space viewed by the user wearing the AR display device, the AR display device Then, the augmented reality space is displayed by synthesizing the guidance information for guiding the position where the item exists in the real space. Thereby, even if there is an item that is insufficient with the predetermined device outside the range that the user is looking at, the item can be easily found.

  As a specific means, for example, the position where each item is placed is registered in advance, and the position of the missing item is recognized by a predetermined device. Alternatively, when the user views the missing item with a predetermined device, the position of the item is specified from the shooting data. Then, the relative position between the position of the range indicated by the photographing data and the position of the product is recognized. Then, the AR display device displays guidance information for guiding the position where the item exists.

[6] When there is a mark whose size is less than or equal to a predetermined value when the mark is superimposed on each of the existing items, the information processing apparatus replaces at least the mark with the size less than or equal to the predetermined value. Any one of [1] to [5], wherein an augmented reality space obtained by combining an article corresponding to the mark and an augmented mark that overlaps a predetermined range around the mark with the actual space is displayed on the AR display device. AR display system described in 1.

  In the inventions described in [6] and [16] below, when an image of a predetermined mark is superimposed on an article in shooting data and the size of the mark is equal to or smaller than a predetermined value, the size is a predetermined value. Instead of the following marks, the AR display device displays an augmented reality space in which an image corresponding to the mark and an extended mark image that overlaps a predetermined range around the item are combined with the real space. Thus, for example, even if the user wearing the AR display device is far away from the product or the product itself is small, a mark indicating the product that is lacking in the predetermined device is displayed to the user. Displayed in an easy-to-understand manner. When displaying a plurality of marks, only a mark having a size equal to or smaller than a predetermined value among the plurality of marks may be replaced with an extension mark, or all the marks may be changed to an extension mark.

[7] The information processing apparatus includes:
(1) Always
(2) When the predetermined device exists in the photographing data received from the AR display device,
(3) When the AR display device is located in and around the storage location for the item,
(4) When a predetermined item exists in the photographing data received from the AR display device,
(5) When a transmission request for the management information is received from the predetermined device,
(6) The management information is acquired by communicating with the device at any timing when a job submission notification is received from the predetermined device to which a job is submitted [1] to [6] The AR display system according to any one of [6].

  In the invention described in [7] and [17] below, in the case of (1), the management information is periodically and repeatedly acquired at regular intervals. In addition, the predetermined item (4) indicates materials or parts specified in advance. In the case of (5), the management information may be received when it is transmitted from the image forming apparatus 10.

[8] The information processing device acquires position information of the AR display device, and a user of the AR display device passes through the storage location of the item indicated by the management information from the position indicated by the position information. Estimating the required travel time by adding a predetermined margin time to the time required to move to the installation location of the apparatus, and at least from the timing when the operation of the predetermined apparatus stops due to the lack of items indicated by the management information The AR display system according to any one of [1] to [7], wherein the user is warned of the shortage of the item at a time point before the travel time.

  In the inventions described in [8] and [18] below, the time until the position information of the AR display device is acquired and arrives at the predetermined device via the storage location of the item indicated by the management information. In addition, the required travel time is estimated by adding a predetermined margin time. Then, the user is warned of the shortage of the goods at least at the time before the movement required time from the timing at which the operation of the predetermined apparatus stops due to the lack of the goods indicated by the management information. Thus, immediately after the confirmation, the user who confirmed the warning goes to the storage location to procure the goods and go to the predetermined device, so that the shortage of the item is confirmed before the operation of the predetermined device stops. Can be resolved.

[9] The information processing apparatus acquires user information indicating a user who is using the AR display apparatus, and the job associated with the user is in an uncompleted state in an apparatus corresponding to the management information. The AR display system according to any one of [1] to [8], wherein an augmented reality space obtained by superimposing and combining the images of the marks is displayed on the AR display device only when it exists.

  In the inventions described in [9] and [19] below, only when a user who has input a job that has not been completed in a predetermined device matches a user who is using the AR display device, The AR display device displays the augmented reality space synthesized by overlapping the marks. Thereby, information unnecessary for the user wearing the AR display device is not displayed.

[10] The information processing apparatus performs image analysis on an image indicated by the photographed data, thereby determining whether or not an item indicated by the management information is present, and specifying a position of the existing item. The AR display system according to any one of [1] to [9].

  In the inventions described in [10] and [20] below, the image indicated by the photographic data is subjected to image analysis to determine whether or not the item indicated by the management information is present and to specify the position of the existing item. To do.

[11] An AR display unit that displays an augmented reality space obtained by combining an image with a real space that the user is viewing;
A photographing unit for photographing the real space that the user is viewing;
An acquisition unit that acquires management information indicating at least one item of a material that needs replenishment and a part that needs replacement in a predetermined device;
When there is an item indicated by the management information in the shooting data obtained by shooting by the shooting unit, the position of the existing item is specified, and an image of a predetermined mark is superimposed on the existing item and synthesized. A control unit that displays the augmented reality space on the AR display unit;
An AR display device comprising:

[12] The acquisition unit can acquire the management information from a plurality of devices, acquires the identification information of the device related to the management information together with the management information,
The control unit includes an augmented reality space in which the identification information of the device corresponding to the management information indicating the existing item is further combined with the augmented reality space obtained by superimposing the mark image on the existing item. The AR display device according to [11], which is displayed on an AR display unit.

[13] The control unit causes the AR display unit to display the augmented reality space in which the mark corresponding to one device is associated with identification information indicating the device, while making the mark different for each device. The AR display device according to [12], which is characterized.

[14] The management information includes urgency information indicating urgency when the item is needed,
The AR display device according to [11], wherein the control unit changes a display form of the mark according to the urgency of an item corresponding to the mark.

[15] When the item indicated by the management information exists outside the range of the real space corresponding to the image data, the control unit expands the guidance information for guiding the position where the item exists in the real space. The AR display device according to any one of [11] to [14], wherein a real space is displayed on the AR display unit.

[16] When there is a mark whose size is equal to or smaller than a predetermined value when the mark is superimposed on each of the existing items, the control unit replaces at least the mark whose size is equal to or smaller than the predetermined value, The AR display device according to any one of [11] to [15], wherein an item corresponding to the mark and an extended mark overlapping a predetermined range around the item are displayed.

[17] The acquisition unit includes:
(1) Always
(2) When the predetermined device exists in the shooting data,
(3) When the device is located in the storage area for the item and its surroundings,
(4) When a predetermined item exists in the shooting data,
(5) When a transmission request for the management information is received from the predetermined device,
(6) The management information is acquired by communicating with the device at any timing when a job submission notification is received from the predetermined device to which a job is submitted [11] to [16] The AR display device according to any one of [16].

[18] A position detector that detects the position of the device itself is provided,
The control unit takes time required for the user of the device to move from the position of the device detected by the position detection unit to the installation location of the predetermined device via the storage location of the item indicated by the management information. The required travel time is estimated by adding a predetermined allowance time, and at least at the time prior to the travel required time from the timing at which the operation of the predetermined device is stopped due to a lack of goods indicated by the management information. The AR display device according to any one of [11] to [17], wherein the device user is warned of the lack of the item.

[19] The image of the mark is specified only when the control unit identifies a user who is using the device, and a job associated with the user exists in an uncompleted state in the device corresponding to the management information. The AR display device according to any one of [11] to [18], wherein the augmented reality space synthesized by overlaying is displayed on the AR display unit.

[20] The control unit performs image analysis on an image indicated by the photographed data, thereby determining whether or not an item indicated by the management information exists, and specifying a position of the existing item. The AR display device according to any one of [11] to [19].

[21] The information processing apparatus in the AR display system according to any one of [1] to [10].

[22] A program for causing an information processing apparatus to operate as the information processing apparatus according to [21].

  According to the AR display system, the AR display device, the information processing device, and the program according to the present invention, the user can easily use the AR technology to easily obtain a target material or part from a large number of items placed in a storage location. Can find out.

It is a figure which shows the structural example of AR display system in which AR glasses which concern on embodiment of this invention are used. It is a figure which shows the example of a display when the user wearing AR glasses looks at a material storage. It is a block diagram which shows the electrical schematic structure of AR glasses. 1 is a block diagram illustrating a schematic electrical configuration of an image forming apparatus. It is a block diagram which shows the electrical schematic structure of a server. It is a figure different from FIG. 2 which shows the example of a display when the user wearing AR glasses looks at a material storage place. It is a figure different from FIG. 2, FIG. 6 which shows a display example when the user wearing AR glasses looks at the material storage. It is a figure different from FIG.2, FIG.6, FIG.7 which shows the example of a display when the user wearing AR glasses looks at a material storage place. It is a figure which shows the example of the information which management information shows. It is a figure different from FIG.2, FIG.6, FIG.7 and FIG. 8 which show the example of a display when the user wearing AR glasses looks at a material storage. It is a figure different from FIG.2, FIG.6, FIG.7, FIG.8 and FIG. 10 which show the example of a display when the user with AR glasses looks at a material storage place. It is a flowchart which shows the process performed in a server. It is a flowchart which shows the production | generation process of synthetic data.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 shows a configuration of an AR display system 2 in which AR glasses 40 serving as an AR display device according to an embodiment of the present invention are used. The AR display system 2 includes an AR glasses 40, a server 70 serving as an information processing apparatus according to an embodiment of the present invention, one or a plurality of image forming apparatuses 10 installed in an office, a wireless router 5, Are connected to a network 3 such as a LAN (Local Area Network). The AR glasses 40 are connected to the wireless router 5 by wireless communication, and are connected to the network 3 via the wireless router 5.

  The image forming apparatus 10 has a copy function for optically reading a document and printing the duplicate image on a recording paper, and a scan function for saving image data of the read document as a file or transmitting it to an external terminal via the network 3 A so-called MFP (MFP) having a printer function for forming an image of print data received from an external device via the network 3 on a recording sheet and printing it out. In the embodiment of the present invention, two image forming apparatuses 10 are connected to the AR display system 2 and are distinguished from MFP 1 and MFP 2. When any one of the MFP 1 and the MFP 2 is indicated, the image forming apparatus 10 is indicated.

  The AR glasses 40 is a wearable computer that displays an augmented reality space by a so-called head mounted display (HMD) method. The AR glasses 40 include a camera unit 52 (see FIG. 3) that captures a sight (real space) that the user is viewing, and transmits shooting data obtained by shooting with the camera unit 52 to the server 70.

The server 70 acquires management information indicating at least one item (material and component are collectively referred to as item) of the material that needs replenishment and the component that needs to be replaced in the image forming apparatus 10. In addition, when there is an item indicated by the management information in the photographing data received from the AR glasses 40, the position of the existing item is specified. Then, data for displaying the augmented reality space obtained by superimposing an image of a predetermined mark on the existing item on the AR glasses 40 (referred to as synthesized data) is generated and transmitted to the AR glasses 40.

  Based on the composite data received from the server 70, the AR glasses 40 displays an augmented reality space in which an image of a predetermined mark is superimposed on the item indicated by the management information in the scene (real space) that the user is viewing. To do.

  Specifically, when the user sees a material storage location that is a storage location for materials to be replenished to the image forming apparatus 10 or parts to be replaced, the AR glasses 40 are materials that need to be replenished by the image forming apparatus 10, or An augmented reality space in which images of predetermined marks are superimposed on a component that needs to be replaced (items that are insufficient in the image forming apparatus 10) is displayed. As a result, the user can easily find out the deficient items in the image forming apparatus 10 from among a large number of items placed in the material storage area. The display form of the predetermined mark is not particularly limited as long as it is easy for the user to recognize, but in the embodiment of the present invention, a frame line along the outer periphery of the item to be stacked is set as the predetermined mark. (See FIG. 2).

  FIG. 2 shows an example of a display state (augmented reality space) when the user wearing the AR glasses 40 looks at the material storage. The AR glasses 40 are worn around the user's eyes like glasses. The AR glasses 40 include a transmission part 41 corresponding to lens portions of the glasses respectively disposed in front of the wearer's left and right eyes when worn, a bridge 42 connecting the left and right transmission parts 41, and left and right transmission parts 41. It is provided with a vine portion 43 that extends from both sides in the same direction and is hooked on the ear of the wearer.

  The AR glasses 40 employ a method (so-called light transmission type) in which only an image of a portion to be virtually displayed is superimposed on an actual scene viewed by a user. The user sees the real world through the transmission unit 41. The transmissive part 41 is provided with a translucent display screen part 44 capable of displaying an image, and allows the user to see the augmented reality space by superimposing an arbitrary image on the scene of the real world.

  FIG. 2 shows an example of a state in which a frame line along the outer periphery of the product is displayed on the display screen unit 44 on the transmission unit 41 of the AR glasses 40 so as to overlap the product in the real world. In FIG. 2, the frame line corresponds to a predetermined mark, and the product on which the frame line is superimposed is a product that is insufficient in the image forming apparatus 10.

  FIG. 3 shows an electrical schematic configuration of the AR glasses 40. The AR glasses 40 include a ROM (Read Only Memory) 47, a RAM (Random Access Memory) 48, a nonvolatile memory 49, an orientation sensor unit 51, a camera unit 52, a CPU (Central Processing Unit) 46 serving as a control unit via a bus. The operation unit 53, the display screen unit 44, the wireless communication unit 54, the image processing unit 55, the position information acquisition unit 56, the authentication unit 57, and the like are connected to each other.

  The CPU 46 controls the operation of the AR glasses 40 according to the program stored in the ROM 47. The ROM 47 stores programs and fixed data. The RAM 48 is used as a work memory for temporarily storing various data when the CPU 46 executes a program. The nonvolatile memory 49 stores various setting information, user information, and the like.

  The direction sensor unit 51 detects the direction and orientation of the AR glasses 40 and changes thereof. The direction sensor unit 51 is configured by combining a plurality of geomagnetic sensors and a plurality of acceleration sensors. The direction sensor unit 51 detects the posture of the user wearing the AR glasses 40, the direction and angle that the user is facing (they match the direction and direction in which the camera unit 52 is shooting), and the AR The direction and speed of the operation when the glasses 40 are tilted are detected, and the detection result is notified to the CPU 46. Based on the notification from the direction sensor unit 51, the CPU 46 recognizes the posture and angle of the user wearing the AR glasses 40 and the direction and speed of the tilted operation.

  The camera unit 52 captures the front of the AR glasses 40 (a scene viewed by the wearer). More specifically, the image is taken in almost the same range as the field of view when the wearer looks forward. The camera unit 52 captures a moving image and captures, for example, an image of 30 frames per second.

  The operation unit 53 is a small number of switches and the like, and is used for adjusting the brightness of an image displayed on the display screen unit 44.

  The wireless communication unit 54 performs a function of performing wireless communication with the wireless router 5 and exchanging various information with the server 70 via the wireless router 5 and the network 3.

  The image processing unit 55 enlarges / reduces or deforms an image displayed on the display screen unit 44.

  The position information acquisition unit 56 detects the position of the own apparatus using a GPS function. Other methods may be used as the position detection method.

  The authentication unit 57 has a function of performing user authentication. Specifically, user authentication is performed at the time of login for the user wearing the device, and the login user is specified. In the present embodiment, it is assumed that login is required to use the AR glasses 40, and user authentication is always performed before use. The user authentication method may be any method such as authentication using a user ID (Identification) and a password.

  FIG. 4 is a block diagram illustrating a schematic configuration of the image forming apparatus 10. The image forming apparatus 10 includes a CPU 11 that comprehensively controls the operation of the image forming apparatus 10. The CPU 11 includes a ROM 12, a RAM 13, a nonvolatile memory 14, a hard disk device 15, an automatic document feeder (ADF) 16, an image reading unit 17, an operation unit 18, a display unit 19, a network communication unit 20, and an image process. The unit 21 and the printer unit 22 are connected.

  The CPU 11 is based on an OS (Operating System) program, and executes middleware, application programs, and the like. Various programs are stored in the ROM 12, and each function of the image forming apparatus 10 is realized by the CPU 11 executing various processes according to these programs.

  The RAM 13 is used as a work memory for temporarily storing various data when the CPU 11 executes processing based on a program, an image memory for storing image data, and the like.

  The nonvolatile memory 14 is a memory (flash memory) whose stored contents are not destroyed even when the power is turned off, and is used for storing various setting information. The hard disk device 15 is a large-capacity nonvolatile storage device, and stores various programs and data in addition to print data and image data.

  The image reading unit 17 performs a function of optically reading a document and acquiring image data. The image reading unit 17 sequentially, for example, a light source that irradiates light on the document, a line image sensor that receives the reflected light for one line in the width direction, and a line-by-line reading position in the length direction of the document. An optical path composed of a moving unit for moving, a lens, a mirror, and the like for guiding reflected light from the document to the line image sensor to form an image, and a conversion unit for converting an analog image signal output from the line image sensor into digital image data And so on.

  The automatic document feeder 16 feeds a document set on the platen one by one from the top one by one in order, passes the reading position of the image reading unit 17 and discharges it to a predetermined discharge position. Fulfill.

  The operation unit 18 and the display unit 19 constitute an operation panel that receives operations such as job input from the user. The display unit 19 is composed of a liquid crystal display (LCD) and the like, and functions to display various operation screens, setting screens, and the like. The operation unit 18 includes various operation switches such as a start button and a touch panel provided on the physical screen of the display unit 19. The touch panel detects a coordinate position where the physical screen of the display unit 19 is touched with a touch pen or a finger.

  The network communication unit 20 performs a function of communicating with an external device such as the server 70 through the network 3 such as a LAN. The network communication unit 20 also has a wireless LAN function.

  The image processing unit 21 performs rasterization processing for converting print data into image data, image data compression, expansion processing, and the like in addition to processing such as image enlargement / reduction and rotation.

  The printer unit 22 has a function of forming an image corresponding to the image data on a recording sheet. Here, it has a recording paper transport device, a photosensitive drum, a charging device, a laser unit, a developing device, a transfer separation device, a cleaning device, and a fixing device, and forms an image by an electrophotographic process. It is configured as a so-called laser printer. Other methods may be used for image formation.

  The image forming apparatus 10 monitors the remaining amount of materials in the apparatus itself, the replacement timing of parts, and the like. For example, the remaining amount of paper and toner set in the paper feed tray and the replacement time of parts such as the photosensitive drum and the fixing device are monitored. The image forming apparatus 10 transmits, to the server 70, management information indicating at least one of the materials that need to be replenished and the components that need to be replaced, obtained by the above-described monitoring. Based on this management information, the server 70 grasps materials that need to be replenished in the image forming apparatus 10 that is the transmission source and parts that need replacement (insufficient items).

  FIG. 5 is a block diagram illustrating a schematic configuration of the server 70. The server 70 is configured by connecting a ROM 71, a RAM 73, a nonvolatile memory 74, a hard disk device 75, a network communication unit 76, and the like to a CPU 71 as a control unit via a bus.

  The CPU 71 controls the operation of the server 70 according to a program stored in the ROM 72. The ROM 72 stores programs and fixed data. The RAM 73 is used as a work memory for temporarily storing various data when the CPU 71 executes a program.

  The nonvolatile memory 74 is a memory (flash memory) whose stored contents are not destroyed even when the power is turned off, and is used for storing various setting information. The hard disk device 75 is a large-capacity nonvolatile storage device, and stores management information received from the image forming apparatus 10, shooting data received from the AR glasses 40, and the like.

  The network communication unit 76 functions to communicate with external devices such as the AR glasses 40 and the server 70 through the network 3 such as a LAN. The network communication unit 76 also has a wireless LAN function.

  The server 70 performs image analysis on the shooting data received from the AR glasses 40, and if there is an item indicated by the management information received from the image forming apparatus 10 in the image indicated by the shooting data, specifies the position thereof. To do. Then, data (synthetic data) for displaying the augmented reality space in which the image of the predetermined mark is superimposed on the item whose position is specified and is synthesized on the AR glasses 40 is generated and transmitted to the AR glasses 40.

  Next, the display state (augmented reality space) when the user wearing the AR glasses 40 looks at the material storage area will be described with a plurality of examples.

  FIG. 6 shows a display state (expanded) when the user wearing the AR glasses 40 looks at the material storage area when there are missing items (materials that need replenishment and parts that need replacement) in the MFP 1. An example of (real space) is shown. In FIG. 6, a frame is displayed so as to overlap a specific item, and identification information of an MFP that lacks the specific item (in the figure, MFP 1 indicates that the item is insufficient). ) Is displayed. As a result, the user can easily find the missing item in the MFP 1 from among the many items placed in the material storage area.

  FIG. 7 shows an example of a display state (augmented reality space) when the user wearing the AR glasses 40 looks at the material storage area when there are missing items in the MFP 1 and the MFP 2. In FIG. 7, the item that is lacking in the MFP 2 (referred to as item B) is displayed so that a frame line that is filled with diagonal lines is superimposed. Items that are insufficient in the MFP 1 (referred to as item A) are displayed such that frame lines that are not filled in are superimposed.

  In FIG. 7, identification information of MFPs that lack the items with the frame lines overlapped, that is, MFP 1 and MFP 2 is displayed. The identification information of the MFP 1 is displayed without being filled in as in the case of the mark (frame line) superimposed on the item A. The identification information of the MFP 2 is displayed in a state in which the inside is filled with diagonal lines in the same manner as the mark (frame line) superimposed on the item B.

  As described above, in FIG. 7, the mark displayed on the item (item A) that is deficient in the MFP 1 and the internal display form of the identification information of the MFP 1, and the mark superimposed on the item (item B) deficient in the MFP 2 The internal display format is aligned. Therefore, the user can grasp that the item B is a missing item in the MFP 2 and that the item A is a missing item in the MFP 1.

  As described above, in the AR display system 2, the display form of the mark to be overlaid on the item deficient in the MFP 1 is different from the display form of the mark to be overlaid on the item deficient in the MFP 2. In addition, a mark to be overlaid on the item deficient in the MFP 1 and the display form of the identification information of the MFP 1 and a mark to be overlapped on the item deficient in the MFP 2 and the display form of the identification information of the MFP 2 are displayed in association with each other. As a result, the missing items in the MFP 1 and the MFP 2 are displayed separately, so that the user can grasp which of the MFP 1 and the MFP 2 is the missing item. .

  FIG. 8 shows an example of a display state (augmented reality space) when the user wearing the AR glasses 40 looks at the material storage area when there is a shortage in the MFP 1 and shortly short items. . In FIG. 8, items that will be shortly displayed are displayed such that a frame line whose inside is shaded is overlaid. Items that are deficient in the MFP 1 are displayed so that a frame line that is not filled in is overlaid.

  As described above, in the AR display system 2, display forms of predetermined marks (frame lines in the figure) to be combined so as to be superimposed on the item are changed according to the urgency of the time required by the image forming apparatus 10. . In FIG. 8, a mark that is superimposed on an item that is already insufficient in MFP 1 and a mark that is superimposed on an item that will soon be insufficient are displayed separately. Thus, an item on which an image of a predetermined mark is superimposed is already displayed on MFP 1. It is possible to grasp whether it is a shortage or a shortage soon.

  Next, management information acquired by the server 70 will be described. In the embodiment of the present invention, the server 70 displays information relating to jobs that have not been completed in the image forming apparatus 10 (standby jobs), information on the remaining amount of materials, information on parts that need replacement, and the like. Obtained as management information. FIG. 9 shows a management information table 81 that is an example of management information received by the server 70 from the MFP 1.

  The management information table 81 includes, for each job waiting in the MFP 1 (No. 1 to No. 4), identification information of the user who has input the job, and the number of sheets output by the job (including paper size information). The remaining amount of paper immediately before execution of the job is displayed. Four jobs (No1 to No4 jobs) are executed in the order of No1, No2, No3, and No4.

  The No. 1 job is a print job submitted by the user A, and four A4 sheets are scheduled to be output, and the remaining amount of A4 sheets is 20 sheets immediately before the execution of the job.

  The job No. 2 is a print job submitted by the user B, and five A4 sheets are scheduled to be output. The remaining amount of A4 sheets is 16 sheets just before the job is executed.

  The job No. 3 is a print job submitted by the user C, and is scheduled to output 10 sheets of A4 paper. The remaining amount of A4 paper immediately before the execution of the job is 11 sheets.

  The job No. 4 is a print job submitted by the user D, and five A4 sheets are scheduled to be output, and the remaining amount of A4 sheets is one sheet immediately before the execution of the job. That is, when the MFP 1 executes a No. 4 job without replenishing A4 paper, the paper runs out and the job is interrupted.

  The server 70 generates data (synthetic data) for causing the AR glasses 40 to display the augmented reality space as shown in FIGS. 6 and 8 based on the management information table 81 and the imaging data received from the AR glasses 40. , Transmitted to the AR glasses 40. For example, if the paper runs out after the job is executed, composite data for displaying on the AR glasses 40 that the paper is insufficient is generated. Further, when the threshold value of the remaining amount of paper (for example, 10 sheets) provided in advance is reduced, composite data for displaying on the AR glasses 40 that the paper will soon be short is generated.

  In the embodiment of the present invention, the server 70 acquires identification information indicating the user specified by the authentication unit 57 from the AR glasses 40, and the job submitted by the user to the image forming apparatus 10 corresponding to the management information. Is displayed in an augmented reality space on the AR glasses 40 only when the image of a predetermined mark is superimposed on the AR glasses 40 only when the image is incomplete (running or waiting). Data (composite data) is generated and transmitted to the AR glasses 40. Since a user who has not submitted a job to the image forming apparatus 10 or a user who has already completed the submitted job is using the AR glasses 40, information on the missing items is not displayed on the image forming apparatus 10. Only information necessary for the user wearing the AR glasses 40 is displayed.

  The server 70 always receives position information from the AR glasses 40, and adds a predetermined margin time to the time required to move from the position of the AR glasses 40 to the installation location of the image forming apparatus 10 via the material storage. Estimated travel time. Then, the AR glasses 40 are short of items at least at the time required for the movement from the timing at which the operation of the image forming apparatus 10 is stopped (job is interrupted) due to the shortage of the items indicated by the management information. Display a warning. Immediately after the confirmation, the user who confirmed the warning goes to the material storage area to procure materials or parts, and returns to the image forming apparatus 10 to stop the operation of the image forming apparatus 10 (the job is interrupted). You can replenish materials and replace parts.

The server 70
(1) Always
(2) When the image forming apparatus 10 exists in the shooting data received from the AR glasses 40,
(3) When the AR glasses 40 are located in the material storage area and the vicinity thereof (location information is always received from the AR glasses 40)
(4) When a predetermined item exists in the shooting data received from the AR glasses 40,
(5) When a management information transmission request is received from the image forming apparatus 10,
(6) When a job submission notification is received from the image forming apparatus 10 to which a job has been submitted,
The management information is acquired by communicating with the image forming apparatus 10 at any of the timings.

  In (1), for example, the management information is periodically acquired at regular intervals. In (4), the predetermined item refers to a material or part designated in advance. (5) may be configured to receive the management information when the management information is transmitted from the image forming apparatus 10.

  Next, a modified example of the display state (augmented reality space) when the user wearing the AR glasses 40 looks at the material storage will be described.

(Modification 1)
FIG. 10 shows guidance information for guiding the position where the product exists when the missing item in the MFP 1 is outside the range of the real space that the user is viewing (in the figure, the direction in which the position exists is shown). An example of a state of displaying an arrow) is shown.

  The server 70 registers the position where each item is placed in advance in the material storage area, and recognizes the position of the missing item in the MFP 1. Alternatively, the position of the missing item is specified in the MFP 1 from the shooting data obtained by the user looking over the entire material storage.

  Then, the server 70 recognizes the direction and distance in which the item is located with respect to the position in the range indicated by the shooting data. For example, it recognizes whether it is on the right side, the left side, or how far away it is from the position the user is looking at. Thereafter, guidance information indicating the direction is transmitted, and this direction is displayed on the AR glasses 40 with an arrow. As a result, even if there is an item that is lacking in the image forming apparatus 10 outside the range of the real space that the user is viewing, the item can be easily found.

  In the embodiment of the present invention, the server 70 is used only when the distance between the position of the range indicated by the shooting data and the position of the product is equal to or smaller than the predetermined value, or when the AR glasses 40 are within the predetermined range. The guidance information is transmitted. For example, the server 70 transmits guidance information only when the user wearing the AR glasses 40 is in the material storage area, and causes the AR glasses 40 to display an arrow indicating the direction in which the item is present.

(Modification 2)
FIG. 11 shows an example of a display state (augmented reality space) when the user sees the material storage when the user wearing the AR glasses 40 is away from the material storage.

  In FIG. 11, a rectangular frame (mark) is displayed so as to overlap a predetermined range around an item that is lacking in the MFP 1. Since the distance between the user and the material storage area is large, each item appears small in the user's field of view and shooting data. When a frame line (mark) is displayed so as to circumscribe a small object, it is difficult for the user to recognize where the frame line (mark) is displayed.

  Therefore, when the image of a predetermined mark (a frame line in the figure) is overlaid only on the existence range of the item in the shooting data, the server 70 determines that the size is smaller than the predetermined value. In place of the mark below the predetermined value, data (composite data) for displaying the augmented reality space obtained by compositing the image of the extended mark overlapping the predetermined range around the product with the real space on the AR glasses 40 is generated. Send to. Thereby, for example, even when the distance between the user wearing the AR glasses 40 and the item is large or the item itself is small, a mark indicating the item that is insufficient in the image forming apparatus 10 is displayed. Displayed in an easy-to-understand manner for the user. In FIG. 11, when the material storage area is viewed from a remote location, it can be seen where the items that are insufficient in the image forming apparatus 10 are located. When the user approaches the material storage area, only the item is overlaid with a mark, and the item can be specified.

  When a predetermined mark is displayed at a plurality of locations, only a mark whose size is a predetermined value or less among the plurality of marks may be replaced with an extension mark, or all the marks may be changed to an extension mark.

  FIG. 12 shows processing performed by the server 70 until the AR glasses 40 display the augmented reality space in the AR display system 2. In FIG. 12, the server 70 has previously acquired identification information of the user who is using the AR glasses 40.

  First, the server 70 waits until the management information is acquired from the image forming apparatus 10 (step S001; No). When the timing is reached (step S001; Yes), the management information is transmitted together with the identification information of the image forming apparatus 10. Obtain (step S002).

  If no uncompleted job in the image forming apparatus 10 indicated by the management information is input by the user who is using the AR glasses 40 (step S003; No), the process is terminated. If there is a job submitted by the user who is using the AR glasses 40 (step S003; Yes), the missing items in the image forming apparatus 10 are specified from the management information (step S004).

  Thereafter, the AR glasses 40 are notified that the item specified in step S004 is insufficient in the image forming apparatus 10 (step S005). Upon receiving this notification, the AR glasses 40 display information indicating that there is a material shortage in the image forming apparatus 10 and the name of the material shortage.

  Next, imaging data is received from the AR glasses 40. The server 70 performs image analysis on the received image data and checks whether there is a pre-registered type of item. For example, when printing paper and toner are registered, it is checked whether the printing paper and toner exist in the shooting data.

  If there is no registered item in the shooting data (step S006; No), the process returns to step S006 and continues. If there is a registered item in the shooting data (step S006; Yes), the position of the item is specified (step S007). When there are a plurality of items, the positions of the plurality of items are specified.

  Next, it is determined whether or not the product whose position has been specified is a product that is insufficient in the image forming apparatus 10. If there is no shortage of items in the image forming apparatus 10 among the items whose positions have been specified (step S008; No), this processing is terminated. For example, if the A3 printing paper is insufficient in the image forming apparatus 10 and the printing paper in the shooting data is A4 size, the process ends.

  If there is a shortage of goods in the image forming apparatus 10 among the goods whose positions have been specified (step S008; Yes), an image of a predetermined mark is superimposed on the goods, and the goods that are short of the goods Data (synthetic data) for displaying the augmented reality space obtained by combining the identification information of the forming apparatus 10 on the AR glasses 40 is generated (step S009). Then, the composite data is transmitted to the AR glasses 40 (step S010), and this process is terminated.

  Note that steps S006 to S010 of this process are repeated each time image data is received from the AR glasses 40. The server 70 deletes the shooting data after a certain period of time has elapsed since the reception.

  FIG. 13 shows the composite data generation process performed in step S009 of FIG.

  First, in the item whose position is specified in step S007 of FIG. 12, when there is a mark whose size is equal to or smaller than a predetermined value when the image of the predetermined mark is superimposed on the item that is insufficient in the image forming apparatus 10. (Step S101; Yes) Instead of a mark having a size equal to or smaller than a predetermined value, an augmented reality space in which an image of an extended mark that overlaps a predetermined range around an article corresponding to the mark is combined with the real space is displayed on the AR glasses 40 Data to be processed (composite data) is generated (step S102), and this process is terminated.

  When there is no mark whose size is equal to or smaller than the predetermined value (step S101; No), the image of the predetermined mark is overlaid on the missing item in the image forming apparatus 10, and the identification information of the image forming apparatus 10 is synthesized. Data (synthetic data) for displaying the augmented reality space on the AR glasses 40 is generated (step S103), and this process ends.

  Note that as in the first modification, guidance information for guiding the position of an item outside the real space that the user is viewing may be displayed on the AR glasses 40. The method described above is employed when displaying the guidance information.

  As described above, by using the AR glasses 40 that play a role as the AR display device according to the present embodiment, the user can select a target material or part (image) from a large number of items placed in the material storage area. It is possible to easily find items that are insufficient in the forming apparatus 10.

  For example, when a plurality of image forming apparatuses 10 are juxtaposed in an office, when a user wearing the AR glasses 40 looks at the material storage area, an image of a predetermined mark is displayed on items that are insufficient in the image forming apparatus 10. Are superimposed and displayed. In addition, the image forming apparatus 10 in which the product is lacking is also displayed. Accordingly, when looking at the material storage area, it is possible to grasp at a glance which items placed in the material storage area are insufficient in which image forming apparatus 10.

  The embodiment of the present invention has been described with reference to the drawings. However, the specific configuration is not limited to that shown in the embodiment, and there are changes and additions within the scope of the present invention. Are also included in the present invention.

  In the embodiment, the server 70 specifies the presence / absence of an item and the position of the item previously registered in the shooting data by image analysis, but the method for specifying the position of the item is not limited to this. Other methods may be used. For example, the position of each item is registered in advance, the position of the range indicated by the shooting data is specified by GPS or the like, and it is determined whether or not the item indicated by the management information exists within the specified range. There may be.

  In the embodiment of the present invention, in the AR display system 2, the AR glasses 40 display the augmented reality space based on the composite data received from the server 70. For example, the role of the server 70 is AR glasses. 40 may fulfill. That is, the AR glasses 40 may be configured to acquire management information from the image forming apparatus 10 and generate composite data based on the management information and shooting data obtained by shooting by the camera unit 52. Further, the present invention may be a server 70 alone in the embodiment and a program that causes the server 70 to operate as described in the embodiment.

  In the embodiment, the AR display device is the head-mounted AR glasses 40, but the AR display device is not limited thereto. For example, a mobile terminal with a camera such as a smartphone may be used. The captured data captured by the camera may be transmitted to the server 70, and the augmented reality space may be displayed according to the composite data received from the server 70.

2 ... AR display system 3 ... Network (LAN)
5 ... Wireless router 10 ... Image forming apparatus (MFP1, MFP2)
11 ... CPU
12 ... ROM
13 ... RAM
DESCRIPTION OF SYMBOLS 14 ... Non-volatile memory 15 ... Hard disk device 16 ... Automatic document conveyance part 17 ... Image reading part 18 ... Operation part 19 ... Display part 20 ... Network communication part 21 ... Image processing part 22 ... Printer part 40 ... AR glasses 41 ... Transmission part 42 ... Bridge 43 ... Tail part 44 ... Display screen part 46 ... CPU
47 ... ROM
48 ... RAM
DESCRIPTION OF SYMBOLS 49 ... Non-volatile memory 51 ... Direction sensor part 52 ... Camera part 53 ... Operation part 54 ... Wireless communication part 55 ... Image processing part 56 ... Position information acquisition part 57 ... Authentication part 70 ... Server 71 ... CPU
72 ... ROM
73 ... RAM
74 ... Non-volatile memory 75 ... Hard disk device 76 ... Network communication part 81 ... Management information table

Claims (22)

An AR display device including an AR display unit that displays an augmented reality space obtained by synthesizing an image with a real space viewed by a user, and an imaging unit that captures the real space viewed by the user, and communication with the AR display device An AR display system including a possible information processing device,
The AR display device transmits photographing data obtained by photographing by the photographing unit to the information processing device,
The information processing apparatus acquires management information indicating at least one item of a material that needs replenishment and a part that needs to be replaced in a predetermined device, and the information is received in the imaging data received from the AR display device If the item indicated by the management information exists, data for specifying the position of the existing item and causing the AR display device to display an augmented reality space in which an image of a predetermined mark is superimposed on the existing item and synthesized. Is generated and transmitted to the AR display device,
The AR display system displays an augmented reality space obtained by superimposing the image of the mark on the existing item based on the data. The information processing device can acquire the management information from a plurality of devices, acquires the identification information of the device related to the management information together with the management information, and the image of the mark on the existing item as the data Generating data for causing the AR display device to display the augmented reality space in which the identification information of the device corresponding to the management information indicating the existing item is further synthesized in the augmented reality space synthesized by overlapping 2. The AR display system according to claim 1, wherein The information processing device displays, as the data, the augmented reality space in which the mark corresponding to one device is associated with identification information indicating the device on the AR display device, while making the mark different for each device. The AR display system according to claim 2, wherein data for generating the data is generated. The management information includes urgency information indicating urgency when the item is needed,
The AR display system according to claim 1, wherein the information processing apparatus changes a display form of the mark according to the urgency of an item corresponding to the mark. When there is an item indicated by the management information outside the real space corresponding to the shooting data received from the AR display device, the information processing device actually provides guidance information for guiding the position where the item exists. The AR display system according to any one of claims 1 to 4, wherein the augmented reality space combined with the space is displayed on the AR display device. When there is a mark whose size is less than or equal to a predetermined value when the mark is superimposed on each of the existing items, the information processing apparatus replaces at least the mark with the size less than or equal to the predetermined value. 6. The AR display device displays an augmented reality space in which an augmented mark that overlaps an article corresponding to the item and an extension mark that overlaps a predetermined range around the item is displayed on the AR display device. Display system. The information processing apparatus includes:
(1) Always
(2) When the predetermined device exists in the photographing data received from the AR display device,
(3) When the AR display device is located in and around the storage location for the item,
(4) When a predetermined item exists in the photographing data received from the AR display device,
(5) When a transmission request for the management information is received from the predetermined device,
(6) The management information is acquired by communicating with the device at any timing when a job submission notification is received from the predetermined device to which a job is submitted. The AR display system according to any one of 6. The information processing apparatus acquires position information of the AR display apparatus, and a user of the AR display apparatus installs the predetermined apparatus from a position indicated by the position information via a storage location of an item indicated by the management information. Estimate the required travel time by adding a predetermined margin time to the time required to move to a location, and at least the travel required from the timing when the operation of the predetermined device is stopped due to the lack of goods indicated by the management information. The AR display system according to any one of claims 1 to 7, wherein the user is warned of a shortage of the item at a time point before the time. The information processing apparatus acquires user information indicating and identifying a user who is using the AR display apparatus, and a job associated with the user exists in an uncompleted state in an apparatus corresponding to the management information The AR display system according to any one of claims 1 to 8, wherein an augmented reality space in which the images of the marks are superimposed and combined is displayed on the AR display device. The information processing apparatus performs image analysis on an image indicated by the photographed data, thereby determining whether or not an item indicated by the management information exists and specifying a position of the existing item. Item 10. The AR display system according to any one of Items 1 to 9. An AR display unit that displays an augmented reality space obtained by combining an image with a real space viewed by a user;
A photographing unit for photographing the real space that the user is viewing;
An acquisition unit that acquires management information indicating at least one item of a material that needs replenishment and a part that needs replacement in a predetermined device;
When there is an item indicated by the management information in the shooting data obtained by shooting by the shooting unit, the position of the existing item is specified, and an image of a predetermined mark is superimposed on the existing item and synthesized. A control unit that displays the augmented reality space on the AR display unit;
An AR display device comprising: The acquisition unit is capable of acquiring the management information from a plurality of devices, and acquires identification information of a device related to the management information together with the management information,
The control unit includes an augmented reality space in which the identification information of the device corresponding to the management information indicating the existing item is further combined with the augmented reality space obtained by superimposing the mark image on the existing item. The AR display device according to claim 11, wherein the AR display unit is displayed on the AR display unit. The control unit makes the mark different for each device, and causes the AR display unit to display the augmented reality space in which a mark corresponding to one device is associated with identification information indicating the device. The AR display device according to claim 12. The management information includes urgency information indicating urgency when the item is needed,
The AR display device according to claim 11, wherein the control unit changes a display form of the mark according to the urgency of an item corresponding to the mark. When there is an item indicated by the management information outside the range of the real space corresponding to the shooting data, the control unit creates an augmented reality space in which the guide information for guiding the position of the item is combined with the real space. It displays on the said AR display part. AR display apparatus as described in any one of Claim 11 thru | or 14 characterized by the above-mentioned. When there is a mark whose size is not more than a predetermined value when the mark is superimposed on each of the existing items, the control unit replaces at least the mark with the size not more than the predetermined value on the mark. The AR display device according to any one of claims 11 to 15, wherein an expansion mark overlapping a corresponding item and a predetermined range around the corresponding item is displayed. The acquisition unit
(1) Always
(2) When the predetermined device exists in the shooting data,
(3) When the device is located in the storage area for the item and its surroundings,
(4) When a predetermined item exists in the shooting data,
(5) When a transmission request for the management information is received from the predetermined device,
(6) The management information is acquired by communicating with the device at any timing when a job submission notification is received from the predetermined device to which a job is submitted. The AR display device according to any one of 16. It has a position detection unit that detects the position of its own device,
The control unit takes time required for the user of the device to move from the position of the device detected by the position detection unit to the installation location of the predetermined device via the storage location of the item indicated by the management information. The required travel time is estimated by adding a predetermined allowance time, and at least at the time before the travel required time from the timing at which the operation of the predetermined device is stopped due to the lack of goods indicated by the management information. The AR display device according to any one of claims 11 to 17, wherein the device user is warned of the lack of the item. The control unit identifies a user who is using the device, and superimposes the mark image only when a job associated with the user exists in an incomplete state in the device corresponding to the management information. The AR display device according to any one of claims 11 to 18, wherein the augmented reality space is displayed on the AR display unit. The control unit performs image analysis on an image indicated by the photographed data, thereby determining whether or not an item indicated by the management information is present, and specifying a position of the existing item. The AR display device according to any one of 11 to 19.   The information processing apparatus in the AR display system according to any one of claims 1 to 10.   A program for causing an information processing apparatus to operate as the information processing apparatus according to claim 21.