JP2023140285A - Information processing apparatus, information processing system, device management method, and program - Google Patents

Abstract

To configure settings for an electronic device using information used in another electronic device.SOLUTION: A management apparatus configured to communicate with a plurality of electronic devices and an imaging apparatus over a communication link includes: a state storage unit; a state management unit which includes an object state management unit, a device state management unit, and an output information determination unit; an image receiving unit which manages the electronic devices and a state of an object, decides content of an operation instruction to be transmitted to the electronic devices, updates, based on a received image, a registered object list stored in the state storage unit, updates, based on received device operation information, a registered device list and the registered object list, and transmits the received image to the state management unit; an operation information receiving unit which receives the device operation information from the electronic devices and transmits the received device operation information to the state management unit; an output information determination unit which determines content of an operation instruction to be transmitted to the electronic devices on the basis of the stored state management information, and transmits the determined operation instruction to an operation instruction unit; and the operation instruction unit which transmits the operation instruction received from the state management unit to the electronic devices.SELECTED DRAWING: Figure 6

Description

The present invention relates to an information processing device, an information processing system, a device management method, and a program.

In cloud computing, there is a technology that enables single sign-on by linking software. For example, Patent Document 1 discloses an invention in which a user who is authenticated in a certain service can omit authentication in other services by using a common authentication infrastructure for a plurality of services.

In authentication using hardware such as electronic equipment, there is a technique for associating authentication processing of each piece of hardware. For example, Patent Document 2 discloses that by linking a system that manages entry and exit to an area that handles confidential information and electronic devices in the area, electronic devices that are not authenticated by the entry/exit management system can An invention is disclosed that does not permit authentication on a device.

In coordination between hardware, it is difficult to determine the identity of a user. The invention disclosed in Patent Document 2 utilizes the authentication status of the room entry/exit management system for authentication using electronic equipment, but each user needs to perform an authentication operation for each user. For example, if it is possible to determine the identity of a user who authenticated with a first electronic device and a user who is near a second electronic device, authentication information input by the user with the first electronic device can be used to authenticate the user with the second electronic device. electronic devices can perform authentication.

One embodiment of the invention aims to configure a second electronic device using information used in a first electronic device.

In order to solve the above problems, an information processing device according to an embodiment of the present invention is an information processing device that can communicate with a first electronic device, a second electronic device, and a photographing device via a network. , a first image receiving unit that receives a first image of a first user using a first electronic device from a photographing device; and a first image receiving unit that receives usage information of the first electronic device and the first image. a first user identification unit that identifies the first user who used the first electronic device based on the authentication information input by the first user identified by the first user identification unit on the first electronic device; an input information receiving section that receives from a first electronic device; and a second image receiving section that receives a second image of a second user who is present at an operable position of the second electronic device from a photographing device. a second user specifying unit that specifies a second user present at an operable position of the second electronic device from a second image; and a first user and a second user specified by the first user specifying unit. An operation instruction section that transmits authentication information or screen setting information included in the authentication information to the second electronic device based on the second user information and authentication information specified by the identification section.

According to one embodiment of the present invention, a second electronic device can be configured using information used in a first electronic device.

FIG. 1 is a diagram illustrating an example of the overall configuration of a device management system in an embodiment. 1 is a diagram illustrating an example of a hardware configuration of a computer in an embodiment. FIG. 1 is a diagram illustrating an example of a hardware configuration of a celestial sphere imaging device in an embodiment. FIG. FIG. 2 is a diagram illustrating an example of a hardware configuration of an MFP in an embodiment. FIG. 1 is a diagram illustrating an example of the hardware configuration of an electronic whiteboard in an embodiment. FIG. 1 is a diagram illustrating an example of a functional configuration of a device management system in an embodiment. It is a figure showing an example of state management information in one embodiment. FIG. 3 is a diagram illustrating an example of a first basic flowchart in one embodiment. It is a figure which shows an example of the 2nd basic flowchart in one embodiment. FIG. 3 is a diagram illustrating an example of a device management method in an embodiment. FIG. 3 is a diagram illustrating an example of object state update processing in an embodiment. FIG. 3 is a diagram illustrating an example of a detection object list creation process in an embodiment. FIG. 3 is a diagram illustrating an example of registered object list update processing in one embodiment. FIG. 3 is a diagram illustrating an example of registered object list update processing in one embodiment. It is a figure showing the 1st example of group ID grant processing in one embodiment. It is a figure showing the 2nd example of group ID grant processing in one embodiment. FIG. 3 is a diagram illustrating an example of device status update processing in an embodiment. FIG. 3 is a diagram illustrating an example of output information determination processing in an embodiment. FIG. 3 is a diagram illustrating an example of output condition determination processing in an embodiment. It is a figure showing an example of device output information in one embodiment.

Embodiments of the present invention will be described in detail below with reference to the drawings. Note that in the drawings, components having the same functions are designated by the same numbers, and redundant explanation will be omitted.

[Embodiment]
One embodiment of the present invention is a device management system that manages multiple electronic devices. The device management system in this embodiment achieves cooperation between electronic devices by instructing other electronic devices to perform operations using information input by one electronic device.

Conventionally, in cooperation between hardware, the identity of a user has been determined based on possession of the same device (for example, an IC (Integrated Circuit) card or a mobile information terminal, etc.). As a result, even if the hardware cooperated, authentication operations were required on each piece of hardware. With single sign-on based on software coordination, when a user authenticated in one service accesses another service, the operations are performed from the same device, so the identity of the user can be easily determined.

In the device management system according to the present embodiment, a management device tracks a person who moves between electronic devices based on an image taken of a space where a plurality of electronic devices are installed. Thereby, the management device can determine the identity of a user who operated a certain electronic device and a user who is near another electronic device.

If the identity of the user can be determined, it is possible to cause another electronic device to perform an operation using information input on one electronic device. For example, if an electronic device near a user performs authentication using authentication information input by the user on another electronic device, single sign-on through hardware cooperation can be achieved.

<Overall configuration of device management system>
First, the overall configuration of the device management system in this embodiment will be described with reference to FIG. 1. FIG. 1 is a diagram showing an example of the overall configuration of a device management system in this embodiment.

As shown in FIG. 1, the device management system 1 in this embodiment includes a management device 10, one or more monitoring devices 20, and a plurality of electronic devices 30. The management device 10, the monitoring device 20, and the electronic device 30 are each connected to the communication network N1.

The communication network N1 is configured so that connected devices can communicate with each other. The communication network N1 is constructed, for example, by a wired communication network such as the Internet, a LAN (Local Area Network), or a WAN (Wide Area Network).

The communication network N1 includes not only wired communication but also wireless communication such as wireless LAN or short-range wireless communication, WiMAX (Worldwide Interoperability for Microwave Access), LTE (Long Term Evolution), 5G (5th Generation), etc. This may include a mobile communication network.

The monitoring device 20 and the electronic device 30 are installed in the managed space R1. The managed space R1 may be a single space such as a room in a building, or may be a plurality of spaces connected by a door, a hallway, or other accessible space.

An example of the managed space R1 is an office conference room, an office room, or the like. Another example of the managed space R1 is a sales floor or a backyard in a small-scale store such as a convenience store. Another example of the managed space R1 is a hotel lobby or a guest room. Other examples of the managed space R1 are an airport, a bookstore, a factory, or the like. The managed space R1 is not limited to these, and may be any space in which electronic equipment that can be operated by multiple users is installed.

The management device 10 is an information processing device such as a PC (Personal Computer), a workstation, or a server that manages the electronic device 30. The management device 10 transmits an operation instruction to the electronic device 30 based on the image acquired by the monitoring device 20. An example of the management device 10 is a computer.

The monitoring device 20 is an electronic device that acquires an image including the vicinity of the electronic device 30 installed in the managed space R1. The monitoring device 20 may acquire video (that is, a time series of images). The monitoring device 20 is installed at a position where it can photograph all of the plurality of electronic devices 30.

An example of the monitoring device 20 is a spherical photographing device. Another example of the monitoring device 20 is a plurality of network cameras. When the monitoring device 20 is a plurality of network cameras, each network camera is arranged with its angle of view adjusted so that no blind spot occurs in the managed space R1.

Hereinafter, when there is a plurality of monitoring devices 20, to distinguish each one, they will be described using branch numbers such as "monitoring device 20-1" and "monitoring device 20-2."

The electronic device 30 is a variety of electronic devices used by users. The electronic device 30 is configured so that only an authenticated user can use it. The electronic device 30 is configured to enable authentication using hardware such as an IC (Integrated Circuit) card.

An example of the electronic device 30 is an image forming device (a printer, a FAX device, an MFP (Multifunction Peripheral/Product/Printer), a scanner device, etc.). Another example of the electronic device 30 is an electronic whiteboard (IWB: a whiteboard having an electronic blackboard function that allows mutual communication).

Hereinafter, when distinguishing each of the plurality of electronic devices 30, they will be described using branch numbers such as "electronic device 30-1" and "electronic device 30-2."

The electronic device 30 is not limited to an image forming device or an electronic blackboard as long as it is a device with a communication function. That is, the electronic device 30 includes, for example, a projector (PJ), an output device such as a digital signage, a head up display (HUD) device, an industrial machine, an imaging device, a sound collection device, a medical device, a network home appliance, and a connected vehicle. The computer may be a car, a notebook PC (Personal Computer), a mobile phone, a smartphone, a tablet terminal, a game console, a PDA (Personal Digital Assistant), a digital camera, a wearable PC, a desktop PC, or the like.

<Hardware configuration of device management system>
Next, the hardware configuration of each device included in the device management system in this embodiment will be explained with reference to FIGS. 2 to 5.

≪Computer hardware configuration≫
FIG. 2 is a diagram showing an example of a hardware configuration when the management device 10 is implemented by a computer.

As shown in FIG. 2, the computer in one embodiment includes a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, a RAM (Random Access Memory) 503, an HD (Hard Disk) 504, an HDD ( Hard Disk Drive) controller 505, display 506, external device connection I/F (Interface) 508, network I/F 509, bus line 510, keyboard 511, pointing device 512, DVD-RW (Digital Versatile Disk Rewritable) drive 514, media Equipped with I/F516.

Among these, the CPU 501 controls the operation of the entire computer. The ROM 502 stores programs used to drive the CPU 501, such as IPL (Initial Program Loader). RAM 503 is used as a work area for CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls reading and writing of various data to the HD 504 under the control of the CPU 501.

The display 506 displays various information such as a cursor, menu, window, characters, or images. External device connection I/F 508 is an interface for connecting various external devices. The external device in this case is, for example, a USB (Universal Serial Bus) memory, a printer, or the like. The network I/F 509 is an interface for data communication using the communication network N1. The bus line 510 is an address bus, a data bus, etc. for electrically connecting each component such as the CPU 501 shown in FIG. 2.

Further, the keyboard 511 is a type of input means that includes a plurality of keys for inputting characters, numerical values, various instructions, and the like. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, and the like. The DVD-RW drive 514 controls reading and writing of various data on a DVD-RW 513, which is an example of a removable recording medium. Note that it is not limited to DVD-RW, but may be DVD-R or the like. The media I/F 516 controls reading or writing (storage) of data to a recording medium 515 such as a flash memory.

≪Hardware configuration of spherical imaging device≫
FIG. 3 is a diagram illustrating an example of a hardware configuration when the monitoring device 20 is implemented as an omnidirectional photographing device. In the following, the spherical photographing device is assumed to be a spherical (omnidirectional) photographing device using two image sensors, but any number of image sensors may be used as long as it is two or more. Also, it does not necessarily have to be a device dedicated to omnidirectional photography; by attaching an aftermarket omnidirectional imaging unit to a regular digital camera or smartphone, it can essentially have the same functionality as a omnidirectional photography device. You can.

As shown in FIG. 3, the omnidirectional imaging device in one embodiment includes an imaging unit 601, an image processing unit 604, an imaging control unit 605, a microphone 608, a sound processing unit 609, and a CPU (Central Processing Unit) 611. , ROM (Read Only Memory) 612, SRAM (Static Random Access Memory) 613, DRAM (Dynamic Random Access Memory) 614, operation unit 615, external device connection I/F 616, communication unit 617, antenna 617a, acceleration/direction sensor 618 , a gyro sensor 619, an acceleration sensor 620, and a concave terminal 621 for Micro USB.

Among these, the imaging unit 601 includes wide-angle lenses (so-called fisheye lenses) 602a and 602b each having an angle of view of 180° or more for forming a hemispherical image, and two imaging units provided corresponding to each wide-angle lens. It includes elements 603a and 603b. The image sensors 603a and 603b are image sensors such as CMOS (Complementary Metal Oxide Semiconductor) sensors and CCD (Charge Coupled Device) sensors that convert optical images formed by fisheye lenses 602a and 602b into electrical signal image data and output the image data. It has a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks, and a group of registers in which various commands and parameters necessary for the operation of this image sensor are set.

The imaging elements 603a and 603b of the imaging unit 601 are each connected to the image processing unit 604 via a parallel I/F bus. On the other hand, the imaging elements 603a and 603b of the imaging unit 601 are connected to the imaging control unit 605 via a serial I/F bus (such as an I2C bus). The image processing unit 604, the imaging control unit 605, and the sound processing unit 609 are connected to the CPU 611 via a bus 610. Further, the bus 610 is also connected to a ROM 612, an SRAM 613, a DRAM 614, an operation unit 615, an external device connection I/F (Interface) 616, a communication unit 617, an acceleration/direction sensor 618, and the like.

The image processing unit 604 takes in image data output from the image sensors 603a and 603b through the parallel I/F bus, performs predetermined processing on each image data, and then synthesizes these image data. , create equirectangular projection image data.

Generally, the imaging control unit 605 uses the I2C bus to set commands and the like in register groups of the imaging devices 603a and 603b, with the imaging control unit 605 as a master device and the imaging devices 603a and 603b as slave devices. Necessary commands and the like are received from the CPU 611. Further, the imaging control unit 605 also uses the I2C bus to take in status data and the like of the register groups of the imaging elements 603a and 603b, and sends it to the CPU 611.

Furthermore, the imaging control unit 605 instructs the imaging elements 603a and 603b to output image data at the timing when the shutter button of the operation unit 615 is pressed. Some omnidirectional photography devices may have a preview display function or a function that supports video display on a display (for example, a smartphone display). In this case, image data is output continuously from the image sensors 603a and 603b at a predetermined frame rate (frames/minute).

The imaging control unit 605 also functions as a synchronization control unit that synchronizes the output timing of image data of the imaging elements 603a and 603b in cooperation with the CPU 611, as will be described later. Note that in this embodiment, the omnidirectional photographing device is not provided with a display, but may be provided with a display section.

Microphone 608 converts sound into sound (signal) data. The sound processing unit 609 takes in sound data output from the microphone 608 through the I/F bus, and performs predetermined processing on the sound data.

The CPU 611 controls the overall operation of the omnidirectional photographing apparatus and executes necessary processing. ROM612 stores various programs for CPU611. The SRAM 613 and DRAM 614 are work memories that store programs executed by the CPU 611, data being processed, and the like. In particular, the DRAM 614 stores image data that is currently being processed by the image processing unit 604 and data of processed equirectangular projection images.

The operation unit 615 is a general term for operation buttons such as the shutter button 615a. By operating the operation unit 615, the user inputs various shooting modes, shooting conditions, and the like.

The external device connection I/F 616 is an interface for connecting various external devices. The external device in this case is, for example, a USB (Universal Serial Bus) memory, a PC (Personal Computer), or the like. The data of the equirectangular projection image stored in the DRAM 614 can be recorded on an external media via this external device connection I/F 616, or transferred to an external device such as a smartphone via the external device connection I/F 616 as necessary. It may be sent to a terminal (device).

The communication unit 617 communicates with an external terminal such as a smartphone using short-range wireless communication technology such as Wi-Fi, NFC (Near Field Communication), or Bluetooth (registered trademark) via an antenna 617a provided on the spherical photographing device. Communicate with (device). This communication unit 617 can also transmit data of an equirectangular projection image to an external terminal (device) such as a smartphone.

The acceleration/azimuth sensor 618 calculates the azimuth of the omnidirectional imaging device from the earth's magnetism and outputs azimuth information. This orientation information is an example of related information (metadata) in accordance with Exif, and is used for image processing such as image correction of captured images. Note that the related information also includes data such as the date and time when the image was taken and the data capacity of the image data. Further, the acceleration/azimuth sensor 618 is a sensor that detects changes in angle (roll angle, pitch angle, yaw angle) accompanying movement of the omnidirectional photographing device 6. A change in angle is an example of related information (metadata) according to Exif, and is used for image processing such as image correction of a captured image.

Furthermore, the acceleration/direction sensor 618 is a sensor that detects acceleration in three axial directions. The omnidirectional imaging device calculates the attitude (angle with respect to the direction of gravity) of its own device (the omnidirectional imaging device) based on the acceleration detected by the acceleration/azimuth sensor 618. By providing the acceleration/azimuth sensor 618 in the omnidirectional photographing device, the accuracy of image correction is improved.

≪Hardware configuration of MFP≫
FIG. 4 is a diagram illustrating an example of a hardware configuration when the electronic device 30 is implemented as an MFP.

As shown in FIG. 4, the MFP in one embodiment includes a controller 910, a short-range communication circuit 920, an engine control section 930, an operation panel 940, and a network I/F (Interface) 950.

Of these, the controller 910 includes a CPU (Central Processing Unit) 901, a system memory (MEM-P) 902, a north bridge (NB) 903, a south bridge (SB) 904, and an ASIC (Application Specific Integrated circuit) 906, a local memory (MEM-C) 907 as a storage unit, an HDD (Hard Disk Drive) controller 908, and an HD (Hard Disk) 909 as a storage unit. (Accelerated Graphics Port) bus 921.

Among these, the CPU 901 is a control unit that performs overall control of the MFP. NB903 is a bridge for connecting CPU901, MEM-P902, SB904, and AGP bus 921, and is a memory controller that controls reading and writing to MEM-P902, a PCI (Peripheral Component Interconnect) master, and AGP target. has.

The MEM-P902 includes a ROM (Read Only Memory) 902a, which is a memory for storing programs and data that realizes each function of the controller 910, and a RAM (Random Access Memory) 902b. Note that the program stored in the RAM 902b is configured to be provided as an installable or executable file recorded on a computer-readable recording medium such as a CD-ROM, CD-R, or DVD. You may.

The SB904 is a bridge for connecting the NB903 to PCI devices and peripheral devices. The ASIC 906 is an IC (Integrated Circuit) for image processing that includes hardware elements for image processing, and has the role of a bridge that connects the AGP bus 921, the PCI bus 922, the HDD 908, and the MEM-C 907, respectively. This ASIC906 includes a PCI target and an AGP master, an arbiter (ARB) that is the core of the ASIC906, a memory controller that controls the MEM-C907, and multiple DMACs (Direct Memory Access Controllers) that rotate image data using hardware logic, etc. , and a PCI unit that transfers data between the scanner section 931 and the printer section 932 via the PCI bus 922. Note that the ASIC 906 may be connected to a USB (Universal Serial Bus) interface or an IEEE 1394 (Institute of Electrical and Electronics Engineers 1394) interface.

MEM-C907 is a local memory used as a copy image buffer and code buffer. The HD 909 is a storage for storing image data, font data used during printing, and forms. The HD 909 controls data reading or writing to the HD 909 under the control of the CPU 901 . AGP bus 921 is a bus interface for graphics accelerator cards proposed to speed up graphics processing, and can speed up graphics accelerator cards by directly accessing MEM-P902 with high throughput. .

Further, the short-range communication circuit 920 includes a short-range communication circuit 920a. The short-range communication circuit 920 is a communication circuit such as NFC or Bluetooth.

Further, the engine control section 930 includes a scanner section 931 and a printer section 932. The operation panel 940 also includes a panel display section 940a such as a touch panel that displays current setting values, a selection screen, etc., and accepts input from the operator, as well as displaying setting values for image forming conditions such as density setting conditions. An operation panel 940b is provided, which includes a numeric keypad for accepting instructions, a start key for accepting copy start instructions, and the like. The controller 910 controls the entire MFP, for example, controls drawing, communication, input from the operation panel 940, and the like. The scanner section 931 or the printer section 932 includes image processing sections such as error diffusion and gamma conversion.

Note that the MFP can sequentially switch and select a document box function, a copy function, a printer function, and a facsimile function using an application switching key on the operation panel 940. When the document box function is selected, the mode becomes document box mode, when the copy function is selected, the mode becomes copy mode, when the printer function is selected, the mode becomes printer mode, and when the facsimile mode is selected, the mode becomes facsimile mode.

Further, the network I/F 950 is an interface for data communication using the communication network N1. Near field communication circuit 920 and network I/F 950 are electrically connected to ASIC 906 via PCI bus 922.

≪Hardware configuration of electronic whiteboard≫
FIG. 5 is a diagram illustrating an example of a hardware configuration when the electronic device 30 is implemented as an electronic blackboard.

As shown in FIG. 5, the electronic blackboard in one embodiment includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, an SSD (Solid State Drive) 204, It includes a network I/F (Interface) 205 and an external device connection I/F 206.

Among these, the CPU 201 controls the operation of the entire electronic whiteboard. The ROM 202 stores programs used to drive the CPU 201, such as the CPU 201 and an IPL (Initial Program Loader). RAM 203 is used as a work area for CPU 201. The SSD 204 stores various data such as electronic whiteboard programs. Network controller 205 controls communication with communication network N1. External device connection I/F 206 is an interface for connecting various external devices. The external devices in this case are, for example, a USB (Universal Serial Bus) memory 230 and external devices (microphone 240, speaker 250, camera 260).

The electronic blackboard also includes a capture device 211, a GPU 212, a display controller 213, a contact sensor 214, a sensor controller 215, an electronic pen controller 216, a short-range communication circuit 219, an antenna 219a of the short-range communication circuit 219, a power switch 222, and a selector. It is equipped with switches 223.

Among these, the capture device 211 causes the display of an external PC (Personal Computer) 270 to display video information as a still image or a moving image. A GPU (Graphics Processing Unit) 212 is a semiconductor chip that specializes in graphics. The display controller 213 controls and manages screen display in order to output the output image from the GPU 212 to the display 280 or the like.

The contact sensor 214 detects when the electronic pen 290, the user's hand H, or the like comes into contact with the display 280. Sensor controller 215 controls processing of contact sensor 214. The contact sensor 214 performs coordinate input and coordinate detection using an infrared cutoff method. This method of inputting and detecting coordinates is such that two light receiving and emitting devices installed at both ends of the upper side of the display 280 emit a plurality of infrared rays in parallel to the display 280. This is a method of receiving light that is reflected by a reflecting member and returns along the same optical path as the light emitted by the light receiving element. The contact sensor 214 outputs the ID of the infrared rays emitted by the two light receiving and emitting devices blocked by the object to the sensor controller 215, and the sensor controller 215 identifies the coordinate position that is the contact position of the object.

The electronic pen controller 216 communicates with the electronic pen 290 to determine whether or not the display 280 is touched with the tip of the pen or the end of the pen. The near field communication circuit 219 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The power switch 222 is a switch for turning ON/OFF the power of the electronic blackboard. The selection switches 223 are, for example, a group of switches for adjusting the brightness, shade, etc. of the display on the display 280.

Further, the electronic blackboard includes a bus line 210. The bus line 210 is an address bus, a data bus, etc. for electrically connecting each component such as the CPU 201 shown in FIG. 5.

Note that the contact sensor 214 is not limited to the infrared cutoff type, but can also be a capacitive type touch panel that identifies the touch position by detecting changes in capacitance, or a touch sensor that identifies the touch position by voltage changes between two opposing resistive films. Various detection means may be used, such as a resistive touch panel that uses a resistive film, or an electromagnetic induction touch panel that detects electromagnetic induction caused by a contact object coming into contact with the display section to specify the contact position. Furthermore, the electronic pen controller 216 may determine whether or not a portion of the electronic pen 290 that the user grasps or other portions of the electronic pen is touched, in addition to the pen tip and pen tail of the electronic pen 290.

<Functional configuration of device management system>
Next, an example of the functional configuration of the device management system in this embodiment will be described with reference to FIG. 6. FIG. 6 is a block diagram illustrating the functional configuration of each device included in the device management system in this embodiment.

≪Functional configuration of management device≫
As shown in FIG. 6, the management device 10 in this embodiment includes an image receiving section 11, an operation information receiving section 12, a state managing section 13, an operation instruction section 14, and a state storage section 100. The state management section 13 in this embodiment includes an object state management section 131, a device state management section 132, and an output information determination section 133.

The image receiving unit 11, the operation information receiving unit 12, the state management unit 13, and the operation instruction unit 14 execute processing that causes the CPU 501 and the HDD controller 505 to execute a program that is expanded from the HD 504 to the RAM 503 as shown in FIG. 2, for example. realized by

The state storage unit 100 is realized using, for example, the HD 504 shown in FIG. 2. Reading or writing of data stored in the HD 504 is performed, for example, via the HDD controller 505.

The state storage unit 100 stores state management information for managing the states of devices and objects existing in the managed space R1. The state management information in this embodiment includes a registered object list, a registered device list, and a detected object list.

The registered object list is a list that manages information regarding objects existing in the management target space R1. The object in this embodiment is a person.

The registered device list is a list that manages information regarding devices existing in the managed space R1. The device in this embodiment is an electronic device 30.

The detected object list is a list that manages objects detected in the management target space R1. The detected object list is a temporary list used to update the registered object list.

Here, the state management information in this embodiment will be explained with reference to FIG. 7. FIG. 7(A) is a conceptual diagram showing an example of a registered object list.

As shown in FIG. 7A, the registered object list in this embodiment includes a registered object ID, a group ID, an existence confirmation flag, attribute information, and device operation information as data items.

The registered object ID is identification information that identifies an object included in the registered object list.

The group ID is identification information that identifies a group to which a plurality of registered objects belong. A group ID is assigned when it is determined that some of the registered objects included in the registered object list belong to the same group. The group ID of registered objects that are not determined to belong to a group is set to an initial value (for example, zero).

A group is, for example, co-workers, parents and children, or friends. Members belonging to a group jointly carry out a job such as a meeting or shopping. Therefore, by recognizing groups, it is expected that the behavior of people can be understood more accurately. For example, when multiple employees hold a meeting in a conference room equipped with an MFP and an electronic whiteboard, it would be convenient if one employee authenticates with the MFP and other employees can omit authentication when using the electronic whiteboard. expensive.

The existence confirmation flag is a flag indicating whether or not the registered object exists in the managed space R1. For example, the presence confirmation flag is set to 1 if it exists, and 0 if it does not exist.

Attribute information is information accompanying a registered object and is information necessary for management. In this embodiment, it is the position of the registered object and the time when the position was confirmed (hereinafter also referred to as "image acquisition time"). The position of the registered object is expressed in three-dimensional coordinates (ie, numerical values on each axis in an orthogonal coordinate system such as the X-axis, Y-axis, Z-axis, etc.).

The method for acquiring position information differs depending on the type of monitoring device 20. For example, if the monitoring device 20 is a network camera, a known technique can be used, such as acquiring the distance to the object with a stereo camera and mapping it along with the direction to three-dimensional coordinates.

Note that the attribute information may include various information indicating the characteristics of the registered object in addition to the position of the registered object. For example, the color or shape of the registered object can be used. When the object is a person, the shape changes depending on the posture, so it is preferable to use color. Since the number of types of colors or the area of each color also changes depending on the posture, it is preferable to use as a condition whether at least one color is continuous when determining whether objects are the same.

The device operation information is information regarding the device operation performed by the registered object. The device operation information may include input information input by a user in operating the device. The device operation information includes, for example, the function executed by the user on the electronic device, the time of operation, charge information as compensation for the operation, language used, and the like. The input information is authentication information in an authentication operation, setting information in a device setting operation, or the like.

The number of objects existing in the managed space R1 changes from moment to moment. Therefore, the number of registered objects included in the registered object list is variable.

FIG. 7(B) is a conceptual diagram showing an example of a registered device list. As shown in FIG. 7(B), the registered device list in this embodiment includes a registered device ID, attribute information, device input information, and device output information as data items.

The registered device ID is identification information that identifies a device included in the registered device list.

Attribute information is information associated with registered devices and is information necessary for management. In this embodiment, the information includes the location of the registered device, power on/off status, whether or not it is in sleep mode, whether or not authentication is required, option installation status, and status information such as under maintenance/scheduled maintenance time/in energy saving mode. The position of the registered device is expressed in three-dimensional coordinates similarly to the registered object.

Device input information (hereinafter also referred to as "usage information") is predetermined information that is managed when a registered device is operated. The device input information may include input information input by a user in operating the device. The device input information includes, for example, the function executed by the user on the registered device, the time of operation, information on the fee to be charged as compensation for the operation, and the like. The device input information may include screen setting information set by the user in operating the device. The screen setting information includes, for example, the language used, the font of characters used, the size of characters used, and the like.

The device output information is information registered in advance for the registered device. The device output information is a list that associates output conditions that trigger the transmission of an operation instruction with operation instructions to be transmitted to the registered device when the output conditions are met.

The number of devices installed in the managed space R1 may change. Therefore, the number of registered devices included in the registered device list is variable.

FIG. 7(C) is a conceptual diagram showing an example of a detected object list. As shown in FIG. 7C, the detected object list in this embodiment includes a detected object ID, group ID, and attribute information as data items.

The detected object ID is identification information that identifies an object included in the detected object list.

The group ID is identification information that identifies a group that includes a plurality of detected objects. A group ID is assigned when it is determined that some of the detected objects included in the detected object list belong to the same group. The group ID of a detected object that is not determined to belong to a group is set to an initial value (for example, zero).

Attribute information is information accompanying a detected object and is information necessary for management. In this embodiment, these are the position of the detected object and the image acquisition time. The position of the detected object is expressed in three-dimensional coordinates similarly to the registered object.

The number of objects detected in the managed space R1 changes from moment to moment. Therefore, the number of detected objects included in the detected object list is variable.

The explanation will be returned to FIG. 6. The image receiving unit 11 receives images from the monitoring device 20. The image receiving section 11 sends the received image to the state managing section 13 in response to a request from the state managing section 13 .

The operation information receiving unit 12 receives device operation information from the electronic device 30. The operation information receiving section 12 sends the received device operation information to the state management section 13 in response to a request from the state management section 13 .

The state management section 13 uses the object state management section 131 and the device state management section 132 to manage the states of the electronic device 30 and the object. Further, the state management unit 13 uses the output information determination unit 133 to determine the content of the operation instruction to be transmitted to the electronic device 30.

The object state management section 131 updates the registered object list stored in the state storage section 100 based on the image received from the image reception section 11.

The device state management section 132 updates the registered device list and the registered object list stored in the state storage section 100 based on the device operation information received from the operation information reception section 12.

The output information determining unit 133 determines the content of the operation instruction to be transmitted to the electronic device 30 based on the status management information stored in the status storage unit 100. The output information determining section 133 sends the determined operation instruction to the operation instruction section 14.

The operation instruction unit 14 transmits the operation instruction received from the state management unit 13 to the electronic device 30.

≪Functional configuration of monitoring device≫
As shown in FIG. 6, the monitoring device 20 in this embodiment includes an image acquisition section 21 and an image transmission section 22.

The image acquisition unit 21 acquires an image including the vicinity of the electronic device 30 installed in the managed space R1. The image acquisition unit 21 is realized, for example, by a process in which a program loaded from the ROM 612 to the SRAM 613 shown in FIG. 3 is executed by the CPU 611 and the imaging control unit 605.

The image transmitting unit 22 transmits the image acquired by the image acquiring unit 21 to the management device 10. The image transmitter 22 is realized, for example, by a process in which a program expanded from the ROM 612 to the SRAM 613 shown in FIG. 3 is executed by the CPU 611 and the external device connection I/F 616.

≪Functional configuration of electronic equipment≫
As shown in FIG. 6, the electronic device 30 in this embodiment includes an operation information transmitter 31 and a device controller 32.

The operation information transmitting unit 31 and the device control unit 32 are realized, for example, by processing that causes the CPU 901 and the HDD controller 908 to execute a program loaded from the HD 909 onto the RAM 902b shown in FIG.

The operation information transmitting unit 31 transmits device operation information regarding operations performed by the user to the management device 10.

The device control unit 32 receives operation instructions from the management device 10. The device control unit 32 controls the operation of the electronic device 30 based on the received operation instruction. For example, the device control unit 32 authenticates the user using authentication information included in the operation instruction. Further, for example, the device control unit 32 changes the settings of the electronic device 30 using the setting information included in the operation instruction.

<Processing procedure of device management method>
Next, a device management method executed by the device management system in this embodiment will be described with reference to FIGS. 8 to 20.

≪Basic flowchart≫
The device management method in this embodiment includes many double loop processes in which all combinations of data included in a certain list and data included in a certain list are processed. Therefore, in this embodiment, a basic flowchart showing the framework of double loop processing is introduced, and the following explanation will focus on the processing in the basic flowchart. Note that each process in the basic flowchart is a process performed on one combination of two data.

The basic flowchart in this embodiment includes two basic flowcharts. The first basic flowchart is used when processing two data contained in different lists in a double loop. The second basic flowchart is used when processing two data included in the same list in a double loop.

FIG. 8 is a flowchart showing an example of the first basic flowchart in this embodiment.

In step S101, list A is read. The number of data in list A is assumed to be N. In step S102, list B is read. The number of data in list B is assumed to be M.

In step S103, a variable n is initialized to 1. In step S104, a variable m is initialized to 1. In step S105, processing A1 is executed.

In step S106, it is determined whether the variable m is equal to the number M of data. This means whether or not combinations of the n-th data in list A with all data in list B have been processed. If the variable m is different from the data number M (NO), the process advances to step S107. If the variable m is equal to the data number M (YES), the process advances to step S109.

In step S107, processing A3 is executed. In step S108, variable m is incremented. After that, the process returns to step S106.

In step S109, processing A2 is executed. In step S110, it is determined whether the variable n is equal to the number N of data. This means whether or not all data in list A has been combined with all data in list B. If the variable n is different from the data number N (NO), the process advances to step S111. If the variable n is equal to the data number N (YES), the process advances to step S113.

In step S111, processing A5 is executed. In step S112, variable n is incremented. After that, the process returns to step S104.

In step S113, processing A4 is executed. With the above, processing for all combinations of data included in list A and list B is completed.

FIG. 9 is a flowchart showing an example of the second basic flowchart in this embodiment.

In step S201, list A is read. The number of data in list A is assumed to be N.

In step S202, a variable n is initialized to 1. In step S203, n+1 is assigned to variable m. In step S204, processing B1 is executed.

In step S205, it is determined whether the variable m is equal to N. This means whether or not combinations of the n-th data in list A with all data in list A have been processed. If the variable m is different from N (NO), the process advances to step S206. If the variable m is equal to N (YES), the process advances to step S208.

In step S206, processing B3 is executed. In step S207, the variable m is incremented. After that, the process returns to step S204.

In step S208, processing B2 is executed. In step S209, it is determined whether the variable n is equal to N-1. This means whether all data in list A has been combined with all other data in list A. If the variable n is different from N-1 (NO), the process advances to step S210. If the variable n is equal to N-1 (YES), the process advances to step S212.

In step S210, processing B5 is executed. In step S211, the variable n is incremented. After that, the process returns to step S203.

In step S212, processing B4 is executed. With the above, processing for all combinations of data included in list A is completed.

≪Device management method≫
FIG. 10 is a flowchart illustrating an example of a device management method executed by the device management system in this embodiment.

The device management system repeatedly executes the flowchart shown in FIG. 10 at predetermined time intervals. Although the time interval may be set arbitrarily, it is, for example, 3 seconds.

In step S1, the image acquisition unit 21 included in the monitoring device 20 acquires a first image of the vicinity of the electronic device 30-1. Furthermore, the image acquisition unit 21 acquires a second image of the vicinity of the electronic device 30-2. The monitoring device 20 is installed so that the vicinity of the electronic devices 30-1 and 30-2 installed in the managed space R1 is included in the angle of view. Therefore, the first image acquired by the image acquisition unit 21 captures the vicinity of the electronic device 30-1. Furthermore, the second image acquired by the image acquisition unit 21 captures the vicinity of the electronic device 30-2.

The vicinity of the electronic device 30-1 or the electronic device 30-2 is the range of positions where the user can operate the electronic device 30-1 or the electronic device 30-2 (hereinafter also referred to as "operable position"). . The operable position is a position where the user can reach the electronic device 30-1 or the electronic device 30-2. The operable position is, for example, a position within 1 meter from the electronic device 30-1 or the electronic device 30-2. Therefore, the first image is an image of the electronic device 30-1 and the user using the electronic device 30-1. Further, the second image is an image taken of the electronic device 30-2 and the user using the electronic device 30-2.

If the operable positions of the electronic device 30-1 and the electronic device 30-2 can be photographed in one image, the operable positions of the electronic device 30-1 and the electronic device 30-2 can be captured in one image. You may also photograph the location. In this case, the first image and the second image are the same image.

The first image and second image acquired by the image acquisition unit 21 are not limited to still images, and may be moving images.

Next, the image acquisition unit 21 sends the acquired first image and second image to the image transmission unit 22. The image transmitter 22 receives the first image and the second image from the image acquirer 21. Next, the image transmitting unit 22 transmits the received first image and second image to the management device 10. In the management device 10, the image receiving unit 11 receives the first image and the second image from the monitoring device 20.

In step S2, the state management unit 13 included in the management device 10 requests the image reception unit 11 for the first image and the second image. Next, the state management section 13 inputs the first image and the second image received from the image reception section 11 to the object state management section 131.

Subsequently, the object state management unit 131 executes an object state update process to be described later, and updates the registered object list of the state management information stored in the state storage unit 100. When there are a plurality of monitoring devices 20, the object state management unit 131 executes object state update processing for each of the plurality of images received from each of the plurality of monitoring devices 20.

≪Object state update processing≫
Here, the object state update process in this embodiment will be explained with reference to FIGS. 11 to 16. FIG. 11 is a flowchart illustrating an example of the object state update process (step S2 in FIG. 10) executed by the object state management unit 131 in this embodiment.

In step S<b>21 , the object state management unit 131 executes a detected object list creation process, which will be described later, and creates a detected object list based on the image input from the state management unit 13 .

FIG. 12 is a flowchart illustrating an example of the detected object list creation process (step S21 in FIG. 11) executed by the object state management unit 131 in this embodiment.

In step S21-1, the object state management section 131 clears the detected object list stored in the state storage section 100. That is, all data included in the detected object list is deleted.

In step S21-2, the object state management section 131 acquires the image input from the state management section 13. The image may be one image received from one monitoring device 20 or multiple images received from multiple monitoring devices 20.

Next, the object state management unit 131 divides the image into predetermined blocks. Subsequently, the object state management unit 131 measures the distance between the camera and the object photographed in the image for each block. Further, the object state management unit 131 obtains three-dimensional coordinates from the measured distance. A known method may be used to convert the distance into three-dimensional coordinates.

In step S21-3, the object state management unit 131 acquires the image acquisition time T from the acquired image. If the image acquisition time cannot be acquired from the image, the current time may be acquired as the image acquisition time T.

In step S21-4, the object state management unit 131 analyzes the acquired image and detects the object photographed in the image. In this embodiment, a person photographed in an image is detected.

A known method can be used to detect a person. For example, pattern matching using a machine learning model can be used. Specifically, a person can be extracted from an image using a known region-based convolutional neural network (R-CNN) or the like.

Furthermore, detection accuracy can be improved by pattern matching the cut out image of the person with a comparison image stored in advance. Note that in addition to these methods, various known person detection methods can be used.

In step S21-5, the object state management unit 131 assigns a detected object ID that identifies the detected object.

In step S21-6, the object state management unit 131 registers the detected object ID and attribute information (position of the detected object and image acquisition time) in the detected object list. Note that at this stage, the group ID is set to an initial value indicating that it does not belong to a group.

The explanation will be returned to FIG. 11. In step S22, the object state management unit 131 executes a registered object list update process, which will be described later, and updates the registered object list based on the detected object list.

13 and 14 are flowcharts showing an example of the registered object list update process (step S22 in FIG. 11) executed by the object state management unit 131 in this embodiment.

The registered object list update process is performed within the framework of the first basic flowchart (see FIG. 8). Specifically, the detected object list is set to list A, the registered object list is set to list B, and processing A1 (FIG. 13) and processing A4 (FIG. 14) are executed for all combinations of detected objects and registered objects. Note that there is no process corresponding to processes A2, A3, and A5.

FIG. 13 is a flowchart illustrating an example of the process A1 of the registered object list update process. Process A1 is a process for determining the identity of the registered object and the detected object and updating the position of the registered object.

Here, it is assumed that the process is performed on a combination of the nth detected object in the detected object list and the mth registered object in the registered object list.

In step S22-1, the object state management unit 131 clears the presence confirmation flag of the m-th registered object. That is, the presence confirmation flag of the m-th registered object is set to 0. This means that it is unclear whether or not the registered object exists in the managed space R1.

In step S22-2, the object state management unit 131 calculates the distance X between the three-dimensional coordinates of the m-th registered object and the three-dimensional coordinates of the n-th detected object.

In step S22-3, the object state management unit 131 determines whether the distance X is less than or equal to a predetermined threshold (for example, 1 meter). If the distance X exceeds the threshold (NO), the object state management unit 131 ends processing A1. This means that it has been determined that they are not the same person because they are far apart. If the distance X is less than or equal to the threshold (YES), the object state management unit 131 advances the process to step S22-4.

In step S22-4, the object state management unit 131 adds the three-dimensional position and image acquisition time of the n-th detected object to the attribute information of the m-th registered object. This means that because the distance between them is close, it is determined that they are the same person and that the position of the n-th detected object is the position of the m-th registered object at the image acquisition time T.

As long as the registered object is detected from the image, the attribute information in the registered object list will increase each time the object state update process is executed. In other words, the attribute information of the registered object list represents a time series of the positions at which the registered object is detected within the managed space. Therefore, the movement of the registered object can be tracked using the attribute information of the registered object list.

In step S22-5, the object state management unit 131 sets the existence confirmation flag of the m-th registered object. That is, the presence confirmation flag of the m-th registered object is set to 1. This means that since an object matching the registered object was detected in step S22-3, it was determined that the registered object is present in the managed space R1 even at the image acquisition time T.

FIG. 14 is a flowchart illustrating an example of process A4 of the registered object list update process. Process A4 is a process for deleting registered objects whose existence could not be confirmed in process A1 from the registered object list.

In step S22-6, the object state management unit 131 initializes the variable m to 1.

In step S22-7, the object state management unit 131 determines whether the existence of the m-th registered object has been confirmed. Specifically, it is determined whether the presence confirmation flag is 1 or 0. If the presence confirmation flag is 1 (YES), the object state management unit 131 advances the process to step S22-9. If the existence confirmation flag is 0 (NO), the object state management unit 131 advances the process to step S22-8.

In step S22-8, the object state management unit 131 deletes the m-th registered object from the registered object list.

In step S22-9, the object state management unit 131 determines whether the variable m is equal to the number M of data. If the variable m is different from the data number M (NO), the object state management unit 131 advances the process to step S22-10. If the variable m is equal to the data number M (YES), the object state management unit 131 ends the process.

In step S22-10, the object state management unit 131 increments the variable m. Thereafter, the object state management unit 131 returns the process to step S22-7.

The explanation will be returned to FIG. 11. In step S23, the object state management unit 131 executes group ID assignment processing, which will be described later, and assigns a group ID to the registered objects included in the registered object list.

The group ID assignment process is performed within the framework of the second basic flowchart (see FIG. 9). Specifically, the registered object list is set as list A, and processing B1 (FIG. 15 or 16) is executed for all combinations of two registered objects. Note that there is no process equivalent to processes B2-B5.

Various methods can be considered for the process of determining whether a certain object and another object are included in the same group. In this embodiment, group determination based on positional proximity and group determination based on group behavior will be described. However, the method of group determination is not limited to these, and any technique may be used as long as it is possible to determine groups of objects from images.

FIG. 15 is a flowchart showing a first example of the group ID assignment process (step S23 in FIG. 11) executed by the object state management unit 131 in this embodiment. A first example of group ID assignment processing is group determination based on positional proximity.

Group determination based on positional proximity is a process of registering two objects that have been detected in close proximity a predetermined number of times as a group. The predetermined number of times is, for example, five times. If the time interval for executing the device management method is 3 seconds, two objects detected at close positions for 15 consecutive seconds will be determined to be in the same group.

Here, it is assumed that the process is executed for the combination of the m-th registered object and the n-th registered object in the registered object list.

In step S23A-1, the object state management unit 131 initializes a variable k and a variable j to 1. The variable k is a counter representing the number of times it has been determined whether two objects are close to each other. Variable j is a counter representing the number of times it is determined that two objects are close to each other.

In step S23A-2, the object state management unit 131 calculates the distance X between the three-dimensional coordinates of the m-th registered object k times before and the three-dimensional coordinates of the n-th registered object k times before.

In step S23A-3, the object state management unit 131 determines whether the distance X is less than a predetermined threshold (for example, 1 meter). If the distance X is less than the threshold (YES), the object state management unit 131 advances the process to step S23A-4. If the distance X is equal to or greater than the threshold (NO), the object state management unit 131 advances the process to step S23A-5.

In step S23A-4, the object state management unit 131 increments the variable j.

In step S23A-5, the object state management unit 131 increments the variable k.

In step S23A-6, the object state management unit 131 determines whether the variable k is equal to the predetermined number of times K. If the variable k is different from the predetermined number of times K (NO), the object state management unit 131 returns the process to step S23A-2. If the variable k is equal to the predetermined number of times K (YES), the object state management unit 131 advances the process to step S23A-7.

In step S23A-7, the object state management unit 131 determines whether the variable j is equal to the predetermined number of times K. If the variable j is different from the predetermined number of times K (NO), the object state management unit 131 ends the process. If the variable j is equal to the predetermined number of times K (YES), the object state management unit 131 advances the process to step S23A-8.

In step S23A-8, the object state management unit 131 determines whether a group ID is assigned to either the m-th registered object or the n-th registered object. If a group ID has been assigned (YES), the object state management unit 131 advances the process to step S23A-9. If a group ID has not been assigned (NO), the object state management unit 131 advances the process to step S23A-10.

In step S23A-9, the object state management unit 131 determines one of the group IDs assigned to the m-th registered object and the n-th registered object as the group ID to be assigned. Next, the object state management unit 131 sets the determined group ID to the group ID of the m-th registered object and the n-th registered object in the registered object list.

The object state management unit 131 determines the group ID to be assigned as follows. If a group ID is assigned to only one registered object, the group ID is set as the assigned group ID. That is, registered objects to which no group ID is assigned are added to existing group members.

If both registered objects are assigned group IDs, first, the registered objects after the m+1st are sorted by group ID to identify the registered object having the same group ID as the m-th registered object. Next, the group IDs of the m-th registered object and all identified registered objects are updated with the group ID of the n-th registered object. As a result, all members of the group to which the m-th registered object belongs belong to the same group as the n-th registered object. This makes it possible to understand groups to which three or more registered objects belong.

In step S23A-10, the object state management unit 131 issues a new group ID that does not overlap with other group IDs. Next, the object state management unit 131 sets the issued new group ID to the group ID of the m-th registered object and the n-th registered object in the registered object list.

FIG. 16 is a flowchart showing a second example of the group ID assignment process (step S23 in FIG. 11) executed by the object state management unit 131 in this embodiment. A second example of group ID assignment processing is group determination based on group actions.

Group determination based on group behavior is a process in which group determination is performed by image analysis after creating a detected object list. Therefore, when performing group determination based on group actions, step S23 is executed between step S21 and step S22.

Here, it is assumed that the process is executed for a combination of the m-th detected object and the n-th detected object in the detected object list.

In step S23B-1, the object state management unit 131 calculates the distance X between the three-dimensional coordinates of the n-th sensing object and the three-dimensional coordinates of the m-th sensing object.

In step S23B-2, the object state management unit 131 determines whether the distance X is less than a predetermined threshold (for example, 1 meter). If the distance X is equal to or greater than the threshold (NO), the object state management unit 131 ends the process. If the distance X is less than the threshold (YES), the object state management unit 131 advances the process to step S23B-3.

In step S23B-3, the object state management unit 131 calculates the midpoint between the three-dimensional coordinates of the n-th sensing object and the three-dimensional coordinates of the m-th sensing object. Specifically, the object state management unit 131 divides the sum of the X coordinate, Y coordinate, and Z coordinate of the two detected objects by two.

In step S23B-4, the object state management unit 131 extracts an image P having a radius of Y pixels centered on the midpoint. The radius Y is set according to the resolution of the image so that the range included in the image P is about 1 m in real space.

In step S23B-5, the object state management unit 131 reads out the group action image stored in advance. A group action image is a collection of images representing actions determined to belong to the same group. For example, it is an image of two people greeting each other or shaking hands.

In step S23B-6, the object state management unit 131 calculates the degree of similarity between the video P and each group action image by pattern matching or the like.

In step S23B-7, the object state management unit 131 determines whether any of the calculated similarities exceeds a predetermined threshold. If the degree of similarity exceeds the threshold (YES), the object state management unit 131 advances the process to step S23B-8. If there is no similarity exceeding the threshold (NO), the object state management unit 131 ends the process.

In step S23B-8, the object state management unit 131 issues a new group ID that does not overlap with other group IDs. Next, the object state management unit 131 sets the issued new group ID to the group ID of the m-th detected object and the n-th detected object in the detected object list.

In step S22-4 (see FIG. 13) of the registered object list update process executed thereafter, the object state management unit 131 sets the group ID of the detected objects determined to be the same object to the group ID of the registered objects. do. Note that if an existing group ID is assigned to the registered object, steps S23A-8 to S23A-10 (see FIG. 15) of the first example of group ID assignment processing are executed.

The explanation will be returned to FIG. 10. In step S3, the operation information transmitting unit 31 included in the electronic device 30-1 determines whether the operation content is information that should be notified to the management device 10, depending on the operation performed by the user on the electronic device 30-1. Determine. This determination is made based on whether or not the content matches any of the predetermined operation contents.

When the operation information transmitting unit 31 determines that the operation content should be notified, the operation information transmitting unit 31 transmits device operation information related to the device operation to the management device 10. In the management device 10, the operation information receiving section 12 receives device operation information from the electronic device 30-1.

The operation details and operation information in this embodiment are illustrated below.

<Example 1>
Operation details: Device login Operation information: Registered device ID, device authentication ID (if authentication is required), login time

<Example 2>
Operation details: Processing that incurs charges (copy output, use of pay-as-you-go software, etc.)
Operation information: Registered device ID, device authentication ID (if authentication is required), fee, input data, language used

<Example 3>
Operation details: None Operation information: Status information such as maintenance, energy saving mode, etc.

The registered device ID is identification information included in the registered device list stored in the status storage unit 100 of the management device 10. A registered device ID is assigned to each electronic device 30 installed in the managed space R1. The device authentication ID is authentication information used for authentication for the user of the electronic device 30 to use the electronic device 30.

In step S4, the state management section 13 included in the management device 10 requests operation information from the operation information reception section 12. Next, the state management section 13 inputs the operation information received from the operation information reception section 12 to the device state management section 132.

Subsequently, the device status management unit 132 executes a device status update process to be described later, and updates the registered device list of the status management information stored in the status storage unit 100.

≪Device status update processing≫
Here, the device status update process in this embodiment will be described with reference to FIG. 17. FIG. 17 is a flowchart illustrating an example of the device status update process (step S4 in FIG. 10) executed by the device status management unit 132 in this embodiment.

In step S41, the device status management unit 132 identifies registered devices included in the registered device list based on the registered device ID included in the device operation information. Next, the device status management unit 132 sets the received device operation information to the device input information of the specified registered device.

In step S42, the device state management unit 132 initializes a variable n to 1.

In step S43, the device status management unit 132 calculates the distance X between the three-dimensional coordinates of the specified registered device and the latest three-dimensional coordinates of the nth registered object. Note that the three-dimensional coordinates of the registered device are set in advance using the same method as the three-dimensional coordinates of the registered object. Since the installation position of the registered device may be moved, it is necessary to periodically update the three-dimensional coordinates of the registered device, but the update frequency may be low.

In step S44, the device state management unit 132 determines whether the distance X is less than a predetermined threshold (for example, 1 meter). If the distance X is equal to or greater than the threshold (NO), the device state management unit 132 advances the process to step S47. If the distance X is less than the threshold (YES), the device state management unit 132 advances the process to step S45.

In step S45, the device state management unit 132 adds the device input information of the specified registered device to the device operation information of the nth registered object. The device operation information in the registered object list is configured to be able to store a predetermined number of device input information.

In step S46, the device state management unit 132 determines whether the variable n is equal to the number N of data. If the variable n is different from the data number N (NO), the device state management unit 132 advances the process to step S47. If the variable n is equal to the data number N (YES), the device state management unit 132 ends the process.

In step S47, the device state management unit 132 increments the variable n. After that, the device status management unit 132 returns the process to step S43.

The explanation will be returned to FIG. 10. In step S5, the output information determination unit 133 included in the management device 10 executes an output information determination process, which will be described later, and issues an operation instruction to be transmitted to the electronic device 30 based on the status management information stored in the status storage unit 100. decide.

<<Output information determination process>>
Here, the output information determination process in this embodiment will be explained with reference to FIGS. 18 and 19. FIG. 18 is a flowchart illustrating an example of the output information determination process (step S5 in FIG. 10) executed by the output information determination unit 133 in this embodiment.

The output information determination process is performed within the framework of the first basic flowchart (see FIG. 8). Specifically, the registered device list is set to list A, the registered object list is set to list B, and processing A1 (FIG. 18) is executed for all combinations of registered devices and registered objects. Note that there is no process equivalent to processes A2-A5.

Here, it is assumed that the process is executed for the combination of the nth registered device in the registered device list and the mth registered object in the registered object list.

In step S51, the output information determining unit 133 calculates the distance X between the latest three-dimensional coordinates of the n-th registered device and the latest three-dimensional coordinates of the m-th registered object.

In step S52, the output information determining unit 133 determines whether the distance X is less than a predetermined threshold (for example, 1 meter). If the distance X is equal to or greater than the threshold (NO), the device state management unit 132 ends the process. If the distance X is less than the threshold (YES), the device state management unit 132 advances the process to step S53.

In step S53, the output information determining unit 133 acquires the output conditions included in the device output information of the nth registered device from the registered device list stored in the status storage unit 100.

In step S54, the output information determining unit 133 acquires device operation information for the m-th registered object from the registered object list stored in the state storage unit 100.

In step S55, the output information determining unit 133 determines whether any of the device operation information acquired in step S54 satisfies the output conditions acquired in step S53. If all of the device operation information does not satisfy the output conditions (NO), the device status management unit 132 ends the process. If any of the device operation information satisfies the output conditions (YES), the device status management unit 132 advances the process to step S56.

In step S56, the output information determining unit 133 obtains an operation instruction corresponding to the output condition obtained in step S53 from the device output information of the nth registered device. Note that the operation instructions for the registered device may include instructions for adaptively performing processing according to the attribute information of the registered device. For example, a condition such as not transmitting an operation instruction when the registered device is in an inoperable state may be included. The inoperable state is, for example, a power-off state or a hibernation state.

FIG. 19 is a flowchart showing a modification of the output condition determination process (step S55 in FIG. 18) executed by the output information determining unit 133 in this embodiment.

In the output information determination process shown in FIG. 18, in step S55, it is determined whether the output condition is satisfied based only on the device operation information of the m-th registered object. In the output information determination process shown in FIG. 19, it is determined whether the output conditions are satisfied by referring to device operation information other than the m-th registered device.

In step S55-1, the output information determining unit 133 determines whether any of the device operation information of the m-th registered object satisfies the output conditions of the n-th registered device. If any of the device operation information satisfies the output conditions (YES), the output information determining unit 133 advances the process to step S55-10. If all of the device operation information does not satisfy the output conditions (NO), the output information determining unit 133 advances the process to step S55-2.

In step S55-2, the output information determining unit 133 determines whether device operation information for other registered objects is necessary based on the output conditions of the nth registered device. If device operation information for another registered object is required (YES), the output information determining unit 133 advances the process to step S55-3. If device operation information for other registered objects is not required (NO), the output information determining unit 133 advances the process to step S55-9.

In step S55-3, the output information determining unit 133 initializes a variable k to 1.

In step S55-4, the output information determining unit 133 determines whether variable k is equal to variable m. If the variable k is equal to the variable m (YES), the output information determining unit 133 advances the process to step S55-5. If the variable k is not equal to the variable m (NO), the output information determining unit 133 advances the process to step S55-7.

In step S55-5, the output information determining unit 133 acquires the device operation information of the kth registered object from the registered object list stored in the state storage unit 100.

In step S55-6, the output information determining unit 133 determines whether any of the device operation information of the kth registered object satisfies the output condition of the nth registered device. If any of the device operation information satisfies the output conditions (YES), the output information determining unit 133 advances the process to step S55-10. If all of the device operation information does not satisfy the output conditions (NO), the output information determining unit 133 advances the process to step S55-7.

In step S55-7, the output information determining unit 133 determines whether the variable k is equal to the number M of data. If the variable k is different from the data number M (NO), the output information determining unit 133 advances the process to step 55-8. If the variable m is equal to the data number M (YES), the output information determining unit 133 advances the process to step S55-9.

In step S55-8, the output information determining unit 133 increments the variable k. After that, the output information determining unit 133 returns the process to step S55-4.

In step S55-9, the output information determining unit 133 determines that the device operation information obtained in step S54 does not satisfy the output condition obtained in step S53, and ends the process.

In step S55-10, the output information determining unit 133 determines that the device operation information obtained in step S54 satisfies the output condition obtained in step S53, and ends the process.

FIG. 20 is a diagram showing an example of device output information in this embodiment. A first example of the output condition of device output information in this embodiment is an output condition that does not require device operation information of other registered objects. A second example of the output condition of device output information in this embodiment is an output condition that requires device operation information of another registered object.

As shown in FIG. 20, the first example of the output condition in this embodiment is that the m-th registered object (hereinafter also referred to as "person α") is connected to another electronic device (here, an MFP). ) is authentication information that can be authenticated by the n-th electronic device (here, an electronic whiteboard).

As shown in FIG. 20, when it is determined that the first example of the output condition is satisfied, one or more of the following operation instructions are determined as the operation instructions to be transmitted to the registered device. The first operation instruction is to automatically authenticate the person α to the electronic blackboard. In this case, the operation instruction to be transmitted includes the authentication information input by the person α using the MFP. The second operation instruction is to display "Mr. α, automatic authentication has been performed" on the electronic blackboard for a few seconds.

The third action instruction, if person α is inputting information (for example, a scanning operation) on the MFP, will say, "Do you want to display the information you just entered on device ○○ (name of MFP)?" and a "yes" or "no" selection button. At this time, if "Yes" is selected, the information entered with the scanner will be displayed on the electronic blackboard.

The fourth operation instruction is to identify the language set in the MFP from the device operation information of the person α, and to reflect it on the display on the electronic blackboard. The fifth operation instruction is not to transmit an operation instruction if the attribute information of the electronic whiteboard is in a dormant state such as "in energy saving mode."

As shown in FIG. 20, the second example of the output condition in this embodiment is that the kth registered object (hereinafter also referred to as "person β") is connected to another electronic device (here, an MFP). ) is authentication information that can be authenticated by the n-th electronic device (here, an electronic whiteboard). Note that it is assumed that the person β has the same group ID as the person α.

As shown in FIG. 20, when it is determined that the second example of the output condition is satisfied, one or more of the following operation instructions are determined as the operation instructions to be transmitted to the registered device. The first operation instruction is to automatically authenticate the person α to the electronic blackboard. In this case, the operation instruction to be transmitted includes the authentication information input by the person β using the MFP. The second operation instruction is to display for a few seconds the message "Automatic authentication has been performed using Mr. β's authentication information" on the electronic blackboard.

The third action instruction is "Do you want to display the information that Mr. β entered on device ○○ (name of MFP)" when person β is inputting information (for example, scanning operation) on the MFP? This is to display the words ``Yes'' or ``No'' selection buttons. At this time, if "Yes" is selected, the information entered with the scanner will be displayed on the electronic blackboard.

The explanation will be returned to FIG. 10. In step S6, the output information determining unit 133 included in the management device 10 sends operation instruction information representing an operation instruction to the operation instruction unit 14. The operation instruction information includes information indicating the determined operation instruction and the destination electronic device 30-2.

The operation instruction section 14 receives operation instruction information from the output information determination section 133. Next, the operation instruction unit 14 transmits the operation instruction included in the operation instruction information to the destination electronic device 30-2 included in the operation instruction information.

In the electronic device 30-2, the device control unit 32 receives an operation instruction from the management device 10. Next, the device control unit 32 executes an operation according to the received operation instruction using the input information included in the operation instruction. For example, the device control unit 32 authenticates the user using authentication information included in the operation instruction. Further, for example, the device control unit 32 changes the settings of the electronic device 30-2 using the setting information included in the operation instruction.

<Effects of embodiment>
The device management system 1 according to the present embodiment tracks the movement of a person based on images taken near the electronic device 30, and provides operation instructions using information input by the user on the first electronic device 30-1. is transmitted to the second electronic device 30-2 that has detected that the user is nearby. Therefore, according to the device management system 1 in this embodiment, it is possible to cause the second electronic device to perform an operation using the information input to the first electronic device.

In particular, by sending an authentication instruction using the authentication information authenticated by the first electronic device 30-1 to the second electronic device 30-2, the user authenticated by the first electronic device 30-1 can Authentication can be performed automatically when moving near the second electronic device 30-2. That is, according to the device management system in this embodiment, single sign-on can be realized through hardware cooperation.

[Application example]
In the above embodiment, the management target space is assumed to be an office conference room, an office room, or the like, and an example in which the electronic device 30 is managed as an OA (Office Automation) device has been mainly described. However, the usage scenes to which the device management system 1 can be applied are not limited to this, and can be applied to various usage scenes.

For example, the device management system 1 can be configured to manage electronic devices installed in a hotel. The hotel is equipped with various electronic devices. For example, an automatic check-in machine may be installed in the lobby. In addition, guest rooms may be equipped with set-top boxes that can play television broadcasts and on-demand videos. By managing these devices with the device management system 1, it becomes possible to automatically set the language set by a hotel guest on the automatic check-in machine as the language of the set-top box.

Further, for example, the device management system 1 can be configured to manage electronic devices installed at an airport. Before boarding an aircraft at the airport, passengers must pass through procedures such as ticketing, check-in, and baggage inspection, and boarding passes are processed using special electronic equipment for each procedure. By managing these devices with the device management system 1, it becomes possible to automatically set the language set in the electronic device performing the previous procedure to the language used in the electronic device performing the next procedure.

Further, for example, the device management system 1 can be configured to manage electronic devices such as search terminals installed in a bookstore. By linking searches performed by customers on search terminals with information about bookshelves that have been moved since then, it can be used to strengthen marketing.

Further, for example, the device management system 1 can be configured to manage electronic devices such as PCs installed in a factory. By correlating and verifying the actions of workers and the operation of equipment installed at the work site, it is possible to monitor the validity of workers' work.

Furthermore, for example, the device management system 1 can be configured to manage electronic devices installed in an unmanned convenience store. Convenience stores are equipped with electronic devices such as MFPs and payment devices. By managing these devices with the device management system 1, it becomes possible, for example, to set the language set by the customer on the MFP as the language on the payment device.

[supplement]
In each of the above embodiments, the management device 10 is an example of an information processing device. The device management system 1 is an example of an information processing system. The monitoring device 20 is an example of a photographing device. The state management unit 13 is an example of a user identification unit. The operation information receiving section 12 is an example of an input information receiving section.

Each function of the embodiments described above can be realized by one or more processing circuits. Here, the term "processing circuit" as used herein refers to a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, or a processor designed to execute each function explained above. This includes devices such as ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), FPGAs (Field Programmable Gate Arrays), and conventional circuit modules.

The devices described in the examples are merely indicative of one of several computing environments for implementing the embodiments disclosed herein. In some embodiments, management apparatus 10 includes multiple computing devices, such as a server cluster. The plurality of computing devices are configured to communicate with each other via any type of communication link, including a network, shared memory, etc., to perform the processes disclosed herein.

Although the embodiments of the present invention have been described in detail above, the present invention is not limited to these embodiments, and various modifications or variations can be made within the scope of the gist of the present invention as described in the claims. Changes are possible.

1 Equipment management system 10 Management device 11 Image reception unit 12 Operation information reception unit 13 Status management unit 14 Operation instruction unit 100 Status storage unit 131 Object status management unit 132 Equipment status management unit 133 Output information determination unit 20 Monitoring device 21 Image acquisition unit 22 Image transmission section 30 Electronic device 31 Operation information transmission section 32 Device control section

Patent No. 6064636 Patent No. 5238409

Claims (12)

An information processing device capable of communicating with a first electronic device, a second electronic device, and a photographing device via a network,
a first image receiving unit that receives a first image taken of a first user using the first electronic device from the photographing device;
a first user identifying unit that identifies the first user who used the first electronic device based on usage information of the first electronic device and the first image;
an input information receiving unit that receives from the first electronic device authentication information that the first user specified by the first user specifying unit inputs on the first electronic device;
a second image receiving unit that receives a second image of a second user present at an operable position of the second electronic device from the photographing device;
a second user identifying unit that identifies the second user who is present at an operable position of the second electronic device from the second image;
Based on the information of the first user identified by the first user identifying unit and the second user identified by the second user identifying unit, and the authentication information, the second electronic device receives the an operation instruction unit that transmits authentication information or screen setting information in the authentication information;
An information processing device comprising: An information processing device capable of communicating with a first electronic device, a second electronic device, and a photographing device via a network,
a first image receiving unit that receives a first image taken of a first user using the first electronic device from the photographing device;
a first user identifying unit that identifies the first user who used the first electronic device based on usage information of the first electronic device and the first image;
an input information receiving unit that receives usage information on the use of the first electronic device by the first user identified by the first user identifying unit from the first electronic device;
a second image receiving unit that receives a second image of a second user present at an operable position of the second electronic device from the photographing device;
a second user identifying unit that identifies the second user who is present at an operable position of the second electronic device from the second image;
Based on the information on the first user identified by the first user identifying unit and the second user identified by the second user identifying unit, and the usage information, the second electronic device an operation instruction unit that transmits usage information;
An information processing device comprising: The information processing device according to claim 1 or 2,
further comprising a state management unit that records position information of the first user and the second user in chronological order using the first image and the second image received from the photographing device;
Information processing device. The information processing device according to claim 3,
The state management unit identifies the first user who operated the first electronic device based on the time series of the location information.
Information processing device. The information processing device according to claim 4,
The state management unit records the position information using the plurality of first images and the second images received from each of the plurality of photographing devices.
Information processing device. The information processing device according to claim 1,
The input information receiving unit further receives status information representing a status of the electronic device,
The operation instruction unit does not transmit the authentication information or the screen setting information to the second electronic device whose status information indicates a dormant state.
Information processing device. An information processing system in which a first electronic device, a second electronic device, a photographing device, and an information processing device can communicate via a network,
The information processing device includes:
a first image receiving unit that receives a first image taken of a first user using the first electronic device from the photographing device;
a first user identifying unit that identifies the first user who used the first electronic device based on usage information of the first electronic device and the first image;
an input information receiving unit that receives from the first electronic device authentication information that the first user specified by the first user specifying unit inputs on the first electronic device;
a second image receiving unit that receives a second image of a second user present at an operable position of the second electronic device from the photographing device;
a second user identifying unit that identifies the second user who is present at an operable position of the second electronic device from the second image;
Based on the information of the first user identified by the first user identifying unit and the second user identified by the second user identifying unit, and the authentication information, the second electronic device receives the an operation instruction unit that transmits authentication information or screen setting information in the authentication information;
An information processing system equipped with. An information processing system in which a first electronic device, a second electronic device, a photographing device, and an information processing device can communicate via a network,
The information processing device includes:
a first image receiving unit that receives a first image taken of a first user using the first electronic device from the photographing device;
a first user identifying unit that identifies the first user who used the first electronic device based on usage information of the first electronic device and the first image;
an input information receiving unit that receives usage information on the use of the first electronic device by the first user identified by the first user identifying unit from the first electronic device;
a second image receiving unit that receives a second image of a second user present at an operable position of the second electronic device from the photographing device;
a second user identifying unit that identifies the second user who is present at an operable position of the second electronic device from the second image;
Based on the information on the first user identified by the first user identifying unit and the second user identified by the second user identifying unit, and the usage information, the second electronic device an operation instruction unit that transmits usage information;
An information processing system equipped with. A computer capable of communicating with the first electronic device, the second electronic device, and the photographing device via a network,
a first image receiving procedure of receiving a first image taken of a first user using the first electronic device from the photographing device;
a first user identification procedure of identifying the first user who used the first electronic device based on usage information of the first electronic device and the first image;
an input information receiving procedure of receiving from the first electronic device authentication information input by the first user identified by the first user identifying procedure using the first electronic device;
a second image receiving procedure of receiving a second image of a second user present at an operable position of the second electronic device from the photographing device;
a second user identification procedure of identifying the second user who is present in an operable position of the second electronic device from the second image;
Based on the information of the first user identified by the first user identification procedure and the second user identified by the second user identification procedure, and the authentication information, the second electronic device receives the an operation instruction procedure for transmitting authentication information or screen setting information in the authentication information;
Equipment management method to perform. A computer capable of communicating with the first electronic device, the second electronic device, and the photographing device via a network,
a first image receiving procedure of receiving a first image taken of a first user using the first electronic device from the photographing device;
a first user identification procedure of identifying the first user who used the first electronic device based on usage information of the first electronic device and the first image;
an input information receiving procedure of receiving usage information on the use of the first electronic device by the first user identified by the first user identifying procedure from the first electronic device;
a second image receiving procedure of receiving a second image of a second user present at an operable position of the second electronic device from the photographing device;
a second user identification procedure of identifying the second user who is present in an operable position of the second electronic device from the second image;
Based on the information on the first user identified by the first user identification procedure and the second user identified by the second user identification procedure, and the usage information, the second electronic device an operating instruction procedure for transmitting usage information;
Equipment management method to perform. a computer capable of communicating with the first electronic device, the second electronic device, and the photographing device via the network;
a first image receiving procedure of receiving a first image taken of a first user using the first electronic device from the photographing device;
a first user identification procedure of identifying the first user who used the first electronic device based on usage information of the first electronic device and the first image;
an input information receiving procedure of receiving from the first electronic device authentication information input by the first user identified by the first user identifying procedure using the first electronic device;
a second image receiving procedure of receiving a second image of a second user present at an operable position of the second electronic device from the photographing device;
a second user identification procedure of identifying the second user who is present in an operable position of the second electronic device from the second image;
Based on the information of the first user identified by the first user identification procedure and the second user identified by the second user identification procedure, and the authentication information, the second electronic device receives the an operation instruction procedure for transmitting authentication information or screen setting information in the authentication information;
A program to run. a computer capable of communicating with the first electronic device, the second electronic device, and the photographing device via the network;
a first image receiving procedure of receiving a first image taken of a first user using the first electronic device from the photographing device;
a first user identification procedure of identifying the first user who used the first electronic device based on usage information of the first electronic device and the first image;
an input information receiving procedure of receiving usage information on the use of the first electronic device by the first user identified by the first user identifying procedure from the first electronic device;
a second image receiving procedure of receiving a second image of a second user present at an operable position of the second electronic device from the photographing device;
a second user identification procedure of identifying the second user who is present in an operable position of the second electronic device from the second image;
Based on the information on the first user identified by the first user identification procedure and the second user identified by the second user identification procedure, and the usage information, the second electronic device an operating instruction procedure for transmitting usage information;
A program to run.

Images