JP7215621B1 - Information processing device, information processing system, device management method and program - Google Patents

Abstract

An information processing apparatus and the like for setting a second electronic device using information used in a first electronic device are provided. A management device 10 capable of communicating with a first electronic device 30-1, a second electronic device 30-2 and a monitoring device 20 via a network uses the first electronic device from the monitoring device. receive a first image of a first user who is using the first electronic device, identify the first user who used the first electronic device based on the usage information of the first electronic device and the first image; A first user receives authentication information input by a first electronic device from the first electronic device, and photographs a second user present at an operable position of the second electronic device from a photographing device. a state management unit that receives the second image and specifies from the second image a second user present at an operable position of the second electronic device; information and authentication information of the first user and the second user; and an operation instructing unit that transmits the authentication information or the screen setting information in the authentication information to the second electronic device based on the above. [Selection drawing] Fig. 6

Description

The present invention relates to an information processing device, an information processing system, a device management method, and a program.

In cloud computing, there is a technology for realizing single sign-on by linking software. For example, Patent Literature 1 discloses an invention in which a user authenticated by one service can omit authentication by another service by using a common authentication infrastructure for a plurality of services.

In authentication by hardware such as electronic equipment, there is a technique for associating authentication processing of each hardware. For example, in Patent Document 2, by linking a system that manages entry/exit to an area where confidential information is handled and an electronic device in the area, an electronic An invention is disclosed that does not allow authentication at the device.

In cooperation between hardware, it is difficult to determine the identity of users. The invention disclosed in Patent Literature 2 utilizes the authentication status of an entry/exit management system in authentication with electronic devices, but each user needs to perform an authentication operation. For example, if it is possible to determine the identity of a user authenticated by the first electronic device and a user near the second electronic device, the authentication information entered by the user on the first electronic device can be used to identify the second electronic device. of electronic devices can perform authentication.

One embodiment of the present invention aims at configuring a second electronic device using information used in the first electronic device.

In order to solve the above problems, an information processing apparatus according to one embodiment of the present invention is an information processing apparatus capable of communicating with a first electronic device, a second electronic device, and a photographing device via a network. a first image receiving unit that receives a first image of a first user using a first electronic device from a photographing device; usage information of the first electronic device and the first image; a first user identification unit that identifies a first user who uses the first electronic device; An input information receiving unit that receives input information from a first electronic device, and a second image receiving unit that receives a second image of a second user present at an operable position of the second electronic device from a photographing device. a second user specifying unit that specifies a second user present at an operable position of the second electronic device from the second image; and the first user and the second user specified by the first user specifying unit. an operation instructing unit that transmits the authentication information or the screen setting information in the authentication information to the second electronic device based on the second user information and the authentication information specified by the specifying unit.

According to one embodiment of the invention, the information used by the first electronic device can be used to configure the second electronic device.

It is a figure showing an example of the whole composition of a device management system in one embodiment. It is a figure which shows an example of the hardware constitutions of the computer in one embodiment. It is a figure which shows an example of the hardware constitutions of the omnidirectional imaging device in one embodiment. 1 is a diagram showing an example of a hardware configuration of an MFP in one embodiment; FIG. It is a figure which shows an example of the hardware constitutions of the electronic blackboard in one Embodiment. It is a figure showing an example of functional composition of a device management system in one embodiment. It is a figure which shows an example of the state management information in one Embodiment. It is a figure which shows an example of the 1st basic flowchart in one embodiment. It is a figure which shows an example of the 2nd basic flowchart in one Embodiment. It is a figure which shows an example of the device management method in one Embodiment. It is a figure which shows an example of the object state update process in one embodiment. It is a figure which shows an example of the detection object list|wrist creation process in one Embodiment. It is a figure which shows an example of the registered object list|wrist update process in one Embodiment. It is a figure which shows an example of the registered object list|wrist update process in one Embodiment. It is a figure which shows the 1st example of the group ID provision process in one Embodiment. It is a figure which shows the 2nd example of the group ID provision process in one Embodiment. It is a figure which shows an example of the apparatus state update process in one Embodiment. It is a figure showing an example of output information decision processing in one embodiment. It is a figure which shows an example of the output condition determination process in one Embodiment. It is a figure which shows an example of the apparatus output information in one Embodiment.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the drawings, constituent parts having the same function are denoted by the same numbers, and redundant explanations are omitted.

[Embodiment]
One embodiment of the present invention is a device management system that manages a plurality of electronic devices. The device management system according to the present embodiment realizes cooperation between electronic devices by instructing other electronic devices to operate using information input by one electronic device.

Conventionally, in cooperation between hardware, the identity of a user has been determined by possessing the same device (for example, an IC (Integrated Circuit) card, a mobile information terminal, or the like). As a result, even if the hardware is linked, the authentication operation is required for each hardware. With single sign-on through software linkage, when a user authenticated by a certain service accesses another service, the user's identity can be easily determined because the operation is performed from the same device.

In the device management system according to the present embodiment, a management device tracks a person who moves between electronic devices based on an image of a space in which a plurality of electronic devices are installed. Thereby, the management device can determine the identity between the user who operated a certain electronic device and the user who is in the vicinity of another electronic device.

If it is possible to determine the identity of a user, it is possible to cause another electronic device to perform an operation using information input by one electronic device. For example, if an electronic device near the user performs authentication using the authentication information that the user has entered on another electronic device, single sign-on can be achieved through hardware cooperation.

<Overall configuration of device management system>
First, the overall configuration of the device management system according to this embodiment will be described with reference to FIG. FIG. 1 is a diagram showing an example of the overall configuration of a device management system according to this embodiment.

As shown in FIG. 1, the device management system 1 in this embodiment includes a management device 10, one or more monitoring devices 20, and a plurality of electronic devices 30. FIG. The management device 10, the monitoring device 20, and the electronic device 30 are each connected to the communication network N1.

The communication network N1 is configured so that connected devices can communicate with each other. The communication network N1 is constructed by a wired communication network such as the Internet, a LAN (Local Area Network), or a WAN (Wide Area Network).

The communication network N1 includes not only wired communication but also wireless LAN, wireless communication such as short-range wireless communication, WiMAX (Worldwide Interoperability for Microwave Access), LTE (Long Term Evolution), or 5G (5th Generation). of mobile communication networks may be included.

The monitoring device 20 and the electronic device 30 are installed in the managed space R1. The managed space R1 may be a single space such as a room in a building, or may be a plurality of spaces connected by an accessible space such as a door or corridor.

An example of the managed space R1 is an office conference room, office, or the like. Another example of the managed space R1 is a sales floor, backyard, or the like in a small store such as a convenience store. Another example of the managed space R1 is a hotel lobby or guest room. Another example of the managed space R1 is an airport, a bookstore, a factory, or the like. The managed space R1 is not limited to these, and may be any space in which electronic devices that can be operated by a plurality of users are installed.

The management device 10 is an information processing device such as a PC (Personal Computer), workstation, or server that manages the electronic device 30 . The management device 10 transmits an operation instruction to the electronic device 30 based on the image acquired by the monitoring device 20 . An example of the management device 10 is a computer.

The monitoring device 20 is an electronic device that acquires an image including the vicinity of the electronic device 30 installed in the managed space R1. Monitoring device 20 may acquire video (ie, a time series of images). The monitoring device 20 is installed at a position where all of the electronic devices 30 can be photographed.

An example of the monitoring device 20 is an omnidirectional imaging device. Another example of monitoring device 20 is a plurality of network cameras. When the monitoring device 20 is a plurality of network cameras, each network camera is arranged with its angle of view adjusted so that there is no blind spot in the managed space R1.

Hereinafter, when there are a plurality of monitoring devices 20, when distinguishing between them, they are described using branch numbers such as "monitoring device 20-1" and "monitoring device 20-2".

The electronic device 30 is various electronic devices used by a user. The electronic device 30 is configured so that only authorized users can use it. The electronic device 30 is configured to allow authentication using hardware such as an IC (Integrated Circuit) card.

An example of the electronic device 30 is an image forming device (printer, FAX device, MFP (Multifunction Peripheral/Product/Printer), scanner device, etc.). Another example of the electronic device 30 is an electronic whiteboard (IWB (Interactive White Board: a whiteboard having an electronic blackboard function capable of mutual communication)).

In the following, when distinguishing among the plurality of electronic devices 30, they are described using serial numbers such as “electronic device 30-1” and “electronic device 30-2”.

The electronic device 30 is not limited to an image forming device or an electronic blackboard as long as it is a device having a communication function. That is, the electronic device 30 includes, for example, a PJ (Projector), an output device such as a digital signage, a HUD (Head Up Display) device, an industrial machine, an imaging device, a sound collector, a medical device, a network appliance, an automobile (Connected Car), notebook PC (Personal Computer), mobile phone, smart phone, tablet terminal, game machine, PDA (Personal Digital Assistant), digital camera, wearable PC, desktop PC, or the like.

<Hardware configuration of device management system>
Next, the hardware configuration of each device included in the device management system according to this embodiment will be described with reference to FIGS. 2 to 5. FIG.

<<Computer hardware configuration>>
FIG. 2 is a diagram showing an example of a hardware configuration when the management device 10 is realized by a computer.

As shown in FIG. 2, the computer in one embodiment includes a CPU (Central Processing Unit) 501, ROM (Read Only Memory) 502, RAM (Random Access Memory) 503, HD (Hard Disk) 504, HDD ( Hard Disk Drive) controller 505, display 506, external device connection I / F (Interface) 508, network I / F 509, bus line 510, keyboard 511, pointing device 512, DVD-RW (Digital Versatile Disk Rewritable) drive 514, media It has I/F516.

Among these, the CPU 501 controls the operation of the entire computer. The ROM 502 stores programs used to drive the CPU 501, such as an IPL (Initial Program Loader). A RAM 503 is used as a work area for the CPU 501 . The HD 504 stores various data such as programs. The HDD controller 505 controls reading or writing of various data to/from the HD 504 under the control of the CPU 501 .

A display 506 displays various information such as cursors, menus, windows, characters, or images. The external device connection I/F 508 is an interface for connecting various external devices. The external device in this case is, for example, a USB (Universal Serial Bus) memory, a printer, or the like. A network I/F 509 is an interface for data communication using the communication network N1. A bus line 510 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 501 shown in FIG.

Also, the keyboard 511 is a kind of input means having a plurality of keys for inputting characters, numerical values, various instructions, and the like. The pointing device 512 is a kind of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, and the like. A DVD-RW drive 514 controls reading or writing of various data to a DVD-RW 513 as an example of a removable recording medium. It should be noted that not only DVD-RW but also DVD-R or the like may be used. A media I/F 516 controls reading or writing (storage) of data to a recording medium 515 such as a flash memory.

≪Hardware configuration of omnidirectional camera≫
FIG. 3 is a diagram showing an example of a hardware configuration when the monitoring device 20 is realized by an omnidirectional imaging device. In the following, the omnidirectional imaging device is assumed to be an omnidirectional (omnidirectional) imaging device using two imaging elements, but the number of imaging elements may be two or more. In addition, it does not necessarily have to be a device dedicated to omnidirectional photography. By attaching an omnidirectional imaging unit to a normal digital camera, smartphone, etc., it can have substantially the same functions as an omnidirectional imaging device. may

As shown in FIG. 3, the omnidirectional imaging device in one embodiment includes an imaging unit 601, an image processing unit 604, an imaging control unit 605, a microphone 608, a sound processing unit 609, and a CPU (Central Processing Unit) 611. , ROM (Read Only Memory) 612, SRAM (Static Random Access Memory) 613, DRAM (Dynamic Random Access Memory) 614, operation unit 615, external device connection I/F 616, communication unit 617, antenna 617a, acceleration/direction sensor 618 , a gyro sensor 619, an acceleration sensor 620, and a recessed terminal 621 for Micro USB.

Among them, the imaging unit 601 includes wide-angle lenses (so-called fish-eye lenses) 602a and 602b each having an angle of view of 180° or more for forming a hemispherical image, and two imaging units provided corresponding to each wide-angle lens. It has elements 603a and 603b. The imaging elements 603a and 603b are image sensors such as CMOS (Complementary Metal Oxide Semiconductor) sensors and CCD (Charge Coupled Device) sensors that convert optical images by the fisheye lenses 602a and 602b into image data of electric signals and output them. and a timing generation circuit for generating horizontal or vertical synchronizing signals, pixel clocks, etc., and a group of registers for setting various commands and parameters necessary for the operation of this imaging device.

The imaging elements 603a and 603b of the imaging unit 601 are each connected to the image processing unit 604 via a parallel I/F bus. On the other hand, the imaging elements 603a and 603b of the imaging unit 601 are connected to the imaging control unit 605 via a serial I/F bus (such as an I2C bus). The image processing unit 604 , imaging control unit 605 and sound processing unit 609 are connected to a CPU 611 via a bus 610 . Furthermore, the bus 610 is also connected with a ROM 612, an SRAM 613, a DRAM 614, an operation unit 615, an external device connection I/F (Interface) 616, a communication unit 617, an acceleration/direction sensor 618, and the like.

The image processing unit 604 fetches the image data output from the image sensors 603a and 603b through the parallel I/F bus, performs predetermined processing on each image data, and synthesizes these image data. , to create the equirectangular projection image data.

The imaging control unit 605 generally uses the I2C bus with the imaging control unit 605 as a master device and the imaging elements 603a and 603b as slave devices to set commands and the like in registers of the imaging elements 603a and 603b. Necessary commands and the like are received from the CPU 611 . The imaging control unit 605 also uses the I2C bus to capture status data and the like of the registers of the imaging elements 603 a and 603 b and sends them to the CPU 611 .

Also, the imaging control unit 605 instructs the imaging devices 603a and 603b to output image data at the timing when the shutter button of the operation unit 615 is pressed. Some omnidirectional imaging devices have a preview display function and a function corresponding to moving image display on a display (for example, a smartphone display). In this case, the image data is output continuously from the imaging elements 603a and 603b at a predetermined frame rate (frames/minute).

In addition, as will be described later, the imaging control unit 605 also functions as synchronization control means for synchronizing the output timing of the image data of the imaging elements 603a and 603b in cooperation with the CPU 611. FIG. In this embodiment, the omnidirectional imaging apparatus is not provided with a display, but may be provided with a display unit.

A microphone 608 converts sound into sound (signal) data. The sound processing unit 609 takes in sound data output from the microphone 608 through the I/F bus and performs predetermined processing on the sound data.

A CPU 611 controls the overall operation of the omnidirectional imaging apparatus and executes necessary processing. ROM 612 stores various programs for CPU 611 . An SRAM 613 and a DRAM 614 are work memories, and store programs to be executed by the CPU 611, data during processing, and the like. In particular, the DRAM 614 stores image data being processed by the image processing unit 604 and data of the processed equirectangular projection image.

The operation unit 615 is a general term for operation buttons such as the shutter button 615a. By operating the operation unit 615, the user inputs various shooting modes, shooting conditions, and the like.

The external device connection I/F 616 is an interface for connecting various external devices. The external device in this case is, for example, a USB (Universal Serial Bus) memory, a PC (Personal Computer), or the like. The data of the equirectangular projection image stored in the DRAM 614 is recorded on an external medium via the external device connection I/F 616, or is transmitted to an external device such as a smartphone via the external device connection I/F 616 as necessary. It is sent to a terminal (device).

The communication unit 617 communicates with an external terminal such as a smart phone by short-range wireless communication technology such as Wi-Fi, NFC (Near Field Communication), or Bluetooth (registered trademark) via an antenna 617a provided in the omnidirectional imaging device. Communicate with (device). The communication unit 617 can also transmit the data of the equirectangular projection image to an external terminal (device) such as a smartphone.

An acceleration/azimuth sensor 618 calculates the azimuth of the omnidirectional imaging apparatus from the magnetism of the earth and outputs azimuth information. This azimuth information is an example of related information (metadata) according to Exif, and is used for image processing such as image correction of a captured image. The related information also includes data such as the date and time when the image was taken and the data volume of the image data. The acceleration/azimuth sensor 618 is a sensor that detects changes in angles (Roll angle, Pitch angle, Yaw angle) accompanying movement of the omnidirectional imaging device 6 . A change in angle is an example of related information (metadata) according to Exif, and is used for image processing such as image correction of a captured image.

Furthermore, the acceleration/direction sensor 618 is a sensor that detects acceleration in three axial directions. Based on the acceleration detected by the acceleration/direction sensor 618, the omnidirectional imaging device calculates the attitude (angle with respect to the direction of gravity) of its own device (omnidirectional imaging device). By providing the omnidirectional imaging device with the acceleration/azimuth sensor 618, the accuracy of image correction is improved.

<<Hardware Configuration of MFP>>
FIG. 4 is a diagram showing an example of a hardware configuration when the electronic device 30 is realized by an MFP.

As shown in FIG. 4, the MFP in one embodiment includes a controller 910, a short-range communication circuit 920, an engine control section 930, an operation panel 940, and a network I/F (Interface) 950.

Among them, the controller 910 includes a CPU (Central Processing Unit) 901, a system memory (MEM-P) 902, a north bridge (NB) 903, a south bridge (SB) 904, an ASIC (Application Specific Integrated Circuit) 906, a local memory (MEM-C) 907 that is a storage unit, an HDD (Hard Disk Drive) controller 908, and an HD (Hard Disk) 909 that is a storage unit. (Accelerated Graphics Port) It is configured to be connected by a bus 921 .

Among these, the CPU 901 is a control unit that performs overall control of the MFP. The NB 903 is a bridge for connecting the CPU 901, the MEM-P 902, the SB 904, and the AGP bus 921, and is a memory controller that controls reading and writing with respect to the MEM-P 902, a PCI (Peripheral Component Interconnect) master, and an AGP target. have

The MEM-P 902 includes a ROM (Read Only Memory) 902a, which is a memory for storing programs and data that implement each function of the controller 910, and a RAM (Random Access Memory) 902b. The program stored in the RAM 902b is configured to be provided by being recorded in a computer-readable recording medium such as a CD-ROM, CD-R, DVD, etc. as a file in an installable format or an executable format. You may

The SB 904 is a bridge for connecting the NB 903 with PCI devices and peripheral devices. The ASIC 906 is an image processing IC (Integrated Circuit) having hardware elements for image processing, and serves as a bridge that connects the AGP bus 921, PCI bus 922, HDD 908 and MEM-C 907, respectively. This ASIC 906 includes a PCI target and AGP master, an arbiter (ARB) that forms the core of the ASIC 906, a memory controller that controls the MEM-C 907, and multiple DMACs (Direct Memory Access Controllers) that perform image data rotation, etc. by hardware logic. , and a PCI unit that transfers data between the scanner unit 931 and the printer unit 932 via the PCI bus 922 . Note that the ASIC 906 may be connected to a USB (Universal Serial Bus) interface or an IEEE 1394 (Institute of Electrical and Electronics Engineers 1394) interface.

MEM-C 907 is a local memory used as an image buffer for copying and an encoding buffer. The HD 909 is a storage for accumulating image data, accumulating font data used for printing, and accumulating forms. The HD 909 controls reading or writing of data to or from the HD 909 under the control of the CPU 901 . The AGP bus 921 is a bus interface for graphics accelerator cards proposed to speed up graphics processing, and can speed up the graphics accelerator card by directly accessing the MEM-P 902 with high throughput. .

The near field communication circuit 920 also includes a near field communication circuit 920a. The short-range communication circuit 920 is a communication circuit for NFC, Bluetooth, or the like.

Furthermore, the engine control section 930 is configured by a scanner section 931 and a printer section 932 . The operation panel 940 displays a current setting value, a selection screen, and the like, and a panel display unit 940a such as a touch panel for receiving input from an operator, and setting values for image forming conditions such as density setting conditions. An operation panel 940b is provided which includes a numeric keypad for accepting a copy start instruction, a start key for accepting a copy start instruction, and the like. A controller 910 controls the entire MFP, such as drawing, communication, and input from an operation panel 940 . The scanner unit 931 or printer unit 932 includes an image processing part such as error diffusion and gamma conversion.

The MFP can switch and select the document box function, the copy function, the printer function, and the facsimile function in sequence using the application switching key on the operation panel 940 . The document box mode is set when the document box function is selected, the copy mode is set when the copy function is selected, the printer mode is set when the printer function is selected, and the facsimile mode is set when the facsimile mode is selected.

A network I/F 950 is an interface for data communication using the communication network N1. A short-range communication circuit 920 and a network I/F 950 are electrically connected to the ASIC 906 via a PCI bus 922 .

≪Hardware configuration of the electronic blackboard≫
FIG. 5 is a diagram showing an example of a hardware configuration when the electronic device 30 is realized by an electronic blackboard.

As shown in FIG. 5, the electronic blackboard in one embodiment includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, an SSD (Solid State Drive) 204, A network I/F (Interface) 205 and an external device connection I/F 206 are provided.

Among these, the CPU 201 controls the operation of the entire electronic blackboard. The ROM 202 stores programs used to drive the CPU 201, such as the CPU 201 and an IPL (Initial Program Loader). A RAM 203 is used as a work area for the CPU 201 . The SSD 204 stores various data such as electronic blackboard programs. A network controller 205 controls communication with the communication network N1. The external device connection I/F 206 is an interface for connecting various external devices. The external devices in this case are, for example, a USB (Universal Serial Bus) memory 230 and external devices (microphone 240, speaker 250, camera 260).

In addition, the electronic blackboard includes a capture device 211, a GPU 212, a display controller 213, a contact sensor 214, a sensor controller 215, an electronic pen controller 216, a short-range communication circuit 219, an antenna 219a of the short-range communication circuit 219, a power switch 222 and a selector. Switches 223 are provided.

Among these devices, the capture device 211 causes the display of an external PC (Personal Computer) 270 to display video information as a still image or moving image. A GPU (Graphics Processing Unit) 212 is a semiconductor chip that specializes in graphics. The display controller 213 controls and manages screen display in order to output an output image from the GPU 212 to the display 280 or the like.

The contact sensor 214 detects that the electronic pen 290 , the user's hand H, or the like touches the display 280 . A sensor controller 215 controls the processing of the contact sensor 214 . The contact sensor 214 performs coordinate input and coordinate detection using an infrared shielding method. In this coordinate input and coordinate detection method, two light emitting/receiving devices installed at both ends of the upper side of the display 280 radiate a plurality of infrared rays parallel to the display 280, and are provided around the display 280. This is a method of receiving light that is reflected by a reflecting member and returns along the same optical path as that of light emitted by a light receiving element. The contact sensor 214 outputs to the sensor controller 215 the ID of the infrared rays emitted by the two light emitting/receiving devices blocked by the object, and the sensor controller 215 identifies the coordinate position, which is the contact position of the object.

The electronic pen controller 216 communicates with the electronic pen 290 to determine whether or not the display 280 has been touched with the tip of the pen or the bottom of the pen. The short-range communication circuit 219 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The power switch 222 is a switch for switching ON/OFF of the power supply of the electronic blackboard. The selection switches 223 are, for example, a group of switches for adjusting the brightness and color of the display on the display 280 .

Furthermore, the electronic blackboard has a bus line 210 . A bus line 210 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 201 shown in FIG.

In addition, the contact sensor 214 is not limited to the infrared blocking type, but is a capacitive type touch panel that identifies the contact position by detecting a change in capacitance, or identifies the contact position by a voltage change of two opposing resistive films. Various detection means may be used, such as a resistive touch panel that detects an electromagnetic induction generated when a contact object touches the display unit and identifies the contact position. Further, the electronic pen controller 216 may determine whether or not the part of the electronic pen 290 gripped by the user or other parts of the electronic pen is touched, in addition to the tip and the pen butt of the electronic pen 290 .

<Functional configuration of device management system>
Next, an example of the functional configuration of the device management system according to this embodiment will be described with reference to FIG. FIG. 6 is a block diagram illustrating the functional configuration of each device included in the device management system according to this embodiment.

<<Functional configuration of the management device>>
As shown in FIG. 6 , the management device 10 in this embodiment includes an image reception section 11 , an operation information reception section 12 , a state management section 13 , an operation instruction section 14 and a state storage section 100 . The state management unit 13 in this embodiment includes an object state management unit 131 , a device state management unit 132 and an output information determination unit 133 .

The image receiving unit 11, the operation information receiving unit 12, the state management unit 13, and the operation instructing unit 14 perform, for example, the processing that the CPU 501 and the HDD controller 505 execute the program expanded from the HD 504 shown in FIG. 2 onto the RAM 503. realized by

The state storage unit 100 is implemented using, for example, the HD 504 shown in FIG. Reading or writing of data stored in the HD 504 is performed via the HDD controller 505, for example.

The state storage unit 100 stores state management information for managing the states of devices and objects existing in the managed space R1. The state management information in this embodiment includes a registered object list, a registered device list and a detected object list.

The registered object list is a list for managing information on objects existing in the managed space R1. The object in this embodiment is a person.

The registered device list is a list for managing information on devices existing in the managed space R1. The device in this embodiment is the electronic device 30 .

The detected object list is a list for managing objects detected in the managed space R1. The detected object list is a temporary list used to update the registered object list.

Here, the state management information in this embodiment will be described with reference to FIG. FIG. 7A is a conceptual diagram showing an example of a registered object list.

As shown in FIG. 7A, the registered object list in this embodiment has registered object IDs, group IDs, existence confirmation flags, attribute information, and device operation information as data items.

The registered object ID is identification information for identifying an object included in the registered object list.

A group ID is identification information for identifying a group to which a plurality of registered objects belong. A group ID is assigned when it is determined that some of the registered objects included in the registered object list belong to the same group. An initial value (for example, zero) is set as the group ID of a registered object that has not been determined to belong to a group.

A group is, for example, colleagues, parents and children, or friends. Members belonging to the group jointly perform jobs such as meeting or shopping. Therefore, by recognizing the group, it is expected that the behavior of the person can be grasped more accurately. For example, when a plurality of employees hold a meeting in a conference room in which an MFP and an electronic blackboard are installed, it would be convenient if one employee authenticates with the MFP so that authentication can be omitted when other employees use the electronic blackboard. expensive.

The existence confirmation flag is a flag indicating whether or not the registered object exists in the managed space R1. For example, the existence confirmation flag is set to 1 if it exists and 0 if it does not exist.

Attribute information is information associated with a registered object and is information necessary for management. In the present embodiment, it is the position of the registered object and the time when the position is confirmed (hereinafter also referred to as "image acquisition time"). The position of a registered object is represented by three-dimensional coordinates (that is, numerical values on each axis in an orthogonal coordinate system such as the X-axis, Y-axis, and Z-axis).

The method of acquiring position information differs depending on the type of monitoring device 20 . For example, if the monitoring device 20 is a network camera, a well-known technique can be used such as acquiring the distance to the object with a stereo camera and mapping it together with the direction to three-dimensional coordinates.

In addition to the position of the registered object, the attribute information may include various types of information indicating features of the registered object. For example, the color or shape of the registered object can be used. If the object is a person, it is preferable to use color because the shape changes depending on the pose. Since the number of types of colors or the area of each color also changes depending on the posture, it is preferable to use the condition of whether or not at least one color is continuous when determining whether or not the objects are the same.

The device operation information is information related to the device operation performed by the registered object. The device operation information may include input information input by the user during the device operation. The device operation information includes, for example, a function executed by the user on the electronic device, operation time, charge information as consideration for the operation, language used, and the like. The input information is authentication information in the authentication operation, setting information in the device setting operation, or the like.

The number of objects existing in the managed space R1 changes from moment to moment. Therefore, the number of registered objects included in the registered object list is variable.

FIG. 7B is a conceptual diagram showing an example of the registered device list. As shown in FIG. 7B, the registered device list in this embodiment has registered device IDs, attribute information, device input information, and device output information as data items.

The registered device ID is identification information for identifying devices included in the registered device list.

The attribute information is information associated with the registered device and necessary for management. In this embodiment, the information includes the position of the registered device, power on/off status, sleep mode status, whether authentication is required, option installation status, status information such as under maintenance/scheduled maintenance time/in energy saving mode. The position of the registered device is represented by three-dimensional coordinates in the same way as the registered object.

Device input information (hereinafter also referred to as “usage information”) is predetermined information managed when a registered device is operated. The device input information may include input information entered by the user during the device operation. The device input information includes, for example, the function executed by the user on the registered device, operation time, fee information to be charged as consideration for the operation, and the like. The device input information may include screen setting information set by the user in the device operation. The screen setting information includes, for example, language to be used, font of characters to be used, size of characters to be used, and the like.

The device output information is information registered in advance with respect to the registered device. The device output information is a list that associates an output condition that triggers transmission of an operation instruction and an operation instruction that is transmitted to the registered device when the output condition is satisfied.

The number of devices installed in the managed space R1 may change. Therefore, the number of registered devices included in the registered device list is variable.

FIG. 7C is a conceptual diagram showing an example of the detected object list. As shown in FIG. 7C, the detected object list in this embodiment has detected object IDs, group IDs, and attribute information as data items.

The detected object ID is identification information for identifying an object included in the detected object list.

A group ID is identification information that identifies a group that includes a plurality of sensed objects. A group ID is assigned when some of the detected objects included in the detected object list are determined to belong to the same group. An initial value (for example, zero) is set as the group ID of a detected object that has not been determined to belong to a group.

Attribute information is information associated with a detected object and is information necessary for management. In this embodiment, it is the position of the detected object and the image acquisition time. The position of the detected object is represented by three-dimensional coordinates in the same way as the registered object.

The number of objects detected in the managed space R1 changes from moment to moment. Therefore, the number of detected objects included in the detected object list is variable.

Returning to FIG. 6, description will be made. The image receiving unit 11 receives images from the monitoring device 20 . The image receiving section 11 sends the received image to the state management section 13 in response to a request from the state management section 13 .

The operation information receiving unit 12 receives device operation information from the electronic device 30 . The operation information receiving unit 12 sends the received device operation information to the state management unit 13 in response to a request from the state management unit 13 .

The state management unit 13 uses the object state management unit 131 and the device state management unit 132 to manage the states of the electronic device 30 and the object. The state management unit 13 also uses the output information determination unit 133 to determine the content of the operation instruction to be transmitted to the electronic device 30 .

The object state management section 131 updates the registered object list stored in the state storage section 100 based on the image received from the image reception section 11 .

The device state management section 132 updates the registered device list and the registered object list stored in the state storage section 100 based on the device operation information received from the operation information reception section 12 .

The output information determination unit 133 determines the content of the operation instruction to be transmitted to the electronic device 30 based on the state management information stored in the state storage unit 100 . The output information determining section 133 sends the determined action instruction to the action instructing section 14 .

The operation instruction unit 14 transmits the operation instruction received from the state management unit 13 to the electronic device 30 .

≪Functional configuration of the monitoring device≫
As shown in FIG. 6 , the monitoring device 20 in this embodiment includes an image acquisition section 21 and an image transmission section 22 .

The image acquisition unit 21 acquires an image including the vicinity of the electronic device 30 installed in the managed space R1. The image acquisition unit 21 is implemented by, for example, processing that causes the CPU 611 and the imaging control unit 605 to execute a program developed on the SRAM 613 from the ROM 612 shown in FIG.

The image transmission unit 22 transmits the image acquired by the image acquisition unit 21 to the management device 10 . The image transmission unit 22 is implemented, for example, by a process that causes the CPU 611 and the external device connection I/F 616 to execute a program developed on the SRAM 613 from the ROM 612 shown in FIG.

<<Functional configuration of electronic equipment>>
As shown in FIG. 6 , the electronic device 30 in this embodiment includes an operation information transmission section 31 and a device control section 32 .

The operation information transmission unit 31 and the device control unit 32 are realized by, for example, processing that causes the CPU 901 and the HDD controller 908 to execute a program developed on the RAM 902b from the HD 909 shown in FIG.

The operation information transmission unit 31 transmits to the management device 10 device operation information related to the operation performed by the user.

The device control unit 32 receives operation instructions from the management device 10 . The device control section 32 controls the operation of the electronic device 30 based on the received operation instruction. For example, the device control unit 32 authenticates the user using authentication information included in the operation instruction. Also, for example, the device control unit 32 changes the settings of the electronic device 30 using the setting information included in the operation instruction.

<Processing procedure of device management method>
Next, the device management method executed by the device management system in this embodiment will be described with reference to FIGS. 8 to 20. FIG.

≪Basic flow chart≫
The device management method of this embodiment includes many double loop processes for processing all combinations of data included in a list and data included in the list. Therefore, in this embodiment, a basic flowchart showing the framework of double loop processing is introduced, and the following description focuses on the processing in the basic flowchart. Each process in the basic flow chart is a process performed on one combination of two data.

The basic flowchart in this embodiment includes two basic flowcharts. The first basic flow chart is used when double looping two data contained in different lists. The second basic flow chart is used when double looping two data contained in the same list.

FIG. 8 is a flow chart showing an example of the first basic flow chart in this embodiment.

In step S101, list A is read. Assume that the number of data in list A is N. In step S102, list B is read. Assume that the number of data in the list B is M.

In step S103, a variable n is initialized to 1. In step S104, a variable m is initialized to 1. In step S105, processing A1 is executed.

In step S106, it is determined whether or not the variable m is equal to the number M of data. This means whether or not the combination of the n-th data of list A with all the data of list B has been processed. If the variable m is different from the number of data M (NO), the process proceeds to step S107. If the variable m is equal to the number of data M (YES), the process proceeds to step S109.

In step S107, process A3 is executed. The variable m is incremented in step S108. After that, the process returns to step S106.

In step S109, process A2 is executed. In step S110, it is determined whether or not the variable n is equal to the number N of data. This means whether or not all data in list A have been processed in combination with all data in list B. If the variable n is different from the number of data N (NO), the process proceeds to step S111. If the variable n is equal to the number of data N (YES), the process proceeds to step S113.

In step S111, process A5 is executed. In step S112, the variable n is incremented. After that, the process returns to step S104.

In step S113, process A4 is executed. As described above, the processing for all combinations of data included in list A and list B is completed.

FIG. 9 is a flow chart showing an example of the second basic flow chart in this embodiment.

In step S201, list A is read. Let N be the number of data in list A.

In step S202, a variable n is initialized to 1. In step S203, n+1 is substituted for variable m. In step S204, process B1 is executed.

In step S205, it is determined whether or not the variable m is equal to N. This means whether or not the n-th data in list A has been combined with all the data in list A. If the variable m is different from N (NO), the process proceeds to step S206. If the variable m is equal to N (YES), the process proceeds to step S208.

In step S206, process B3 is executed. At step S207, the variable m is incremented. After that, the process returns to step S204.

In step S208, process B2 is executed. In step S209, it is determined whether or not the variable n is equal to N-1. This means that for every data in list A, the combination with every other data in list A has been processed. If the variable n is different from N-1 (NO), the process proceeds to step S210. If the variable n is equal to N-1 (YES), the process proceeds to step S212.

In step S210, process B5 is executed. In step S211, the variable n is incremented. After that, the process returns to step S203.

In step S212, process B4 is executed. As described above, the processing for all combinations of data included in the list A is completed.

≪Device management method≫
FIG. 10 is a flow chart showing an example of a device management method executed by the device management system according to this embodiment.

The equipment management system repeatedly executes the flowchart shown in FIG. 10 at predetermined time intervals. The time interval may be set arbitrarily, but is, for example, 3 seconds.

In step S1, the image acquiring unit 21 included in the monitoring device 20 acquires a first image of the vicinity of the electronic device 30-1. Also, the image acquisition unit 21 acquires a second image of the vicinity of the electronic device 30-2. The monitoring device 20 is installed so that the vicinity of the electronic device 30-1 and the electronic device 30-2 installed in the managed space R1 is included in the angle of view. Therefore, in the first image acquired by the image acquisition unit 21, the vicinity of the electronic device 30-1 is photographed. Also, the second image acquired by the image acquisition unit 21 captures the vicinity of the electronic device 30-2.

The vicinity of the electronic device 30-1 or the electronic device 30-2 is a range of positions where the user can operate the electronic device 30-1 or the electronic device 30-2 (hereinafter also referred to as “operable position”). . The operable position is a position where the user can reach the electronic device 30-1 or the electronic device 30-2. The operable position is, for example, a position within 1 meter from the electronic device 30-1 or the electronic device 30-2. Therefore, the first image is an image of the electronic device 30-1 and the user using the electronic device 30-1. A second image is an image of the electronic device 30-2 and the user using the electronic device 30-2.

If the operable positions of the electronic devices 30-1 and 30-2 can be captured in one image, the electronic devices 30-1 and 30-2 can be operated in one image. You can take a picture of the position. In this case, the first image and the second image are the same image.

The first image and the second image acquired by the image acquisition unit 21 are not limited to still images, and may be moving images.

Next, the image acquiring section 21 sends the acquired first image and second image to the image transmitting section 22 . The image transmission section 22 receives the first image and the second image from the image acquisition section 21 . Next, the image transmission unit 22 transmits the received first image and second image to the management device 10 . In the management device 10 , the image receiving section 11 receives the first image and the second image from the monitoring device 20 .

In step S2, the state management unit 13 included in the management device 10 requests the image reception unit 11 for the first image and the second image. Next, the state management unit 13 inputs the first image and the second image received from the image reception unit 11 to the object state management unit 131 .

Subsequently, the object state management unit 131 executes object state update processing, which will be described later, and updates the registered object list of the state management information stored in the state storage unit 100 . When there are multiple monitoring devices 20 , the object state management unit 131 executes object state update processing for each of the multiple images received from each of the multiple monitoring devices 20 .

≪Object state update processing≫
Here, object state update processing in the present embodiment will be described with reference to FIGS. 11 to 16. FIG. FIG. 11 is a flowchart showing an example of object state update processing (step S2 in FIG. 10) executed by the object state management unit 131 in this embodiment.

In step S<b>21 , the object state management unit 131 executes detection object list creation processing, which will be described later, and creates a detection object list based on the image input from the state management unit 13 .

FIG. 12 is a flow chart showing an example of the detected object list creation process (step S21 in FIG. 11) executed by the object state management unit 131 in this embodiment.

In step S21-1, the object state management unit 131 clears the detected object list stored in the state storage unit 100. FIG. That is, all data included in the detected object list are deleted.

In step S21-2, the object state management section 131 acquires the image input from the state management section 13. FIG. The image may be one image received from one monitoring device 20 or a plurality of images received from a plurality of monitoring devices 20 .

Next, the object state management unit 131 divides the image into predetermined blocks. Subsequently, the object state management unit 131 measures the distance between the object captured in the image and the camera for each block. Also, the object state management unit 131 obtains three-dimensional coordinates from the measured distance. A known method may be used as a method of converting the distance into three-dimensional coordinates.

In step S21-3, the object state management unit 131 acquires the image acquisition time T from the acquired image. If the image acquisition time cannot be obtained from the image, the current time may be obtained as the image acquisition time T. FIG.

In step S21-4, the object state management unit 131 analyzes the acquired image and detects the object captured in the image. In this embodiment, a person photographed in an image is detected.

A known method can be used to detect a person. For example, pattern matching using machine learning models can be used. Specifically, a person can be extracted from an image using a known Region Based Convolutional Neural Network (R-CNN) or the like.

Furthermore, the accuracy of detection can be improved by performing pattern matching on the clipped image of the person with a pre-stored comparative image. In addition to these, various well-known person detection methods can be used.

In step S21-5, the object state management unit 131 assigns a detected object ID for identifying the detected object.

In step S21-6, the object state management unit 131 registers the detected object ID and attribute information (position and image acquisition time of the detected object) in the detected object list. At this stage, the group ID is set to an initial value indicating that it does not belong to any group.

Returning to FIG. 11, description will be made. In step S22, the object state management unit 131 executes registered object list update processing, which will be described later, and updates the registered object list based on the detected object list.

13 and 14 are flowcharts showing an example of the registered object list update process (step S22 in FIG. 11) executed by the object state management unit 131 in this embodiment.

The registered object list update process is performed within the framework of the first basic flow chart (see FIG. 8). Specifically, the detected object list is set to list A, the registered object list is set to list B, and processing A1 (FIG. 13) and processing A4 (FIG. 14) are executed for all combinations of detected objects and registered objects. There is no process corresponding to the processes A2, A3 and A5.

FIG. 13 is a flowchart showing an example of processing A1 of the registered object list update processing. Processing A1 is processing for determining the identity between a registered object and a detected object and updating the position of the registered object.

Here, it is assumed that processing is executed for a combination of the n-th detected object in the detected object list and the m-th registered object in the registered object list.

In step S22-1, the object state management unit 131 clears the existence confirmation flag of the m-th registered object. That is, 0 is set to the existence confirmation flag of the m-th registered object. This means that it is unknown whether or not the registered object exists in the managed space R1.

In step S22-2, the object state management unit 131 calculates the distance X between the three-dimensional coordinates of the m-th registered object and the three-dimensional coordinates of the n-th detected object.

In step S22-3, the object state management unit 131 determines whether or not the distance X is less than or equal to a predetermined threshold value (eg, 1 meter). If the distance X exceeds the threshold (NO), the object state management unit 131 terminates the processing A1. This means that it is determined that they are not the same person because they are far apart. If the distance X is equal to or less than the threshold (YES), the object state management section 131 advances the process to step S22-4.

In step S22-4, the object state management unit 131 adds the three-dimensional position and video acquisition time of the n-th detected object to the attribute information of the m-th registered object. This means that it is determined that they are the same person and that the position of the n-th detected object is the position at the image acquisition time T of the m-th registered object because the distance between the two is close.

The attribute information of the registered object list increases each time the object state update process is executed as long as the registered object is detected from the image. Conversely, the attribute information of the registered object list represents the time series of positions at which the registered object is detected within the managed space. Therefore, it is possible to trace the movement of the registered object using the attribute information of the registered object list.

In step S22-5, the object state management unit 131 sets the presence confirmation flag of the m-th registered object. That is, 1 is set to the existence confirmation flag of the m-th registered object. This means that since an object matching the registered object was detected in step S22-3, it was determined that the registered object existed in the managed space R1 at the image acquisition time T as well.

FIG. 14 is a flowchart showing an example of processing A4 of the registered object list update processing. A process A4 is a process of deleting a registered object whose existence could not be confirmed in the process A1 from the registered object list.

In step S22-6, the object state management unit 131 initializes the variable m to 1.

In step S22-7, the object state management unit 131 determines whether or not the existence of the m-th registered object has been confirmed. Specifically, whether the presence confirmation flag is 1 or 0 is determined. If the existence confirmation flag is 1 (YES), the object state management unit 131 advances the process to step S22-9. If the existence confirmation flag is 0 (NO), the object state management unit 131 advances the process to step S22-8.

In step S22-8, the object state management unit 131 deletes the m-th registered object from the registered object list.

In step S22-9, the object state management unit 131 determines whether or not the variable m is equal to the number M of data. If the variable m is different from the number of data M (NO), the object state management unit 131 advances the process to step S22-10. If the variable m is equal to the number of data M (YES), the object state management unit 131 terminates the process.

At step S22-10, the object state manager 131 increments the variable m. After that, the object state management unit 131 returns the process to step S22-7.

Returning to FIG. 11, description will be made. In step S23, the object state management unit 131 executes group ID assignment processing, which will be described later, and assigns group IDs to registered objects included in the registered object list.

The group ID assigning process is performed within the framework of the second basic flow chart (see FIG. 9). Specifically, the registered object list is set to list A, and processing B1 (FIG. 15 or 16) is executed for all combinations of two registered objects. Note that there is no process corresponding to the processes B2-B5.

Various methods are conceivable for the process of determining whether a certain object and another object are included in the same group. In this embodiment, group determination based on positional proximity and group determination based on group actions will be described. However, the method of group determination is not limited to these, and any technique may be used as long as it enables group determination of objects from an image.

FIG. 15 is a flow chart showing a first example of the group ID assigning process (step S23 in FIG. 11) executed by the object state management unit 131 in this embodiment. A first example of group ID assignment processing is group determination based on positional proximity.

Group determination based on positional proximity is a process of registering as a group two objects that have been detected at positions that have been close to each other for a predetermined number of times in a row. The predetermined number of times is, for example, five times. Assuming that the time interval for executing the device management method is 3 seconds, two objects detected at close positions continuously for 15 seconds are determined to belong to the same group.

Here, it is assumed that the processing is executed for the combination of the m-th registered object and the n-th registered object in the registered object list.

In step S23A-1, the object state management unit 131 initializes variables k and j to one. Variable k is a counter representing the number of times it is determined whether two objects are close to each other. Variable j is a counter representing the number of times two objects are determined to be close.

In step S23A-2, the object state management unit 131 calculates the distance X between the three-dimensional coordinates of the m-th registered object k times before and the three-dimensional coordinates of the n-th registered object k times ago.

At step S23A-3, the object state management unit 131 determines whether or not the distance X is less than a predetermined threshold value (eg, 1 meter). If the distance X is less than the threshold (YES), the object state management section 131 advances the process to step S23A-4. If the distance X is equal to or greater than the threshold (NO), the object state management section 131 advances the process to step S23A-5.

At step S23A-4, the object state manager 131 increments the variable j.

At step S23A-5, the object state manager 131 increments the variable k.

In step S23A-6, the object state management unit 131 determines whether or not the variable k is equal to the predetermined number of times K. If the variable k is different from the predetermined number of times K (NO), the object state management section 131 returns the process to step S23A-2. If the variable k is equal to the predetermined number of times K (YES), the object state management unit 131 advances the process to step S23A-7.

At step S23A-7, the object state management unit 131 determines whether or not the variable j is equal to the predetermined number of times K. If the variable j is different from the predetermined number of times K (NO), the object state management unit 131 terminates the process. If the variable j is equal to the predetermined number of times K (YES), the object state management unit 131 advances the process to step S23A-8.

In step S23A-8, the object state management unit 131 determines whether or not a group ID is assigned to either the m-th registered object or the n-th registered object. If a group ID has been assigned (YES), the object state management unit 131 advances the process to step S23A-9. If no group ID has been assigned (NO), the object state management unit 131 advances the process to step S23A-10.

In step S23A-9, the object state management unit 131 determines one of the group IDs assigned to the m-th registered object and the n-th registered object as the group ID to be assigned. Next, the object state management unit 131 sets the determined group IDs as the group IDs of the m-th registered object and the n-th registered object in the registered object list.

The object state management unit 131 determines group IDs to be assigned as follows. If a group ID is assigned to only one registered object, the group ID is assigned as the group ID. That is, a registered object to which a group ID is not assigned is added as a member of an existing group.

When group IDs are assigned to both registered objects, first, the registered objects after the (m+1)th registered object are sorted by group ID to identify the registered object having the same group ID as the m-th registered object. Next, the group IDs of the m-th registered object and all identified registered objects are updated with the group ID of the n-th registered object. As a result, all members of the group to which the m-th registered object belongs belong to the same group as the n-th registered object. This makes it possible to grasp groups to which three or more registered objects belong.

At step S23A-10, the object state management unit 131 issues a new group ID that does not overlap with other group IDs. Next, the object state management unit 131 sets the issued new group IDs to the group IDs of the m-th registered object and the n-th registered object in the registered object list.

FIG. 16 is a flow chart showing a second example of the group ID assigning process (step S23 in FIG. 11) executed by the object state management unit 131 in this embodiment. A second example of group ID assignment processing is group determination based on group actions.

Group determination by group action is a process of performing group determination by image analysis after creating a detection object list. Therefore, when performing group determination by group action, step S23 is performed between step S21 and step S22.

Here, it is assumed that processing is executed for a combination of the m-th detected object and the n-th detected object in the detected object list.

In step S23B-1, the object state management unit 131 calculates the distance X between the three-dimensional coordinates of the n-th detected object and the m-th detected object.

In step S23B-2, the object state management unit 131 determines whether or not the distance X is less than a predetermined threshold value (eg, 1 meter). If the distance X is greater than or equal to the threshold (NO), the object state management section 131 terminates the process. If the distance X is less than the threshold (YES), the object state management section 131 advances the process to step S23B-3.

In step S23B-3, the object state management unit 131 calculates the midpoint between the three-dimensional coordinates of the n-th detected object and the three-dimensional coordinates of the m-th detected object. Specifically, the object state management unit 131 divides the sum of the X coordinates, Y coordinates, and Z coordinates of the two detected objects by two.

In step S23B-4, the object state management section 131 extracts the image P with a radius of Y pixels centered on the midpoint. The radius Y is set according to the resolution of the image so that the range included in the image P is approximately 1 m in real space.

In step S23B-5, the object state management section 131 reads out a pre-stored group action image. A group action image is a set of images representing actions determined to belong to the same group. For example, it is an image of two people greeting or shaking hands.

In step S23B-6, the object state management section 131 calculates the degree of similarity between the video P and each group action image by pattern matching or the like.

In step S23B-7, the object state management unit 131 determines whether or not any of the calculated similarities exceeds a predetermined threshold. If there is a degree of similarity exceeding the threshold (YES), the object state management unit 131 advances the process to step S23B-8. If the degree of similarity does not exceed the threshold (NO), the object state management unit 131 terminates the process.

In step S23B-8, the object state management unit 131 issues a new group ID that does not overlap with other group IDs. Next, the object state management unit 131 sets the issued new group IDs to the group IDs of the m-th detected object and the n-th detected object in the detected object list.

In step S22-4 (see FIG. 13) of the registered object list update process executed thereafter, the object state management unit 131 sets the group ID of the detected object determined to be the same object as the group ID of the registered object. do. Note that when an existing group ID is assigned to the registered object, steps S23A-8 to S23A-10 (see FIG. 15) of the first example of the group ID assignment process are executed.

Returning to FIG. 10, description will be made. In step S3, the operation information transmission unit 31 provided in the electronic device 30-1 determines whether or not the operation content is information to be notified to the management device 10 in accordance with the operation performed by the user on the electronic device 30-1. judge. This determination is made based on whether or not it matches with any of the predetermined operation contents.

When the operation information transmission unit 31 determines that the operation content should be notified, the operation information transmission unit 31 transmits the device operation information related to the device operation to the management device 10 . In the management device 10, the operation information receiving section 12 receives the device operation information from the electronic device 30-1.

Operation details and operation information in this embodiment are exemplified below.

<Example 1>
Operation details: Device login Operation information: Registered device ID, device authentication ID (if authentication is required), login time

<Example 2>
Operation details: Processing that incurs charges (copy output, usage of pay-as-you-go software, etc.)
Operation information: registered device ID, device authentication ID (if authentication is required), fee, input data, language used

<Example 3>
Operation details: None Operation information: Status information such as maintenance and energy saving mode

A registered device ID is identification information included in a registered device list stored in the state storage unit 100 of the management device 10 . A registered device ID is assigned to each electronic device 30 installed in the managed space R1. The device authentication ID is authentication information used for authentication for the user of the electronic device 30 to use the electronic device 30 .

In step S4, the state management unit 13 included in the management device 10 requests the operation information reception unit 12 for operation information. Next, the state management section 13 inputs the operation information received from the operation information reception section 12 to the device state management section 132 .

Subsequently, the device state management unit 132 executes device state update processing, which will be described later, and updates the registered device list of the state management information stored in the state storage unit 100 .

≪Device status update processing≫
Here, device status update processing in this embodiment will be described with reference to FIG. 17 . FIG. 17 is a flow chart showing an example of the device status update process (step S4 in FIG. 10) executed by the device status management unit 132 in this embodiment.

In step S41, the device status management unit 132 identifies the registered device included in the registered device list by the registered device ID included in the device operation information. Next, the device state management unit 132 sets the received device operation information to the device input information of the specified registered device.

In step S42, the device state management unit 132 initializes the variable n to 1.

In step S43, the device state management unit 132 calculates the distance X between the three-dimensional coordinates of the identified registered device and the latest three-dimensional coordinates of the n-th registered object. Note that the three-dimensional coordinates of the registered device are set in advance by the same method as for the three-dimensional coordinates of the registered object. Since the installation position of the registered device may be moved, it is necessary to periodically update the three-dimensional coordinates of the registered device, but the update frequency may be low.

In step S44, the device state management unit 132 determines whether or not the distance X is less than a predetermined threshold (for example, 1 meter). When the distance X is equal to or greater than the threshold (NO), the device state management section 132 advances the processing to step S47. If the distance X is less than the threshold (YES), the device state management unit 132 advances the process to step S45.

In step S45, the device state management unit 132 adds the device input information of the specified registered device to the device operation information of the n-th registered object. The device operation information of the registered object list is configured to be able to store a predetermined number of device input information.

In step S46, the device state management unit 132 determines whether or not the variable n is equal to the number N of data. If the variable n is different from the number of data N (NO), the device state management unit 132 advances the process to step S47. If the variable n is equal to the number of data N (YES), the device state management unit 132 terminates the process.

In step S47, the device state management unit 132 increments the variable n. After that, the device state management unit 132 returns the processing to step S43.

Returning to FIG. 10, description will be made. In step S5, the output information determination unit 133 included in the management device 10 executes output information determination processing, which will be described later, and issues an operation instruction to be transmitted to the electronic device 30 based on the state management information stored in the state storage unit 100. decide.

<<Output information determination process>>
Here, the output information determination process in this embodiment will be described with reference to FIGS. 18 and 19. FIG. FIG. 18 is a flowchart showing an example of output information determination processing (step S5 in FIG. 10) executed by the output information determination unit 133 in this embodiment.

The output information determination process is performed within the framework of the first basic flow chart (see FIG. 8). Specifically, the registered device list is set to list A, the registered object list is set to list B, and processing A1 (FIG. 18) is executed for all combinations of registered devices and registered objects. Note that there is no process corresponding to the processes A2-A5.

Here, it is assumed that the processing is executed for the combination of the n-th registered device in the registered device list and the m-th registered object in the registered object list.

In step S51, the output information determination unit 133 calculates the distance X between the latest three-dimensional coordinates of the n-th registered device and the latest three-dimensional coordinates of the m-th registered object.

In step S52, the output information determination unit 133 determines whether or not the distance X is less than a predetermined threshold (for example, 1 meter). If the distance X is greater than or equal to the threshold (NO), the device state management section 132 terminates the process. If the distance X is less than the threshold (YES), the device state management unit 132 advances the process to step S53.

In step S<b>53 , the output information determination unit 133 acquires the output condition included in the device output information of the n-th registered device from the registered device list stored in the state storage unit 100 .

In step S<b>54 , the output information determination unit 133 acquires the device operation information of the m-th registered object from the registered object list stored in the state storage unit 100 .

In step S55, the output information determination unit 133 determines whether or not any of the device operation information acquired in step S54 satisfies the output condition acquired in step S53. If none of the device operation information satisfies the output condition (NO), the device state management unit 132 terminates the process. If any of the device operation information satisfies the output condition (YES), the device state management unit 132 advances the process to step S56.

In step S56, the output information determination unit 133 acquires an operation instruction corresponding to the output condition acquired in step S53 from the device output information of the n-th registered device. It should be noted that the operation instruction of the registered device may include an adaptive process according to the attribute information of the registered device. For example, a condition may be included such that the operation instruction is not transmitted when the registered device is in an inoperable state. The inoperable state is, for example, a power-off state, a resting state, or the like.

FIG. 19 is a flow chart showing a modification of the output condition determination process (step S55 in FIG. 18) executed by the output information determination unit 133 in this embodiment.

In the output information determination process shown in FIG. 18, in step S55, it is determined whether or not the output condition is satisfied based only on the device operation information of the m-th registered object. In the output information determination process shown in FIG. 19, it is determined whether or not the output condition is satisfied by referring to the device operation information of devices other than the m-th registered device.

In step S55-1, the output information determination unit 133 determines whether or not any of the device operation information of the m-th registered object satisfies the output condition of the n-th registered device. If any of the device operation information satisfies the output condition (YES), the output information determining section 133 advances the process to step S55-10. If none of the device operation information satisfies the output condition (NO), the output information determination unit 133 advances the process to step S55-2.

In step S55-2, the output information determination unit 133 determines whether device operation information of another registered object is necessary based on the output condition of the n-th registered device. If the device operation information of another registered object is required (YES), the output information determination unit 133 advances the process to step S55-3. If the device operation information of other registered objects is unnecessary (NO), the output information determination unit 133 advances the process to step S55-9.

In step S55-3, the output information determination unit 133 initializes the variable k to one.

In step S55-4, the output information determination unit 133 determines whether or not the variable k is equal to the variable m. If the variable k is equal to the variable m (YES), the output information determination unit 133 advances the process to step S55-5. If the variable k is not equal to the variable m (NO), the output information determination unit 133 advances the process to step S55-7.

In step S55-5, the output information determination unit 133 acquires the device operation information of the k-th registered object from the registered object list stored in the state storage unit 100. FIG.

In step S55-6, the output information determination unit 133 determines whether or not any of the device operation information of the k-th registered object satisfies the output condition of the n-th registered device. If any of the device operation information satisfies the output condition (YES), the output information determining section 133 advances the process to step S55-10. If none of the device operation information satisfies the output condition (NO), the output information determination unit 133 advances the process to step S55-7.

In step S55-7, the output information determination unit 133 determines whether or not the variable k is equal to the number M of data. If the variable k is different from the number of data M (NO), the output information determining section 133 advances the process to step 55-8. If the variable m is equal to the number of data M (YES), the output information determination unit 133 advances the process to step S55-9.

In step S55-8, the output information determination unit 133 increments the variable k. After that, the output information determination unit 133 returns the process to step S55-4.

In step S55-9, the output information determining unit 133 determines that the device operation information acquired in step S54 does not satisfy the output condition acquired in step S53, and terminates the process.

In step S55-10, the output information determination unit 133 determines that the device operation information acquired in step S54 satisfies the output condition acquired in step S53, and terminates the process.

FIG. 20 is a diagram showing an example of device output information in this embodiment. A first example of the output condition for the device output information in this embodiment is an output condition that does not require device operation information for other registered objects. A second example of the output condition of the device output information in this embodiment is an output condition that requires device operation information of another registered object.

As shown in FIG. 20, the first example of the output condition in this embodiment is that the m-th registered object (hereinafter also referred to as "person α") is another electronic device (here, an MFP). ) is authentication information that can be authenticated by the n-th electronic device (here, an electronic blackboard).

As shown in FIG. 20, when it is determined that the first example of the output condition is satisfied, one or more of the following operation instructions are determined as operation instructions to be transmitted to the registered device. The first operation instruction is to automatically authenticate the person α to the electronic blackboard. In this case, the transmitted operation instruction includes the authentication information input by the person α through the MFP. The second operation instruction is to display "Mr. α, automatic authentication has been performed" on the electronic blackboard for several seconds.

The third operation instruction is, when the person α is inputting information (for example, scanning operation) on the MFP, the text "Do you want to display the information input by the device XX (name of MFP) earlier?" and "Yes" or "No" selection buttons. At this time, if "yes" is selected, the information input by the scanner is displayed on the electronic blackboard.

The fourth operation instruction is to specify the language set in the MFP from the device operation information of the person α and reflect it in the display of the electronic blackboard. A fifth operation instruction is not to transmit an operation instruction if the attribute information of the electronic blackboard is in a sleep state such as "in energy saving mode".

As shown in FIG. 20, the second example of the output condition in this embodiment is that the k-th registered object (hereinafter also referred to as "person β") is another electronic device (here, MFP). ) is authentication information that can be authenticated by the n-th electronic device (here, an electronic blackboard). It is assumed that person β has the same group ID as person α.

As shown in FIG. 20, when it is determined that the second example of the output condition is satisfied, one or more of the following operation instructions are determined as operation instructions to be transmitted to the registered device. The first operation instruction is to automatically authenticate the person α to the electronic blackboard. In this case, the operation instruction to be transmitted includes the authentication information input by the person β through the MFP. The second operation instruction is to display for several seconds on the electronic blackboard the message "Automatic authentication has been performed using the authentication information of Mr. β".

When the person β is inputting information (for example, scanning operation) on the MFP, the third operation instruction is "Do you want to display the information that Mr. β has input on the device XX (name of MFP)?" and a "yes" or "no" selection button. At this time, if "yes" is selected, the information input by the scanner is displayed on the electronic blackboard.

Returning to FIG. 10, description will be made. In step S<b>6 , the output information determination unit 133 included in the management device 10 sends action instruction information representing an action instruction to the action instruction unit 14 . The operation instruction information includes information indicating the determined operation instruction and the destination electronic device 30-2.

The operation instruction unit 14 receives operation instruction information from the output information determination unit 133 . Next, the operation instruction unit 14 transmits the operation instruction included in the operation instruction information to the destination electronic device 30-2 included in the operation instruction information.

In the electronic device 30 - 2 , the device control section 32 receives the operation instruction from the management device 10 . Next, the device control section 32 executes an operation using the input information included in the received operation instruction, according to the received operation instruction. For example, the device control unit 32 authenticates the user using authentication information included in the operation instruction. Also, for example, the device control unit 32 changes the settings of the electronic device 30-2 using the setting information included in the operation instruction.

<Effects of Embodiment>
The device management system 1 according to the present embodiment tracks the movement of a person based on an image of the vicinity of the electronic device 30, thereby giving an operation instruction using information input by the user through the first electronic device 30-1. is transmitted to the second electronic device 30-2 that has detected that the user is nearby. Therefore, according to the device management system 1 of the present embodiment, it is possible to cause the second electronic device to perform an operation using information input to the first electronic device.

In particular, by transmitting an authentication instruction using the authentication information authenticated by the first electronic device 30-1 to the second electronic device 30-2, the user authenticated by the first electronic device 30-1 can Authentication can be automatically performed when the user moves to the vicinity of the second electronic device 30-2. That is, according to the device management system of the present embodiment, single sign-on can be realized through hardware cooperation.

[Application example]
In the above embodiment, the space to be managed is assumed to be an office conference room, office, or the like, and an example in which the electronic device 30 is managed as an OA (Office Automation) device has been mainly described. However, the usage scene to which the device management system 1 can be applied is not limited to this, and can be applied to various usage scenes.

For example, the equipment management system 1 can be configured to manage electronic equipment installed in a hotel. The hotel is equipped with various electronic devices. For example, the lobby may have an automatic check-in machine. In addition, some guest rooms are equipped with set-top boxes capable of reproducing television broadcasts and on-demand videos. By managing these devices with the device management system 1, it becomes possible to automatically set the language used by the hotel guest at the automatic check-in machine as the language used for the set-top box.

Further, for example, the equipment management system 1 can be configured to manage electronic equipment installed at an airport. At airports, it is necessary to go through procedures such as ticketing, check-in, and baggage inspection before boarding an aircraft, and boarding passes are processed by dedicated electronic devices in each procedure. By managing these devices with the device management system 1, it is possible to automatically set the language used by the electronic device that performs the previous procedure to the language that is used by the electronic device that performs the next procedure.

Further, for example, the device management system 1 can be configured to manage electronic devices such as search terminals installed in bookstores. By linking the search performed by the customer with the search terminal and the information on the bookshelf that was moved after that, it can be used to strengthen marketing.

Further, for example, the device management system 1 can be configured to manage electronic devices such as PCs installed in factories. By correlating and verifying the worker's actions and the operation details of equipment installed at the work site, it is possible to monitor the worker's work validity.

Further, for example, the device management system 1 can be configured to manage electronic devices installed in an unmanned convenience store. Electronic devices such as MFPs and settlement devices are installed in convenience stores. By managing these devices by the device management system 1, for example, it is possible to set the language that the customer has set in the MFP as the language used in the settlement device.

[supplement]
In each of the embodiments described above, the management device 10 is an example of an information processing device. The device management system 1 is an example of an information processing system. The monitoring device 20 is an example of an imaging device. The state management unit 13 is an example of a user identification unit. The operation information receiving section 12 is an example of an input information receiving section.

Each function of the embodiments described above may be implemented by one or more processing circuits. Here, the "processing circuit" in this specification means a processor programmed by software to perform each function, such as a processor implemented by an electronic circuit, or a processor designed to perform each function described above. ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), FPGAs (Field Programmable Gate Arrays), and devices such as conventional circuit modules.

The devices described in the examples are merely representative of one of several computing environments for implementing the embodiments disclosed herein. In some embodiments, management device 10 includes multiple computing devices, such as server clusters. Multiple computing devices are configured to communicate with each other over any type of communication link, including a network, shared memory, etc., to perform the processes disclosed herein.

Although the embodiments of the present invention have been described in detail above, the present invention is not limited to these embodiments, and various modifications or Change is possible.

1 Device Management System 10 Management Apparatus 11 Image Receiving Section 12 Operation Information Receiving Section 13 State Management Section 14 Operation Instruction Section 100 State Storage Section 131 Object State Management Section 132 Device State Management Section 133 Output Information Determination Section 20 Monitoring Device 21 Image Acquisition Section 22 image transmission unit 30 electronic device 31 operation information transmission unit 32 device control unit

Japanese Patent No. 6064636 Japanese Patent No. 5238409

Claims (12)

An information processing device capable of communicating with a first electronic device, a second electronic device, and a photographing device via a network,
a first image receiving unit that receives a first image of a first user using the first electronic device from the imaging device;
a first user identification unit that identifies the first user who used the first electronic device based on the usage information of the first electronic device and the first image;
an input information receiving unit configured to receive, from the first electronic device, authentication information input by the first user specified by the first user specifying unit using the first electronic device;
a second image receiving unit that receives a second image of a second user present at an operable position of the second electronic device from the imaging device;
a second user identifying unit that identifies the second user present at an operable position of the second electronic device from the second image;
Based on the information and the authentication information of the first user specified by the first user specifying unit and the second user specified by the second user specifying unit, the an operation instruction unit that transmits authentication information or screen setting information in the authentication information;
Information processing device. An information processing device capable of communicating with a first electronic device, a second electronic device, and a photographing device via a network,
a first image receiving unit that receives a first image of a first user using the first electronic device from the imaging device;
a first user identification unit that identifies the first user who used the first electronic device based on the usage information of the first electronic device and the first image;
an input information receiving unit configured to receive, from the first electronic device, information on usage of the first electronic device by the first user identified by the first user identifying unit;
a second image receiving unit that receives a second image of a second user present at an operable position of the second electronic device from the imaging device;
a second user identifying unit that identifies the second user present at an operable position of the second electronic device from the second image;
Based on the information of the first user specified by the first user specifying unit and the second user specified by the second user specifying unit and the usage information, the an operation instruction unit that transmits usage information;
Information processing device. The information processing device according to claim 1 or 2,
Further comprising a state management unit that records position information of the first user and the second user in chronological order using the first image and the second image received from the imaging device,
Information processing equipment. The information processing device according to claim 3,
The state management unit identifies the first user who has operated the first electronic device based on the time series of the position information.
Information processing equipment. The information processing device according to claim 4,
The state management unit records the position information using a plurality of the first images and the second images received from each of the plurality of imaging devices.
Information processing equipment. The information processing device according to claim 1,
The input information receiving unit further receives state information representing a state of the second electronic device,
wherein the operation instructing unit does not transmit the authentication information or the screen setting information to the second electronic device in which the state information indicates a hibernation state;
Information processing equipment. An information processing system in which a first electronic device, a second electronic device, a photographing device, and an information processing device can communicate via a network,
The information processing device is
a first image receiving unit that receives a first image of a first user using the first electronic device from the imaging device;
a first user identification unit that identifies the first user who used the first electronic device based on the usage information of the first electronic device and the first image;
an input information receiving unit configured to receive, from the first electronic device, authentication information input by the first user specified by the first user specifying unit using the first electronic device;
a second image receiving unit that receives a second image of a second user present at an operable position of the second electronic device from the imaging device;
a second user identifying unit that identifies the second user present at an operable position of the second electronic device from the second image;
Based on the information and the authentication information of the first user specified by the first user specifying unit and the second user specified by the second user specifying unit, the an operation instruction unit that transmits authentication information or screen setting information in the authentication information;
An information processing system comprising An information processing system in which a first electronic device, a second electronic device, a photographing device, and an information processing device can communicate via a network,
The information processing device is
a first image receiving unit that receives a first image of a first user using the first electronic device from the imaging device;
a first user identification unit that identifies the first user who used the first electronic device based on the usage information of the first electronic device and the first image;
an input information receiving unit configured to receive, from the first electronic device, information on usage of the first electronic device by the first user identified by the first user identifying unit;
a second image receiving unit that receives a second image of a second user present at an operable position of the second electronic device from the imaging device;
a second user identifying unit that identifies the second user present at an operable position of the second electronic device from the second image;
Based on the information of the first user specified by the first user specifying unit and the second user specified by the second user specifying unit and the usage information, the an operation instruction unit that transmits usage information;
An information processing system comprising A computer capable of communicating with the first electronic device, the second electronic device and the imaging device via a network,
a first image receiving procedure for receiving, from the imaging device, a first image of a first user using the first electronic device;
a first user identification procedure for identifying the first user who used the first electronic device based on usage information of the first electronic device and the first image;
an input information receiving procedure for receiving, from the first electronic device, authentication information that the first user specified by the first user specifying procedure has entered in the first electronic device;
a second image receiving procedure for receiving, from the photographing device, a second image of a second user present at an operable position of the second electronic device;
a second user identification step of identifying the second user present at an operable position of the second electronic device from the second image;
Based on the authentication information and information of the first user identified by the first user identification procedure and the second user identified by the second user identification procedure, the an operation instruction procedure for transmitting authentication information or screen setting information in the authentication information;
A device management method that implements A computer capable of communicating with the first electronic device, the second electronic device and the imaging device via a network,
a first image receiving procedure for receiving, from the imaging device, a first image of a first user using the first electronic device;
a first user identification procedure for identifying the first user who used the first electronic device based on usage information of the first electronic device and the first image;
an input information receiving procedure for receiving, from the first electronic device, usage information that the first user identified by the first user identifying procedure uses the first electronic device;
a second image receiving procedure for receiving, from the photographing device, a second image of a second user present at an operable position of the second electronic device;
a second user identification step of identifying the second user present at an operable position of the second electronic device from the second image;
Based on the information of the first user identified by the first user identification procedure and the second user identified by the second user identification procedure and the usage information, the an operation instruction procedure for transmitting usage information;
A device management method that implements A computer capable of communicating with the first electronic device, the second electronic device and the imaging device via a network,
a first image receiving procedure for receiving, from the imaging device, a first image of a first user using the first electronic device;
a first user identification procedure for identifying the first user who used the first electronic device based on usage information of the first electronic device and the first image;
an input information receiving procedure for receiving, from the first electronic device, authentication information that the first user specified by the first user specifying procedure has entered in the first electronic device;
a second image receiving procedure for receiving, from the photographing device, a second image of a second user present at an operable position of the second electronic device;
a second user identification step of identifying the second user present at an operable position of the second electronic device from the second image;
Based on the authentication information and information of the first user identified by the first user identification procedure and the second user identified by the second user identification procedure, the an operation instruction procedure for transmitting authentication information or screen setting information in the authentication information;
program to run the A computer capable of communicating with the first electronic device, the second electronic device and the imaging device via a network,
a first image receiving procedure for receiving, from the imaging device, a first image of a first user using the first electronic device;
a first user identification procedure for identifying the first user who used the first electronic device based on usage information of the first electronic device and the first image;
an input information receiving procedure for receiving, from the first electronic device, usage information that the first user identified by the first user identifying procedure uses the first electronic device;
a second image receiving procedure for receiving, from the photographing device, a second image of a second user present at an operable position of the second electronic device;
a second user identification step of identifying the second user present at an operable position of the second electronic device from the second image;
Based on the information of the first user identified by the first user identification procedure and the second user identified by the second user identification procedure and the usage information, the an operation instruction procedure for transmitting usage information;
program to run the

Images