JP2018195194A - Information processing system, information processing apparatus, information processing method and program - Google Patents

Abstract

To provide an information processing system, an information processing apparatus, an information processing method, and a program for appropriately giving a warning according to a motion of a user and a surrounding situation.SOLUTION: An information processing system 1 comprises an acquisition device 100 that acquires images, a warning device 300 that outputs a warning, and an information processing device 200. The acquisition device transmits the acquired image to the information processing device. The warning device outputs a warning based on a content of a warning received from the information processing device. The information processing device specifies motion information representing a motion of the user and peripheral information representing a surrounding environment of the user based on the image received from the acquisition device: specifies a content of a warning regarding an operation of the user based on the operation information and the peripheral information; and transmits the content of the specified warning to the warning device.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing system, an information processing apparatus, an information processing method, and a program.

  With the spread of portable electronic devices such as smartphones, so-called “walking smartphones” that browse and operate smartphones while walking are increasing. A user on a walking smartphone may fall over or collide with others due to carelessness in the front, which is a social problem. It is preferable to warn the user who browses and operates the electronic device while moving in this way, and to prevent accidents such as a fall and a collision. However, depending on the surrounding situation, there is less danger and there is a case where it is not necessary to give a warning, and it is required to give a warning appropriately according to the user's operation and the surrounding situation.

  For example, Patent Document 1 discloses a monitoring system that determines whether or not a monitoring target has performed a predetermined operation based on a moving path and a staying time of the monitoring target based on an image of the monitoring target. Has been.

JP 2011-34458 A

  However, the technique described in Patent Document 1 determines the operation of the monitoring target, and has not been studied for monitoring the situation around the monitoring target and determining the warning content. That is, there is a problem that warning cannot be performed appropriately according to the user's operation and the surrounding situation.

  The present invention has been made in view of the above points, and provides an information processing system, an information processing apparatus, an information processing method, and a program capable of appropriately giving a warning according to a user's operation and surrounding circumstances. With the goal.

  SUMMARY An advantage of some aspects of the invention is that it is an information processing system including an acquisition device that acquires an image, a warning device that outputs a warning, and an information processing device. The acquisition device transmits the acquired image to the information processing device, the warning device outputs a warning based on the content of the warning received from the information processing device, and the information processing device receives the acquisition Based on the image received from the device, the operation information indicating the user's operation and the peripheral information indicating the user's surrounding environment are specified, and the warning regarding the user's operation is determined based on the operation information and the peripheral information. It is an information processing system which specifies the content and transmits the content of the specified warning to the warning device.

  One embodiment of the present invention includes an acquisition unit that acquires an image, a warning unit that outputs a warning, and a control unit, and the control unit operates based on the image acquired by the acquisition unit. And the peripheral information indicating the surrounding environment of the user, the content of the warning regarding the user's operation is specified based on the motion information and the peripheral information, and the content of the specified warning Based on the information processing apparatus, the warning unit outputs a warning based on the information.

  According to another aspect of the present invention, an acquisition step for acquiring an image, a warning step for outputting a warning, operation information representing a user's operation based on the image acquired in the acquisition step, and the surrounding environment of the user are provided. Identifying the surrounding information to be represented, identifying the content of the warning regarding the user's motion based on the operation information and the peripheral information, and based on the identified content of the warning, And a step of outputting a warning.

  According to one embodiment of the present invention, an acquisition procedure for acquiring an image, a warning procedure for outputting a warning, operation information representing a user's operation based on the image acquired in the acquisition procedure, and the user's Based on the procedure for identifying the surrounding information representing the surrounding environment, the procedure for identifying the content of the warning regarding the user's operation based on the operation information and the surrounding information, and the content of the identified warning, And a procedure for causing a warning unit to output a warning.

  According to the present invention, it is possible to appropriately issue a warning according to the user's operation and the surrounding situation.

It is a block diagram which shows the structural example of the information processing system which concerns on 1st Embodiment. It is a figure which shows an example of the user detection which concerns on 1st Embodiment. It is a 1st figure which shows an example of risk degree calculation which concerns on 1st Embodiment. It is a 2nd figure which shows an example of the risk degree calculation which concerns on 1st Embodiment. It is a figure which shows an example of the warning table which concerns on 1st Embodiment. It is a flowchart which shows an example of a process of the information processing system which concerns on 1st Embodiment. It is a block diagram which shows the structural example of the information processing system which concerns on 2nd Embodiment. It is a flowchart which shows an example of a process of the information processing system which concerns on 2nd Embodiment. It is a figure which shows the example of a change of the calculation formula of the risk degree which concerns on 3rd Embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
<First Embodiment>
First, the configuration of the information processing system 1 according to the first embodiment of the present invention will be described.
FIG. 1 is a block diagram illustrating a configuration of an information processing system 1 according to the first embodiment. The information processing system 1 includes an acquisition device 100, an information processing device 200, and a warning device 300. The information processing apparatus 200, the acquisition apparatus 100, and the warning apparatus 300 are connected via a wired or wireless communication line. Note that the acquisition device 100 or the warning device 300 may be configured as an acquisition unit or a warning unit having the same function as the acquisition device 100 or the warning device 300 in the information processing device 200.

  The acquisition device 100 acquires operation information representing the user's operation and peripheral information representing the user's surrounding environment. In the present embodiment, the acquisition device 100 is an imaging device such as a CMOS (Complementary Metal Oxide Semiconductor) camera or a CCD (Charge Coupled Device) camera, and captures images of the user and the user's surroundings within a predetermined range. The acquisition device 100 transmits imaging data (still image or moving image) including the captured image to the information processing device 200. A plurality of acquisition devices 100 may exist. Further, the acquisition device 100 is not limited to the imaging device, and may acquire operation information representing the user's operation and peripheral information representing the user's surrounding environment using various sensors such as an infrared sensor. The acquisition device 100 may provide a communication function for acquiring an image from the acquisition device via a network, for example.

  The information processing apparatus 200 is a computer apparatus having an information processing function, such as a server apparatus or a personal computer. The information processing apparatus 200 includes a control unit 210, a storage unit 220, and a communication unit 230.

  The control unit 210 controls various functions of the information processing apparatus 200. Some or all of the functions of the control unit 210 are realized by, for example, one or more processors such as a CPU (Central Processing Unit) provided in the control unit 210 executing various programs stored in the storage unit 220. May be. The control unit 210 includes a user detection unit 211, a periphery detection unit 212, a risk level calculation unit 213, and a warning identification unit 214.

The user detection unit 211 detects a user (hereinafter, also referred to as a first user) who is moving while satisfying a predetermined condition. In the present embodiment, the user detection unit 211 acquires the imaging data acquired from the acquisition device 100 from the storage unit 220. The user detection unit 211 analyzes the acquired imaging data and extracts a user (person) included in the imaging data. Then, the user detection unit 211 determines whether or not the extracted user is walking or running while holding the electronic device and browsing the electronic device. The user detection unit 211 performs the determination by, for example, information on the position, orientation, moving direction, and angle of the user's body part (head, face, hand, arm, etc.) and the user body part (hand, arm, etc.). This is based on the appearance, shape, etc. of nearby objects (electronic devices, etc.). The user detecting unit 211 identifies information (position, moving direction, moving speed, etc.) about the first user determined to be moving by satisfying a predetermined condition as a result of the determination, and detects the surroundings. To the unit 212 and the risk calculation unit 213.
An example of detection of the first user by the user detection unit 211 will be described later.

The surroundings detection unit 212 detects an object existing around the first user input from the user detection unit 211. Examples of the object include various objects such as persons and animals around the first user, vehicles such as cars and trains, shopping carts and vending machines. In the present embodiment, the periphery detection unit 212 detects a user (hereinafter also referred to as a second user) existing around the first user. The surrounding detection unit 212 acquires the imaging data acquired from the acquisition device 100 from the storage unit 220. The periphery detection unit 212 analyzes the acquired imaging data and extracts a second user existing within a predetermined range from the first user. The surroundings detection unit 212 identifies information (position, moving direction, moving speed, etc.) about the second user as surrounding information, and outputs it to the risk level calculation unit 213.
An example of detection of the second user by the periphery detection unit 212 will be described later. Note that the user detection unit 211 or the periphery detection unit 212 may perform the above-described processing based on imaging data obtained from the plurality of acquisition devices 100.

The risk level calculation unit 213 calculates the risk level based on the detection results of the user detection unit 211 and the periphery detection unit 212. The degree of risk is information representing the degree of risk (influence) that the action of the first user exerts on the first user or the second user. In the present embodiment, the risk level is calculated based on a first risk level that indicates the level of risk for the first user and a second level of risk that indicates the level of risk for the second user. The elements for calculating the first risk level and the second risk level are the number of people (second users) existing around the first user, the moving speed, the moving direction, the position of the first user and the second user, It includes information on moving bodies (automobiles, shopping carts, etc.) around the first user and the second user, information on installed objects and obstacles around the first user and the second user, and the like. The risk level calculation unit 213 calculates the risk level based on the first risk level and the second risk level, and outputs the risk level to the warning specifying unit 214.
An example of risk level calculation by the risk level calculation unit 213 will be described later.

The warning specifying unit 214 specifies the content of the warning related to the operation of the first user based on the risk level input from the risk level calculation unit 213. The content of the warning includes information indicating the target of the warning, the content (message, etc.) and the warning means. The content of the warning may include information indicating the location and timing of the warning. The warning specifying unit 214 specifies the content of the warning based on the warning table held by the storage unit 220. Thereafter, the warning specifying unit 214 transmits a warning result specifying result to the warning device 300 via the communication unit 230.
A specific example of warning content by the warning specifying unit 214 will be described later.

  The storage unit 220 includes, for example, an HDD (Hard Disc Drive), a flash memory, an EEPROM (Electrically Erasable Programmable Read Only Memory), a ROM (Read Only Memory), or a RAM (Random Access Memory). The storage unit 220 stores various information and programs related to the information processing apparatus 200. The storage unit 220 stores, for example, imaging data acquired from the acquisition device 100 via the communication unit 230, a warning table, a specific result of warning content, and the like.

  The communication unit 230 includes an external interface for performing communication between devices via a wired or wireless communication line, and communicates with the acquisition device 100 and other external devices based on the control of the control unit 210.

  The warning device 300 performs various warnings based on the warning information received from the information processing device 200. The warning device 300 includes, for example, a speaker and a directional speaker, and issues a warning by voice. A directional speaker is a speaker that controls the direction of sound propagation and does not diffuse the sound. By using a directional speaker, audio can be selectively sent to a person in a specific narrow range. That is, it is possible to make only the warning target person hear the warning sound and prevent the surrounding people from listening to the unnecessary warning sound. In addition, the warning device 300 may include, for example, a display or an LED (Light Emitting Diode) light, and may give a warning by characters, video, various symbols, flashing lights, and the like. The warning device 300 may provide a communication function that gives a warning to the warning device via a network, for example.

Next, an example of detection of the first user and the second user by the user detection unit 211 and the periphery detection unit 212 will be described.
FIG. 2 is a diagram illustrating an example of user detection in the present embodiment. In the example of FIG. 2, the imaging range (view angle) of the imaging data captured by the acquisition apparatus 100 is indicated by A1.
The user detection unit 211 analyzes an image in the imaging range A1, and extracts a person included in the imaging range A1. The user detection unit 211 may extract a person by various methods. For example, a method based on a background difference that is detected based on a difference between a video in which only the background is captured in advance and an input video may be used. Also, for example, a method of determining whether a person is a pattern image obtained by cutting out an image within a detection window having a set size may be used.

  The user detection unit 211 determines whether or not the extracted user is walking or running while holding the electronic device and browsing the electronic device. In the present embodiment, the user detection unit 211 detects the position of the user's hand and face, and determines whether there is a gripping object at the position of the user's hand. When there is a gripping object, the user detection unit 211 determines from the appearance whether the gripping object is an electronic device. In addition, the user detection unit 211 determines whether the user's face is facing the direction of the electronic device being held for a predetermined time or more. Furthermore, the user detection unit 211 determines whether the user is moving at a predetermined speed or higher. When the above conditions are met, the user detection unit 211 determines that the user is walking or running while holding the electronic device and browsing the electronic device. The user detection unit 211 manages the user as a first user, and outputs information (position, movement direction, movement speed, etc.) of the first user to the periphery detection unit 212. In addition, you may perform the above-mentioned determination using other determination methods (line-of-sight detection etc.).

The surroundings detection unit 212 extracts users existing around the first user based on the first user information acquired from the user detection unit 211. In the example of FIG. 2, the user 1, the user 2, and the user 3 existing in the circle with the radius R are extracted with the user 1 being the first user as the center, and are managed as the second user. The periphery detection unit 212 does not extract the user 4 because the user 4 does not exist within the circle with the radius R. The range for extracting the second user is not limited to the circle centered on the user 1, and for example, the range of the user's traveling direction may be set wide, and the terrain around the first user and the shape of the premises such as the station You may change according to. The surroundings detection unit 212 outputs the extracted information (position, moving direction, moving speed, etc.) of the second user to the risk level calculation unit.
Above, description about the example of the detection of the 1st user by the user detection part 211 and the periphery detection part 212 and a 2nd user is complete | finished.

Next, an example of risk level calculation by the risk level calculation unit 213 will be described.
FIG. 3 is a first diagram illustrating an example of risk level calculation according to the present embodiment. FIG. 3A is a diagram illustrating a calculation example of the first risk level. In the present embodiment, the first risk y is expressed by a function y = f (x) having the number x of second users as a parameter. The risk level calculation unit 213 inputs the number of second users around the first user acquired from the periphery detection unit 212 to the function, and acquires the first risk level. In this example, since the number of second users is three, f (3) = 4 is the first risk level.

  FIG. 3B is a diagram illustrating a calculation example of the second risk level. In the present embodiment, the second risk level y is expressed by a function y = g (x) having the number x of second users as a parameter, as in FIG. The risk level calculation unit 213 inputs the number of second users around the first user acquired from the periphery detection unit 212 to the function, and acquires the second risk level. In this example, since the number of second users is 3, g (3) = 2 is the second risk level.

  The functions y = f (x) and y = g (x) are set according to various conditions and can be various functions. The parameters are not limited to the number of second users, and various parameters can be used. Further, the risk level calculation unit 213 may acquire the first risk level based on a predetermined condition regardless of the function.

FIG. 4 is a second diagram illustrating an example of risk level calculation according to the present embodiment. The risk level calculation unit 213 calculates a risk level based on the calculated first risk level and second risk level. In the present embodiment, as shown in FIG. 4, the risk level calculation unit 213 sets the risk level to four (A, B, C, D) with the horizontal axis as the first risk level and the vertical axis as the second risk level. Classify. In the example of FIG. 4, since the first risk level is level 4 and the second risk level is level 2, the risk level is classified as B. The calculation of the risk level is not limited to the classification based on the above quadrant (matrix), and may be performed by other methods.
This is the end of the description of the example of risk level calculation by the risk level calculation unit 213.

Next, a specific example of warning content by the warning specifying unit 214 will be described.
The warning specifying unit 214 specifies the content of the warning based on the risk input from the risk calculation unit 213 and the warning table held by the storage unit 220.

  FIG. 5 is a diagram illustrating an example of a warning table according to the present embodiment. As shown in FIG. 5, the warning table has a record for each risk level, and each record has fields of “description” and “warning content” for the risk level.

The “explanation” field is a field indicating the explanation of each risk level. For example, the risk level “A” indicates that “the risk level is high for both the person and the surrounding area and a warning is required”.
The field of “warning content” includes fields of “warning target”, “warning message”, and “warning means”.

The “warning target” field is a field indicating a target to be warned.
The “warning message” field is a field indicating the content of the warning to be notified to the warning target by audio, video, or the like.
The “warning means” field is a field indicating a means for giving a warning. The warning means includes a warning by sound using a speaker or a directional speaker, and a warning by characters or video using a display.

For example, when the risk level is “A”, for “first user”, “Please stop walking. It ’s dangerous.”, For “second user”, “There are smartphone users walking around. "This is dangerous."
In the example of FIG. 4, since the risk level is “B”, for “the first user”, “Please stop walking. It ’s dangerous.” And “speech (directional speaker)” In this case, the warning device 300 emits a warning sound toward the first user through a directional speaker.

As described above, the warning specifying unit 214 specifies the warning content (warning target, warning message, warning means) based on the warning table, and transmits the specified result to the warning device 300 via the communication unit 230.
Above, description about the specific example of the warning content by the warning specific | specification part 214 is complete | finished.

Next, a processing flow of the information processing system 1 will be described.
FIG. 6 is a flowchart illustrating an example of processing of the information processing system 1 according to the present embodiment.

(Step S <b> 101) The information processing apparatus 200 acquires imaging data in a predetermined range captured by the acquisition apparatus 100 via the communication unit 230. Thereafter, the process proceeds to step S102.
(Step S102) The user detection unit 211 analyzes the acquired imaging data and extracts a person included in the imaging data. Thereafter, the process proceeds to step S103.

(Step S103) The user detection unit 211 determines whether the extracted user is walking or running (that is, the first user) while holding the electronic device and browsing the electronic device. When it determines with a 1st user (step S103 / YES), it progresses to the process of step S104. If the user is not extracted, or if the extracted user is not determined to be the first user (step S103 / NO), the process returns to step S101.
(Step S <b> 104) The periphery detection unit 212 analyzes the imaging data, and extracts information around the first user (object, second user, etc.). Thereafter, the process proceeds to step S105.

(Step S <b> 105) The risk level calculation unit 213 calculates the risk level based on the first user's operation information input from the user detection unit 211 and the first user's peripheral information input from the periphery detection unit 212. Specifically, as described in the example of FIGS. 3 and 4, the first risk level and the second risk level are calculated, and the risk level is calculated based on the calculated first risk level and second risk level. To do. Thereafter, the process proceeds to step S106.
(Step S106) The warning specifying unit 214 specifies the content of the warning (warning target, warning message, warning means, etc.) based on the risk input from the risk calculation unit 213. The warning identification unit 214 transmits the identification result to the warning device 300 via the communication unit 230. Thereafter, the process proceeds to step S107.

(Step S107) The warning device 300 issues a warning based on the specific result received from the information processing device 200. Specifically, for the warning target (first user, second user, etc.) described in the specific result, the warning message content (such as “Please be careful”) is displayed as warning means (speaker output, display). Warning). Thereafter, the process ends.
This is the end of the description of the processing flow of the information processing system 1.

As described above, the information processing system 1 according to the present embodiment is an information processing system including the acquisition device 100 that acquires an image, the warning device 300 that outputs a warning, and the information processing device 200. The apparatus 100 transmits the acquired image to the information processing apparatus 200, the warning apparatus 300 outputs a warning based on the content of the warning received from the information processing apparatus 200, and the information processing apparatus 200 receives the image received from the acquisition apparatus 100. Based on the action information indicating the action of the user and the peripheral information indicating the user's surrounding environment, the content of the warning about the action of the user is specified based on the action information and the peripheral information. Is transmitted to the warning device 300.
Thereby, a warning can be appropriately given according to a user's operation and surrounding circumstances.

Further, the information processing apparatus 200 according to the present embodiment includes an acquisition unit that acquires an image, a warning unit that outputs a warning, and a control unit 210. The control unit 210 is based on the image acquired by the acquisition unit. Identify the action information that represents the user's action and the surrounding information that represents the user's surrounding environment, identify the contents of the warning about the user's action based on the action information and the surrounding information, and based on the contents of the identified warning The warning part is made to output a warning.
Thereby, a warning can be appropriately given according to a user's operation and surrounding circumstances.

Further, the control unit 210 detects a first user who is moving while satisfying a predetermined condition based on the operation information, and detects an object present around the first user based on the peripheral information. The risk level is calculated based on the detection result of the first user and the object, and the content of the warning regarding the operation of the first user is specified based on the risk level.
Accordingly, it is possible to appropriately specify the content of the warning regarding the action of the user who gives a risk based on the risk level.

Further, the control unit 210 detects a second user existing around the first user based on the peripheral information, and specifies the content of the warning for the first user or the second user based on the degree of risk.
Thereby, based on the degree of danger, it is possible to appropriately warn a user who is in danger of operating or a user in the vicinity thereof.

In addition, the control unit 210 calculates the risk level based on the first risk level for the first user and the second risk level for the second user.
As a result, it is possible to calculate the risk level with higher accuracy by using the respective risk levels of the user in danger and the surrounding users.

In addition, the control unit 210 calculates the degree of risk based on the number of the second users.
As a result, it is possible to appropriately determine the content of the warning according to the degree of congestion around the user who is in danger of operating.

In addition, the control unit 210 detects the user who is walking or running while holding the electronic device and browsing the electronic device as the first user.
Thereby, the user who is carrying out what is called a walk smartphone can be detected, and a warning can be appropriately performed according to the surrounding situation.

<Second Embodiment>
Next, a second embodiment of the present invention will be described. In addition, about the structure similar to embodiment mentioned above, the same code | symbol is attached | subjected and the description is used.

  FIG. 7 is a block diagram showing a configuration of the information processing system 2 according to the present embodiment. The information processing system 2 includes an acquisition device 100, an information processing device 200, a warning device 300, a second acquisition device 400, a second information processing device 500, and a second warning device 600.

  The configurations of the acquisition device 100, the information processing device 200, and the warning device 300 are the same as those in the first embodiment. However, in the present embodiment, the warning specifying unit 214 specifies a place or timing for executing the warning based on the degree of risk. As a result, it is not necessary to give a warning immediately, but it is possible to provide a flexible warning such as alerting the person later (for example, a warning message is displayed on the display screen of the ticket gate when the user passes the ticket gate). Become.

  The warning specifying unit 214 transmits the specifying result via the communication unit 230 to the device specified as the warning place (in this embodiment, the warning is given by the second warning device 600). The identification result may include information for identifying the warning target user. The information for specifying the user may be, for example, a feature amount indicating clothes, physique, personal belongings, and the like.

  The second information processing device 500 is connected to the information processing device 200, the second acquisition device, and the second warning device via a wired or wireless communication line. The second acquisition device 400 or the second warning device 600 is a second acquisition unit or second warning unit having the same function as the second acquisition device 400 or the second warning device 600 in the second information processing device 500. It may be configured as.

  In the present embodiment, the second acquisition device 400 is an imaging device having the same function as the acquisition device 100. The second acquisition device 400 captures images of the user and the surroundings of the user in a predetermined range. The second acquisition device 400 transmits imaging data (still image or moving image) including the captured image to the second information processing device 500.

  The second information processing apparatus 500 is a computer apparatus having an information processing function, for example, a server apparatus or a personal computer. The second information processing apparatus 500 includes a second control unit 510, a second storage unit 520, and a second communication unit 530.

  The second control unit 510 controls various functions of the second information processing apparatus 500. Some or all of the functions of the second control unit 510 include various programs stored in the second storage unit 520 by one or more processors such as a CPU (Central Processing Unit) provided in the second control unit 510, for example. It may be realized by executing. The second control unit 510 includes a second user detection unit 511 and a second warning identification unit 512.

  The second user detection unit 511 analyzes the imaging data acquired from the second acquisition device 400 and determines whether or not the warning target first user is included in the imaging data. For example, the second user detection unit 511 determines that the user is the first user to be warned when the feature amount of the user received from the information processing apparatus 200 matches a predetermined criterion or more. When the second user detecting unit 511 extracts the first user to be warned, the second user detecting unit 511 outputs the result to the second warning identifying unit 512.

  The second warning specifying unit 512 acquires a warning content specifying result from the information processing apparatus 200 via the second communication unit 530. The result of specifying the warning content includes a warning target, a warning message, a warning means, a warning location, and a warning timing. The second warning specifying unit 512 determines the content of the warning based on the warning specifying result, and causes the second warning device 600 to perform the warning. For example, the warning target is “User A”, the warning message is “Please stop walking. Dangerous.”, The warning means is “second warning device (ticket gate display screen)”, and the warning location is “station ticket gate”. The warning timing is “at the time of passing the ticket gate.” In this case, based on the user information acquired from the second user detection unit, the second warning specifying unit 512 passes the ticket gate at the station ticket gate. At the timing, the second warning device (ticket gate display screen) is controlled to display “Stop walking. It is dangerous.”

  The second storage unit 520 includes, for example, an HDD (Hard Disc Drive), a flash memory, an EEPROM (Electrically Erasable Programmable Read Only Memory), a ROM (Read Only Memory), or a RAM (Random Access Memory). The second storage unit 520 stores various information and programs related to the second information processing apparatus 500. The second storage unit 520 stores, for example, imaging data acquired from the second acquisition device 400, a warning identification result acquired from the information processing device 200, and the like.

  The second communication unit 530 includes an external interface for performing communication between devices via a wired or wireless communication line. Based on the control of the second control unit 510, the second acquisition device 400 and other external devices Communicate with.

  The second warning device 600 is installed in a place different from the imaging range by the acquisition device 100 and performs various warnings based on the control of the second information processing device 500. As described above, the second warning device 600 may be a ticket gate display screen, and, like the warning device 300, may be a speaker or a directional speaker, and may give a voice warning, a display or an LED. (Light Emitting Diode) A light may be provided, and a warning may be given by characters, video, various symbols, blinking light, and the like.

Next, a processing flow of the information processing system 2 will be described.
FIG. 8 is a flowchart illustrating an example of processing of the information processing system 2 according to the present embodiment. In FIG. 8, the processing from step S101 to step S107 is the same as that of the first embodiment, and thus the description thereof is omitted, and the description starts from step S201.

(Step S201) The warning specifying unit 214 determines whether or not the warning timing is immediate, and if it is an immediate warning (YES in Step S201), the process proceeds to Step S107. If it is not an immediate warning (step S201 / NO), the process proceeds to step S202.
(Step S202) The warning specifying unit 214 transmits the specific result to a warning device (in this example, the second information processing device connected to the second warning device) installed at the warning execution location. Thereafter, the process proceeds to step S203.

(Step S <b> 203) The second information processing apparatus 500 acquires imaging data in a predetermined range captured by the second acquisition apparatus 400 via the second communication unit 530. Thereafter, the process proceeds to step S204.
(Step S <b> 204) The second user detection unit 511 analyzes the acquired imaging data and extracts a warning target user. When the warning target user is extracted, the second user detection unit 511 notifies the second warning specifying unit 512 that the warning target has been extracted. Thereafter, the process proceeds to step S205.

(Step S205) The second warning specifying unit 512 determines whether the warning location and the warning timing match the specified result. When the warning location and the warning timing match the specified result (step S205 / YES), the second warning specifying unit 512 instructs the second warning device 600 to perform the warning. Thereafter, the process proceeds to step S206. If the warning location or warning timing does not match the specific result (step S205 / NO), the process returns to step S203.
(Step S206) The second warning device 600 issues a warning based on an instruction from the second warning specifying unit 512. Thereafter, the process ends.
Above, description of an example of the process of the information processing system 2 is complete | finished.

As described above, in the information processing system 2 according to the present embodiment, the control unit 210 of the information processing device 200 identifies the timing or place where the warning is given to the first user based on the degree of risk.
Thereby, the timing or place of warning can be specified appropriately.

Further, the control unit 210 causes the second warning device 600 to output a warning to the first user at a location different from the location captured in the image based on the specific result.
Thereby, a warning can be given at an appropriate place according to the degree of danger.

<Third Embodiment>
Next, a third embodiment of the present invention will be described. In addition, about the structure similar to embodiment mentioned above, the same code | symbol is attached | subjected and the description is used.
This embodiment has the same system configuration as the first or second embodiment. In the present embodiment, the risk level calculation unit 213 changes the calculation formula for the first risk level or the second risk level based on the detection result of the user detection unit 211 or the periphery detection unit 212.

  FIG. 9 is a diagram illustrating a modification example of the risk degree calculation formula according to the present embodiment. As illustrated in FIG. 9A, the risk level calculation unit 213 changes the first risk level calculation formula from y = f (x) to y = based on the detection result of the user detection unit 211 or the surrounding detection unit 212. Change to f '(x). Further, as shown in FIG. 9B, the risk level calculation unit 213 calculates the second risk level calculation formula from y = g (x) based on the detection result of the user detection unit 211 or the periphery detection unit 212. Change to y = g ′ (x). An example of the opportunity for the risk level calculation unit 213 to change the calculation formula for each risk level will be described below.

  A first example will be described. In this example, the user detection unit 211 or the periphery detection unit 212 calculates the moving direction and moving speed of the first user or the second user based on the imaging data. The user detection unit 211 or the periphery detection unit 212 may calculate the moving direction and moving speed of the user at the current time from the imaging data (video) for a predetermined period, A predicted value of the moving speed may be calculated. Further, the user detection unit 211 or the periphery detection unit 212 may calculate the moving direction and moving speed of the user using the user's acceleration, body orientation, and the like.

The risk level calculation unit 213 changes the calculation formula for each risk level based on the movement direction and movement speed of the user acquired from the user detection unit 211 or the periphery detection unit 212. Specifically, for example, when there is a low possibility that both users are in contact with each other from the moving direction of the first user and the moving direction of the second user (when both users pass each other and move in the opposite direction, etc.), it is dangerous. As shown in FIG. 9A, the degree calculation unit 213 changes the first risk degree calculation formula to y = f ′ (x), which is lower than y = f (x). Also, for example, when the moving speed of the first user is faster than a predetermined speed, the risk level calculation unit 213 changes the calculation formula for the first risk level to a function that takes a value higher than y = f (x). May be.
In this way, by setting and changing the risk level calculation formula based on the moving direction and speed of the user, the risk level can be calculated more appropriately and used to specify the warning content.

  A second example will be described. In this example, the user detection unit 211 or the periphery detection unit 212 acquires information on the current location of the first user or the second user. The information on the current location is information representing places such as parks, station platforms, roads, bridges, and the like. In addition, the user detection unit 211 or the periphery detection unit 212 may also acquire information accompanying the current location such as the ground being uneven or being muddy and slippery. The user detection unit 211 or the periphery detection unit 212 may analyze the imaging data acquired from the acquisition device 100 and acquire information about the current location and accompanying information. In addition, the user detection unit 211 or the periphery detection unit 212 may acquire information on the current location or accompanying information from an external device (such as a map data server) via the communication unit 230.

The risk level calculation unit 213 changes the calculation formula for each risk level based on information about the current location of the user acquired from the user detection unit 211 or the periphery detection unit 212 and accompanying information. For example, the risk level calculation unit 213 determines that the current location of the user is a park and there is no danger of a car or the like, but there is a flower bed fence around and there is a risk of being stabbed in the event of a fall. change.
In this way, by setting and changing the calculation formula of the risk level based on the information on the current location of the user and the information accompanying the location, the risk level can be calculated more appropriately and used for identifying the warning content. .

  A third example will be described. In this example, the periphery detection unit 212 acquires information about a moving body other than a human being around the first user. The moving body is, for example, a vehicle such as an automobile, a bicycle, or a stroller, a carry for conveying an article, and the like. The periphery detection unit 212 may also acquire accompanying information such as the moving direction and moving speed of the moving object. The periphery detection unit 212 may analyze the imaging data acquired from the acquisition device 100 and acquire information about the moving body and accompanying information. In addition, the periphery detection unit 212 may acquire information on the moving body and accompanying information from an external device via the communication unit 230.

The risk level calculation unit 213 changes the calculation formula for each risk level based on the information about the moving object acquired from the periphery detection unit 212 and the accompanying information. For example, the risk level calculation unit 213 changes the risk level calculation formula to a calculation formula that provides a higher risk level when there are few people in the vicinity but the car is facing the first user.
In this way, by setting and changing the risk level calculation formula based on information about moving objects other than humans around the user and the information accompanying it, the risk level can be calculated more appropriately and the content of the warning can be specified. Can be used.

  A fourth example will be described. In this example, the periphery detection unit 212 acquires information related to installation objects and obstacles around the first user. The installation object and the obstacle are, for example, various objects such as a trash can, a bench, a post, a vending machine, and a guardrail. The periphery detection unit 212 may analyze the imaging data acquired from the acquisition device 100 and acquire information on the installation object and the obstacle. In addition, the periphery detection unit 212 may acquire information regarding an installation object or an obstacle from an external device via the communication unit 230.

The risk level calculation unit 213 changes the calculation formula for each risk level based on the information about the installation object and the obstacle acquired from the surroundings detection unit 212. For example, when there are few people in the vicinity, but the first user is proceeding without decreasing the speed in the direction of the obstacle, the risk level calculation unit 213 obtains a higher risk level as a risk level calculation formula. Change to the calculation formula.
In this way, by setting and changing the calculation formula of the risk level based on the information about the installation object and the obstacle, the risk level can be calculated more appropriately and used for specifying the warning content.

  Note that a part or all of the acquisition device 100, the information processing device 200, the warning device 300, the second acquisition device 400, the second information processing device 500, and the second warning device 600 in each of the embodiments described above is realized by a computer. It may be. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed. Here, the “computer system” is a computer system built in each device, and includes hardware such as an OS (Operating System) and peripheral devices.

  The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” is a medium that dynamically holds a program for a short time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, In this case, a volatile memory inside a computer system that serves as a server or a client may be included that holds a program for a certain period of time. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

  In addition, some or all of the acquisition device 100, the information processing device 200, the warning device 300, the second acquisition device 400, the second information processing device 500, and the second warning device 600 in the above-described embodiment may be integrated into an LSI (Large Scale). Integration) or the like. Each functional unit of each device may be individually made into a processor, or a part or all of them may be integrated into a processor. Further, the method of circuit integration is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. In addition, when an integrated circuit technology that replaces LSI appears due to the advancement of semiconductor technology, an integrated circuit based on the technology may be used.

As described above, the embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to the above, and various design changes and the like can be made without departing from the scope of the present invention. It is possible to
In addition, one aspect of the present invention can be modified in various ways within the scope of the claims, and the technical aspects of the present invention also relate to embodiments obtained by appropriately combining technical means disclosed in different embodiments. Included in the range. Moreover, it is the element described in said each embodiment and modification, and the structure which substituted the element which has the same effect is also contained.

Note that one aspect of the present invention can be understood from the following aspects.
(1) An information processing system including an acquisition device that acquires an image, a warning device that outputs a warning, and an information processing device, wherein the acquisition device transmits the acquired image to the information processing device, and The warning device outputs a warning based on the content of the warning received from the information processing device, and the information processing device includes operation information representing the user's operation based on the image received from the acquisition device and the surroundings of the user Information processing for specifying surrounding information representing an environment, specifying the content of a warning regarding the user's operation based on the operation information and the peripheral information, and transmitting the specified content of the warning to the warning device system.

(2) An acquisition unit that acquires an image, a warning unit that outputs a warning, and a control unit, wherein the control unit represents operation information representing a user's operation based on the image acquired by the acquisition unit, and the control unit Identifying peripheral information representing the surrounding environment of the user, identifying the content of a warning regarding the user's operation based on the operation information and the peripheral information, and based on the identified content of the warning, the warning unit An information processing device that causes a warning to be output.

(3) The control unit detects a first user who is moving while satisfying a predetermined condition based on the operation information, and exists around the first user based on the peripheral information. The object is detected, the risk level is calculated based on the detection result of the first user and the object, and the content of the warning regarding the operation of the first user is specified based on the risk level. Information processing device.

(4) The control unit detects a second user existing around the first user based on the peripheral information, and displays a warning content for the first user or the second user based on the degree of risk. The information processing apparatus according to (3), which is specified.

(5) The information processing apparatus according to (4), wherein the control unit calculates the risk level based on a first risk level for the first user and a second risk level for the second user. .

(6) The information processing apparatus according to (4) or (5), wherein the control unit calculates the degree of risk based on the number of the second users.

(7) The information processing apparatus according to any one of (3) to (6), wherein the control unit calculates the degree of risk based on a moving direction of the first user and the object.

(8) The information processing apparatus according to any one of (3) to (7), wherein the control unit specifies a timing or a place where a warning is given to the first user based on the degree of risk. .

(9) Based on the specific result, the control unit causes the warning device to output a warning to the first user at a location different from the location captured in the image. (8) The information processing apparatus described in 1.

(10) The control unit grips an electronic device and detects the user who is walking or running while browsing the electronic device as the first user, (3) to (9) The information processing apparatus described in 1.

(11) Specifying an acquisition step for acquiring an image, a warning step for outputting a warning, operation information representing a user's operation based on the image acquired in the acquisition step, and peripheral information representing the surrounding environment of the user The step of specifying the content of the warning about the user's operation based on the operation information and the peripheral information, and the step of causing the warning unit to output a warning based on the content of the specified warning. And an information processing method.

(12) An acquisition procedure for acquiring an image on a computer, a warning procedure for outputting a warning,
Based on the operation information representing the user's motion and the peripheral information representing the user's surrounding environment based on the image acquired in the acquisition procedure, and based on the motion information and the peripheral information, the user's A program for executing a procedure for specifying the content of a warning related to an operation and a procedure for causing the warning unit to output a warning based on the specified content of the warning.

1, 2 ... Information processing system, 100 ... Acquisition device, 200 ... Information processing device, 210 ... Control unit, 211 ... User detection unit, 212 ... Periphery detection unit, 213 .. Risk level calculation unit, 214... Warning identification unit, 220... Storage unit, 230 .. communication unit, 300... Warning device, 400. 2 information processing devices, 510 ... second control unit, 511 ... second user detection unit, 512 ... second warning identification unit, 520 ... second storage unit, 530 ... second communication Part, 600 ... second warning device

Claims (12)

An information processing system comprising an acquisition device that acquires an image, a warning device that outputs a warning, and an information processing device,
The acquisition device transmits the acquired image to the information processing device,
The warning device outputs a warning based on the content of the warning received from the information processing device,
The information processing apparatus includes:
Identifying operation information representing a user's operation based on the image received from the acquisition device and surrounding information representing the user's surrounding environment;
Based on the operation information and the surrounding information, identify the content of the warning about the user's operation,
Sending the content of the identified warning to the warning device;
Information processing system. An acquisition unit that acquires an image, a warning unit that outputs a warning, and a control unit;
The controller is
Based on the image acquired by the acquisition unit, specify the operation information representing the user's operation and the peripheral information representing the user's surrounding environment,
Based on the operation information and the surrounding information, identify the content of the warning about the user's operation,
Based on the content of the identified warning, the warning unit outputs a warning,
Information processing device. The controller is
Based on the operation information, a first user who is moving while satisfying a predetermined condition is detected,
Based on the surrounding information, an object existing around the first user is detected,
Based on the detection result of the first user and the object, a risk level is calculated,
Identifying the content of a warning regarding the action of the first user based on the degree of risk;
The information processing apparatus according to claim 2. The controller is
Based on the surrounding information, a second user existing around the first user is detected,
Identifying warning content for the first user or second user based on the degree of risk;
The information processing apparatus according to claim 3. The control unit calculates the risk based on a first risk for the first user and a second risk for the second user.
The information processing apparatus according to claim 4. The control unit calculates the degree of risk based on the number of the second users.
The information processing apparatus according to claim 4 or 5. The control unit calculates the degree of risk based on a moving direction of the first user and the object;
The information processing apparatus according to any one of claims 3 to 6. The control unit specifies a timing or a place to give a warning to the first user based on the degree of risk.
The information processing apparatus according to any one of claims 3 to 7. The control unit causes the warning device to output a warning to the first user at a place different from the place captured in the image based on the specific result.
The information processing apparatus according to claim 8. The control unit grips an electronic device and detects the user walking or running while browsing the electronic device as the first user.
The information processing apparatus according to any one of claims 3 to 9. An acquisition step of acquiring an image;
A warning step that outputs a warning;
Identifying operation information representing a user's operation based on the image obtained in the obtaining step and surrounding information representing the user's surrounding environment;
Identifying the content of the warning about the user's action based on the action information and the peripheral information;
Based on the content of the identified warning, causing the warning unit to output a warning;
An information processing method comprising: On the computer,
An acquisition procedure to acquire an image;
A warning procedure that outputs a warning;
A procedure for specifying operation information representing a user's operation based on the image acquired in the acquisition procedure and peripheral information representing the user's surrounding environment;
A procedure for identifying the content of a warning regarding the user's action based on the action information and the peripheral information;
Based on the contents of the identified warning, a procedure for outputting a warning to the warning unit;
A program for running

Images