WO2015068440A1 - 情報処理装置、制御方法およびプログラム - Google Patents
情報処理装置、制御方法およびプログラム Download PDFInfo
- Publication number
- WO2015068440A1 WO2015068440A1 PCT/JP2014/070652 JP2014070652W WO2015068440A1 WO 2015068440 A1 WO2015068440 A1 WO 2015068440A1 JP 2014070652 W JP2014070652 W JP 2014070652W WO 2015068440 A1 WO2015068440 A1 WO 2015068440A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- information processing
- user
- processing apparatus
- behavior
- unit
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/012—Head tracking input arrangements
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/017—Head mounted
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/16—Sound input; Sound output
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/16—Sound input; Sound output
- G06F3/165—Management of the audio stream, e.g. setting of volume, audio stream path
Definitions
- the present disclosure relates to an information processing apparatus, a control method, and a program.
- a mobile phone, a PDA, a smart phone or other portable terminal is used as an action recording device, and sensors that sense the situation and behavior of a user holding the device include an acceleration sensor, a gyro sensor, and a magnetic sensor.
- sensors that sense the situation and behavior of a user holding the device include an acceleration sensor, a gyro sensor, and a magnetic sensor.
- a sensor or a latitude / longitude acquisition sensor is used.
- the present disclosure proposes an information processing apparatus, a control method, and a program capable of estimating a user's action based on the head movement.
- the data detected by the detection unit that is mounted on the user's head and detects the movement of the head is compared with the action feature amount calculated based on the learning data, and the action of the user is compared.
- An information processing apparatus comprising: an estimation unit that estimates; and a control unit that performs control according to the behavior estimated by the estimation unit. Propose.
- the data detected by the detection unit that is mounted on the user's head and detects the movement of the head is compared with the action feature amount calculated based on the learning data, and the action of the user is compared.
- a control method is proposed that includes an estimation step and a step of performing control according to the estimated behavior.
- the computer compares the data detected by the detection unit that is mounted on the user's head and detects the movement of the head with the behavioral feature amount calculated based on the learning data, and the user A program is proposed for functioning as an estimation unit that estimates the behavior and a control unit that performs control according to the behavior estimated by the estimation unit.
- the behavior estimation system includes an information processing device 1 and a server 20 that are worn on the user's head.
- the information processing apparatus 1 is an eyeglass-type HMD (Head Mounted Display) as shown in FIG. 1, and is worn by a user by, for example, a pattern extending over both heads being placed on both ear shells.
- the information processing apparatus 1 (glasses type HMD), in a wearing state, immediately before the user's eyes, that is, a pair of display units 2a for the left eye and right eye, in a place where a lens in normal glasses is located, 2b is arranged.
- the display unit 2 and the display unit 2 are collectively referred to.
- the display unit 2 displays, for example, a captured image of the real space imaged by the imaging lens 4a, or displays an image so as to be superimposed on an object in the real space.
- the display unit 2 may be a transmissive type, and the information processing device 1 causes the display unit 2 to be in a through state, that is, a transparent or translucent state, so that the user can make the information processing device 1 like glasses. Even if it is always worn, it will not interfere with normal life.
- the information processing apparatus 1 is provided with a head sensor system such as a gravitational acceleration sensor 6a and a gyro sensor 6b for detecting the movement of the user's head while being worn by the user.
- the head sensor system is not limited to the gravitational acceleration sensor 6a and the gyro sensor 6b, and may be a geomagnetic sensor or the like, for example.
- the information processing apparatus 1 is equipped with a vibration unit 7 that vibrates the information processing apparatus 1.
- the imaging lens 4 a described above is arranged facing forward so as to capture an image with the direction visually recognized by the user as the subject direction.
- the information processing apparatus 1 is provided with a pair of earphone speakers 3a and 3b that can be inserted into the user's right ear hole and left ear hole when worn.
- microphones 5 a and 5 b that collect external sound are disposed on the right side of the display unit 2 for the right eye and on the left side of the display unit 2 for the left eye.
- the information processing apparatus 1 may be formed of a mounting unit that is generally a spectacle type or a head mounted type, and it is sufficient that at least the movement of the user's head can be detected.
- one display unit 2 may be provided corresponding to one eye.
- the imaging lens 4 a is disposed forward toward the right eye side, but may be disposed on the left eye side or may be disposed on both sides.
- the earphone speakers 3a and 3b may be provided only for wearing only one ear, not the left and right stereo speakers.
- the microphone may be one of the microphones 5a and 5b. Furthermore, a configuration without the microphones 5a and 5b or the earphone speakers 3a and 3b is also conceivable.
- the information processing apparatus 1 can communicate with the server 20 and transmit / receive data to / from the server 20.
- the user's action recognition is performed using a gravitational acceleration or a latitude / longitude acquisition sensor mounted on a mobile phone terminal, a smartphone, etc., but how to hold a mobile phone terminal, a smartphone, etc. Since the usage of the terminal is not uniformly determined, the types of user actions that can be estimated have been limited. For the same reason, a lot of sensor data is necessary to improve the estimation accuracy of the user's action.
- the types of actions that can be estimated (recognized) in the past include actions based mainly on speed data of movements such as “walking / running, moving cars, moving trains” and “watching the terminal. There are mainly two patterns of actions based on the coordinate data of the terminal such as “I am placing”.
- the usage method of the recognized action includes recording as a life log, reflection to an avatar application, and the like, and does not provide any feedback to the user according to the recognized action.
- the user's behavior is estimated based on the movement of the user's head using a head wearable device (for example, a glasses-type HMD) in which the possessed position is fixed.
- a head wearable device for example, a glasses-type HMD
- Actions that were difficult to (estimate) can also be estimated robustly. Specifically, for example, it is possible to estimate an action accompanied by a head movement such as “speaking with a person”, “mealing”, “sleeping”.
- the head wearable device is always worn uniformly on the head, and there is no difference in how and how to use the terminal depending on the user. By being different, it is possible to solve the problem of enormous learning data.
- FIG. 2 is a block diagram illustrating a basic configuration of the information processing apparatus 1 according to the present embodiment.
- the information processing apparatus 1 includes a main control unit 10, a display unit 2, an audio output unit 3, an imaging unit 4, an audio input unit 5, a head sensor system 6, a vibration unit 7, a communication unit 12, And a storage unit 14.
- the head sensor system 6 is various sensors for detecting the movement of the user's head while the information processing apparatus 1 is attached to the user.
- the head sensor system 6 is realized by, for example, a gravitational acceleration sensor 6a, a gyro sensor 6b, a geomagnetic sensor, or the like.
- the main control unit 10 includes, for example, a microcomputer including a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), a nonvolatile memory, and an interface unit. To control.
- a microcomputer including a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), a nonvolatile memory, and an interface unit. To control.
- the main control unit 10 functions as an action estimation unit 110 and an output control unit 120 as shown in FIG.
- the behavior estimation unit 110 compares the data detected by the head sensor system 6 that is mounted on the user's head and detects the movement of the head, with the behavior feature amount calculated based on the learning data, Estimate behavior.
- the behavior feature amount calculated based on the learning data may be acquired from the server 20 via the communication unit 12 or may be stored in the storage unit 14 in advance.
- the behavior feature amount is a value such as intensity, variance, or correlation analyzed based on learning data, for example.
- the learning data may be teacher data, or head movement detection data collected from each information processing apparatus 1 worn by a large number of users.
- FIG. 3 is a diagram showing detection data of the head movement during the action of placing the glass in a stationary state on the desk.
- the behavior estimation unit 110 of the main control unit 10 compares the behavior feature amount of each behavior with the behavior feature amount at the time of the action of placing the glass in a stationary state on the desk.
- the user's action can be estimated as "the action of placing a glass while standing at a desk”.
- FIG. 4 is a diagram showing detection data of head movement during chewing during meals.
- the behavior estimation unit 110 of the main control unit 10 compares the behavior feature amount of each behavior with the behavior feature amount at the time of mastication during a meal. It can be estimated as “chewing during a meal”. In chewing, periodic fine movement can be detected as shown in FIG.
- FIG. 5 is a diagram showing detection data of head movement during conversation.
- the behavior estimation unit 110 of the main control unit 10 compares the behavior feature amount of each behavior with the behavior feature amount of the head movement during the conversation.
- the behavior can be estimated as “conversation”.
- large and somewhat periodic motion can be detected as shown in FIG.
- FIG. 6 is a diagram showing detection data of a head movement during a nap.
- the behavior estimation unit 110 of the main control unit 10 compares the behavior feature amount of each behavior with the behavior feature amount of the head movement during dozing.
- the behavior can be estimated as “sleeping”.
- the absolute value of the sensor value stays at 200 or less although there is a slight shaking.
- FIG. 7 is a diagram showing detection data of head movement during walking.
- the behavior estimation unit 110 of the main control unit 10 compares the behavior feature amount of each behavior with the behavior feature amount of the head movement during walking.
- the behavior can be estimated as “walking”.
- walking as shown in FIG. 7, periodic values of both acceleration and angular velocity can be detected.
- the behavior that can be estimated by the behavior estimation unit 110 is not limited to the above-described example, and behaviors such as up and down stairs, running, and walking can also be estimated based on the movement of the head.
- the output control unit 120 realizes danger avoidance, communication activation, and the like by performing predetermined control according to the behavior estimated by the behavior estimation unit 110.
- the predetermined control is, for example, display content control, audio output control, vibration control, or activation control of a predetermined application.
- the communication unit 12 has a function of transmitting / receiving data by connecting to an external device by wireless / wired.
- the communication unit 12 according to the present embodiment is connected to the server 20 and receives an action feature amount calculated based on learning data, for example.
- the imaging unit 4 includes a lens system including an imaging lens 4a, a diaphragm, a zoom lens, a focus lens, and the like, a drive system that causes the lens system to perform a focus operation and a zoom operation, and imaging light obtained by the lens system. And a solid-state imaging device array that generates an imaging signal by photoelectric conversion.
- the solid-state imaging device array may be realized by, for example, a CCD (Charge Coupled Device) sensor array or a CMOS (Complementary Metal Oxide Semiconductor) sensor array.
- the imaging lens 4a is arranged facing forward so as to capture the direction viewed by the user as the subject direction when the information processing apparatus 1 is attached to the user. Thereby, the imaging part 4 can image the face of the other party who is talking with the user.
- the audio input unit 5 includes the microphones 5a and 5b shown in FIG. 1, and a microphone amplifier unit and an A / D converter for amplifying the audio signals obtained by the microphones 5a and 5b.
- the main control unit 10 enables voice input by the user by performing voice recognition on the voice data obtained by the voice input unit 5 after performing processing such as noise removal and sound source separation.
- the main control unit 10 performs recording such as noise removal and sound source separation on the audio data obtained by the audio input unit 5 and then stores the audio data in the storage unit 14 to enable recording.
- the display unit 2 displays image data according to control by the output control unit 120. Further, as described above, the display unit 2 may be a transmissive display device.
- the audio output unit 3 includes a pair of earphone speakers 3a and 3b shown in FIG. 1 and an amplifier circuit for the earphone speakers 3a and 3b. Moreover, the audio
- the vibration unit 7 has a function of vibrating the information processing apparatus 1 and is realized by, for example, an eccentric motor, a linear vibrator, a piezoelectric element, or the like.
- the vibration unit 7 can vibrate the information processing apparatus 1 according to control by the output control unit 120 to alert the user.
- the storage unit 14 stores programs for the main control unit 10 to execute various processes. Moreover, the memory
- FIG. 8 is a block diagram showing a basic configuration of the server 20 according to the present embodiment. As illustrated in FIG. 8, the server 20 includes a feature amount calculation unit 26, a communication unit 22, and a teacher data storage unit 24.
- the communication unit 22 is connected to an external device and has a function of transmitting / receiving data.
- the communication unit 22 according to the present embodiment is connected to the information processing apparatus 1 and transmits the feature amount of each action calculated by the feature amount calculation unit 26 described later.
- the teacher data storage unit 24 stores teacher data of each action that can be estimated by the movement of the head.
- the feature amount calculation unit 26 stores teacher data indicating the movement of the head of each action stored in the teacher data storage unit 24 or each of the information collected from the information processing apparatus 1 worn by many users via the communication unit 22. Based on learning data such as detection data of head movement at the time of action, a feature amount of head movement at the time of each action is calculated (learned).
- the method for calculating (learning) the feature amount is not particularly limited, but may be analyzed using, for example, a support vector machine (SVM) or a decision tree.
- SVM support vector machine
- the feature amount calculation unit 26 learns the feature amount by analyzing, for example, the strength, variance, correlation, or the like of head movement detection data (acceleration, angular velocity, etc.).
- FIG. 9 is a flowchart showing a feature amount learning (calculation) process according to the present embodiment.
- the feature amount calculation unit 26 of the server 20 acquires a sensor value (detection data) for each action as teacher data. Specifically, for example, the feature amount calculation unit 26 extracts sensor values (detection data) at the time of each action from the teacher data storage unit 24.
- step S106 the feature amount calculation unit 26 learns (calculates) the feature amount based on the sensor value at the time of each action that becomes teacher data.
- the feature amount calculation unit 26 outputs (transmits) the calculated feature amount to the information processing apparatus 1 worn on the user's head.
- the feature amount calculation unit 26 may store the calculated feature amount in a feature amount storage unit (not shown) included in the server 20. In any case, the feature amount calculation unit 26 transmits the feature amount at each action in response to a request from the information processing apparatus 1.
- FIG. 10 is a flowchart showing basic behavior estimation processing according to this embodiment.
- the main control unit 10 of the information processing apparatus 1 receives various sensor values from the head sensor system 6 mounted on the information processing apparatus 1 mounted on the user's head. (Ie head movement) is acquired.
- the behavior estimation unit 110 compares the acquired various sensor values with the feature amount of the teacher data, and estimates the user's behavior. That is, the behavior estimation unit 110 estimates a user's behavior according to whether or not the behavior of the user's head acquired in real time matches the feature amount based on which teacher data.
- the behavior estimation unit 110 learns the feature amount of each behavior using detection data obtained by detecting the movements of the heads of many other users (unsupervised learning). It is also possible to estimate the behavior based on the learned feature amount.
- the output control unit 120 performs predetermined output control according to the user's behavior estimated by the behavior estimation unit 110.
- the predetermined output control is, for example, at least one of display content control, audio output control, vibration control, and predetermined application activation control.
- the output control unit 120 performs predetermined output control in accordance with the estimated behavior, so that countermeasures for dangers that occur when wearing a head wearable device on a daily basis and living, Activation etc. can be realized.
- FIG. 11 shows an example of display control when the stairs are raised and lowered.
- the output control unit 120 normally displays a message from an acquaintance on the display screen 30 included in the field of view viewed by the user via the transmissive display unit 2 as shown in FIG. indicate.
- the output control unit 120 may change the display position as shown in the display screen 31 in FIG. Display control is performed so as not to hide the feet by changing upwards (or changing the display contents after simplifying them if necessary).
- the output control unit 120 reduces the size of characters and images displayed on the display unit 2 from the normal time (when walking) or displays the display when the action estimation unit 110 estimates “running”. By reducing the display area by simplifying the contents, display control is performed so as not to obstruct the user's field of view, thereby avoiding danger.
- the display control according to the behavior estimated in real time can avoid the danger that occurs when the user wears a transmissive head wearable device on a daily basis.
- FIG. 12 shows an example of a display screen included in the field of view of the user when the information processing apparatus 1 which is a glasses-type HMD equipped with the transmissive display unit 2 is mounted and talking with a person.
- the output control unit 120 performs control to reduce the amount of information to be displayed, as shown in the display screen 33 on the right side of FIG. Prevents obstruction of communication with the real world caused by transparent wearable devices.
- the output control unit 120 displays the number of new arrivals, the sender, the title, etc., as shown in the display screen 32 on the left side of FIG. In the middle, as shown in the display screen 33 on the right in FIG. 12, only the characters “new mail” are displayed. Alternatively, it is possible to prevent obstruction of real-world communication by displaying only icons or displaying nothing.
- FIG. 13 is a diagram for explaining a case where communication is activated by controlling the display position.
- the display position of information displayed on the display screen 34 included in the user's field of view is controlled to a fixed position.
- the display position of the information is not changed from the fixed position even if the face of the other party who is interacting moves. In this way, the information is always displayed at the fixed position, so that the user can always feel secure that the information can be obtained by directing his / her line of sight to the fixed position.
- a method of recognizing the position of the face of the conversation partner and controlling the display position of the display information so as not to overlap the face of the partner can be considered.
- the position of the opponent's face is recognized based on a captured image captured by the imaging unit 4 using, for example, an imaging lens 4a (see FIG. 1) provided outward in the information processing apparatus 1.
- the main control unit 10 starts imaging by the imaging unit 4, performs face recognition based on the captured image, and has a conversation partner. Know the position of the face.
- the output control unit 120 sets the display position of the information displayed on the display unit 2 to the face of the opponent that the user is viewing via the display unit 2 according to the position of the face recognized based on the captured image. Control so that they do not overlap. Further, the output control unit 120 changes the display position of the information in accordance with the change in the position of the opponent's face based on the captured images continuously captured by the imaging unit 4. As a result, as shown in the display screens 36 and 37 on the right in FIG. 13, even when the position of the face of the partner with whom the user is talking changes, the display position of the information is controlled so that it always does not overlap with the face of the partner. Therefore, activation of communication is realized.
- the display control when talking with a person who meets in real space has been described.
- the present embodiment is not limited to this.
- the user's attention target is other than a person. The same can be applied to the case.
- the output control unit 120 reduces the amount of information. Or control to display information while avoiding the screen of the target terminal or TV.
- the display control is performed so as not to obstruct the user's view.
- the convenience of the device 1 is improved.
- the output control unit 120 may perform control for starting a predetermined application according to the estimated behavior. Specifically, for example, when it is estimated by the behavior estimation unit 110 that “doing a nap”, the output control unit 120 activates an application for preventing a dozing. In accordance with such an application, the output control unit 120 can prompt the user to wake up by turning on the vibration unit 7 and applying vibration to the user, for example.
- the output control unit 120 turns off the vibration unit 7 and terminates the dozing prevention application.
- the application that is activated when the doze is estimated in advance may be a power saving application, a video application, or the like, in addition to the doze prevention application described above.
- the output control unit 120 When the power saving application is activated in response to the falling asleep, the output control unit 120 performs control such as turning off the power of the display unit 2 or shifting the power of the information processing apparatus 1 to the sleep mode, and the user falls asleep. Reduce power consumption while you are.
- the output control unit 120 takes a picture of the surroundings with the imaging unit 4 and records the surrounding sounds with the voice input unit 5, so that the user falls asleep. Record your surroundings while you are.
- Each of the applications described above is an example of an application that is activated when a dozing action is estimated, but a user may set in advance which application is activated.
- the estimated action is not limited to “sleeping”.
- the output control unit 120 may activate a preset health management application.
- the output control unit 120 can display a notification such as “attention to overeating” on the display unit 2 to warn the user.
- the behavior estimation unit 110 estimates the mastication behavior based on the movement of the head, and the output control unit 120 counts the number of chewing behaviors and counts the number of chewing behaviors in the health management application. Data may be output.
- the output control unit 120 can automatically activate a predetermined application in accordance with the estimated behavior, and can improve the convenience of the information processing apparatus 1.
- the information processing apparatus 1 is worn on the user's head, and can more robustly estimate the user's behavior based on the movement characteristic of the user's head.
- each information processing device 1 that is attached to the head does not have a different manner of possession (wearing) depending on the person, each information processing device 1 that is attached to a plurality of users has each action that becomes teacher data. Feature quantities based on movement can be used in common. In addition, this makes it possible to solve the problem of enormous learning data when the way of possession varies depending on the person such as a conventional smartphone.
- the information processing apparatus 1 automatically performs predetermined output control in accordance with the estimated behavior, and thus occurs when the head wearable device is worn on a daily basis. Measures against dangers and activation of communication can be realized.
- the teacher data storage unit 24 and the feature amount calculation unit 26 are provided in the server 20, and the behavior estimation unit 110 and the output control unit 120 are provided in the information processing apparatus 1. It is not limited to such an example. For example, all of these configurations may be provided in the server 20 or may be provided in the information processing apparatus 1.
- the display unit 2, the head sensor system 6, and the main control unit 10 are all provided in the wearable device (information processing apparatus 1) attached to the head. It is not limited to such an example.
- the display unit 2 and the head sensor system 6 are provided in a wearable device worn on the head, and the main control unit 10 is a smartphone, a mobile phone terminal, a PDA (Personal Digital) possessed by the user. Assistant), a tablet terminal, or the like.
- this technique can also take the following structures.
- An information processing apparatus comprising: (2) The information processing apparatus according to (1), wherein the estimation unit estimates the user's behavior by comparing the detected data with a behavior feature amount calculated based on teacher data in advance. (3) The information processing apparatus according to (1) or (2), wherein the control unit performs output control according to the estimated behavior.
- the information processing apparatus includes: The information processing apparatus according to any one of (1) to (6), further including a transmissive display unit disposed in front of the user's eyes when the information processing apparatus is attached to the head of the user. Information processing device.
- the control unit is a transmissive display arranged in front of the user's eyes when the information processing apparatus is a wearable device worn on the user's head according to the behavior estimated by the estimation unit
- the information processing apparatus according to any one of (1) to (7), wherein the display content of the unit is controlled.
- the information processing apparatus according to any one of (1) to (8), wherein the control unit performs control so as to start an application that is associated in advance with the action estimated by the estimation unit.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Audiology, Speech & Language Pathology (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Multimedia (AREA)
- Optics & Photonics (AREA)
- User Interface Of Digital Computer (AREA)
- Controls And Circuits For Display Device (AREA)
Abstract
Description
を提案する。
1.本開示の一実施形態による行動推定システムの概要
2.基本構成
2-1.情報処理装置の構成
2-2.サーバの構成
3.動作処理
3-1.特徴量学習処理
3-2.行動推定処理
4.出力制御
4-1.危険回避のための表示制御
4-2.コミュニケーションの活性化
4-3.アプリケーションへの応用
5.まとめ
まず、本開示の一実施形態による行動推定システムの概要について図1を参照して説明する。図1に示すように、本実施形態による行動推定システムは、ユーザの頭部に装着される情報処理装置1と、サーバ20とを含む。
ここで、従来の技術では、携帯電話端末やスマートフォン等に搭載された重力加速度や緯度経度取得センサ等を用いてユーザの行動認識を行っていたが、携帯電話端末やスマートフォン等の保持の仕方や端末の使い方は一様に定まらないので、推定できるユーザの行動の種類は限られていた。また、同様の理由により、ユーザの行動の推定精度を向上させるためには多くのセンサデータが必要であった。
<2-1.情報処理装置の構成>
図2は、本実施形態による情報処理装置1の基本構成を示すブロック図である。図2に示すように、情報処理装置1は、主制御部10、表示部2、音声出力部3、撮像部4、音声入力部5、頭部センサ系6、振動部7、通信部12、および記憶部14を有する。
頭部センサ系6は、情報処理装置1がユーザに装着された状態でユーザの頭部の動きを検知するための各種センサである。頭部センサ系6は、例えば重力加速度センサ6a、ジャイロセンサ6b、または地磁気センサ等により実現される。
主制御部10は、例えばCPU(Central Processing Unit)、ROM(Read Only Memory)、RAM(Random Access Memory)、不揮発性メモリ、インターフェース部を備えたマイクロコンピュータにより構成され、情報処理装置1の各構成を制御する。
通信部12は、無線/有線により外部装置と接続し、データの送受信を行う機能を有する。本実施形態による通信部12は、サーバ20と接続し、例えば学習データに基づいて算出された行動特徴量を受信する。
撮像部4は、撮像レンズ4a、絞り、ズームレンズ、及びフォーカスレンズ等により構成されるレンズ系と、レンズ系に対してフォーカス動作やズーム動作を行わせる駆動系と、レンズ系で得られる撮像光を光電変換して撮像信号を生成する固体撮像素子アレイ等とを有する。固体撮像素子アレイは、例えばCCD(Charge Coupled Device)センサアレイや、CMOS(Complementary Metal Oxide Semiconductor)センサアレイにより実現されてもよい。
音声入力部5は、図1に示したマイクロホン5a、5bと、そのマイクロホン5a、5bで得られた音声信号を増幅処理するマイクアンプ部やA/D変換器を有し、音声データを主制御部10に出力する。主制御部10では、音声入力部5で得られた音声データに対して、ノイズ除去、音源分離等の処理を行った上で音声認識を行うことで、ユーザによる音声入力を可能にする。また、主制御部10では、音声入力部5で得られた音声データに対して、ノイズ除去、音源分離等の処理を行った上で音声データを記憶部14に記憶させることで、録音を可能にする。
表示部2は、出力制御部120による制御に従って、画像データを表示する。また、表示部2は、上述したように、透過型の表示デバイスであってもよい。
音声出力部3は、図1に示した一対のイヤホンスピーカ3a、3bと、そのイヤホンスピーカ3a、3bに対するアンプ回路を有する。また、音声出力部3は、いわゆる骨伝導スピーカとして構成されてもよい。音声出力部3は、出力制御部120による制御に従って、音声信号データを出力(再生)する。
振動部7は、情報処理装置1を振動させる機能を有し、例えば偏心モータ、リニアバイブレータ、ピエゾ素子等により実現される。振動部7は、出力制御部120による制御に従って、情報処理装置1を振動させてユーザに注意を促すことが可能である。
記憶部14は、主制御部10が各種処理を実行するためのプログラム等を記憶する。また、記憶部14は、頭部の動きの学習データに基づいて算出された行動特徴量を記憶していてもよい。
図8は、本実施形態によるサーバ20の基本構成を示すブロック図である。図8に示すように、サーバ20は、特徴量算出部26、通信部22、および教師データ記憶部24を有する。
通信部22は、外部装置と接続し、データの送受信を行う機能を有する。本実施形態による通信部22は、情報処理装置1と接続し、後述する特徴量算出部26により算出された各行動の特徴量を送信する。
教師データ記憶部24は、頭部の動きにより推定し得る各行動の教師データを記憶する。
特徴量算出部26は、教師データ記憶部24に記憶されている各行動の頭部の動きを示す教師データ、または通信部22を介して多数のユーザが装着する情報処理装置1から収集した各行動時の頭部の動きの検知データ等の学習データに基づいて、各行動時の頭部の動きの特徴量を算出(学習)する。特徴量の算出(学習)方法については特に限定しないが、例えばSVM(Support Vector Machine)や決定木等が用いて分析してもよい。具体的には、特徴量算出部26は、例えば頭部の動きの検知データ(加速度や角速度等)の強度、分散、または相関等を分析して特徴量を学習する。
<3-1.特徴量学習処理>
図9は、本実施形態による特徴量学習(算出)処理を示すフローチャートである。図9に示すように、まず、ステップS103において、サーバ20の特徴量算出部26は、教師データとなる各行動時のセンサ値(検知データ)を取得する。具体的には、例えば特徴量算出部26は、教師データ記憶部24から各行動時のセンサ値(検知データ)を抽出する。
図10は、本実施形態による基本的な行動推定処理を示すフローチャートである。図10に示すように、まず、ステップS203において、情報処理装置1の主制御部10は、ユーザの頭部に装着される情報処理装置1に搭載された頭部センサ系6から、各種センサ値(すなわち頭部の動き)を取得する。
<4-1.危険回避のための表示制御>
例えば行動推定部110により「階段を昇降している」と推定された場合、出力制御部120は、ユーザの視界のうち足元を隠さないような表示制御を行い、危険回避を実現する。ここで、階段昇降時の表示制御の一例を図11に示す。
続いて、推定された行動に応じた出力制御によりコミュニケーションを活性化する場合の具体例について説明する。
また、本実施形態による出力制御部120は、推定された行動に応じて、所定のアプリケーションを起動する制御を行ってもよい。具体的には、例えば、行動推定部110により「居眠りをしている」と推定された場合、出力制御部120は、居眠り防止のアプリケーションを起動させる。かかるアプリケーションに従って、出力制御部120は、例えば振動部7をONし、ユーザに振動を与えることで起床を促すことができる。
上述したように、本開示の実施形態による情報処理装置1は、ユーザの頭部に装着され、ユーザの頭部に特徴的な動きに基づいてユーザの行動をよりロバストに推定することができる。
(1)
ユーザの頭部に装着され頭部の動きを検知する検知部により検知されたデータと、学習データに基づいて算出された行動特徴量とを比較し、前記ユーザの行動を推定する推定部と、
前記推定部により推定された行動に応じた制御を行う制御部と、
を備える、情報処理装置。
(2)
前記推定部は、前記検知されたデータと、予め教師データに基づいて算出された行動特徴量とを比較することで、前記ユーザの行動を推定する、前記(1)に記載の情報処理装置。
(3)
前記制御部は、前記推定された行動に応じた出力制御を行う、前記(1)または(2)に記載の情報処理装置。
(4)
前記出力制御は、表示内容の制御、音声出力制御、振動制御、または所定アプリケーションの起動制御の少なくともいずれかである、前記(3)に記載の情報処理装置。
(5)
前記行動特徴量は、学習データに基づいて分析された強度、分散、または相関の値である、前記(1)~(4)のいずれか1項に記載の情報処理装置。
(6)
前記検知部は、ジャイロセンサ、加速度センサ、または地磁気センサの少なくともいずれかである、前記(1)~(5)のいずれか1項に記載の情報処理装置。
(7)
前記情報処理装置は、
前記情報処理装置が前記ユーザの頭部に装着された際に、前記ユーザの眼前に配置される透過型の表示部をさらに備える、前記(1)~(6)のいずれか1項に記載の情報処理装置。
(8)
前記制御部は、前記推定部により推定された行動に応じて、前記情報処理装置が前記ユーザの頭部に装着されるウェアラブルデバイスである場合に、前記ユーザの眼前に配置される透過型の表示部の表示内容を制御する、前記(1)~(7)のいずれか1項に記載の情報処理装置。
(9)
前記制御部は、前記推定部により推定された行動に予め対応付けられたアプリケーションを起動するよう制御する、前記(1)~(8)のいずれか1項に記載の情報処理装置。
(10)
ユーザの頭部に装着され頭部の動きを検知する検知部により検知されたデータと、学習データに基づいて算出された行動特徴量とを比較し、前記ユーザの行動を推定するステップと、
前記推定された行動に応じた制御を行うステップと、
を含む、制御方法。
(11)
コンピュータを、
ユーザの頭部に装着され頭部の動きを検知する検知部により検知されたデータと、学習データに基づいて算出された行動特徴量とを比較し、前記ユーザの行動を推定する推定部と、
前記推定部により推定された行動に応じた制御を行う制御部と、
として機能させるための、プログラム。
2(2a、2b) 表示部
3 音声出力部
3a、3b イヤホンスピーカ
4 撮像部
4a 撮像レンズ
5 音声入力部
5a、5b マイクロホン
6 頭部センサ系
6a 重力加速度センサ
6b ジャイロセンサ
7 振動部
10 主制御部
110 行動推定部
120 出力制御部
12 通信部
14 記憶部
20 サーバ
22 通信部
24 教師データ記憶部
26 特徴量算出部
Claims (11)
- ユーザの頭部に装着され頭部の動きを検知する検知部により検知されたデータと、学習データに基づいて算出された行動特徴量とを比較し、前記ユーザの行動を推定する推定部と、
前記推定部により推定された行動に応じた制御を行う制御部と、
を備える、情報処理装置。 - 前記推定部は、前記検知されたデータと、予め教師データに基づいて算出された行動特徴量とを比較することで、前記ユーザの行動を推定する、請求項1に記載の情報処理装置。
- 前記制御部は、前記推定された行動に応じた出力制御を行う、請求項1に記載の情報処理装置。
- 前記出力制御は、表示内容の制御、音声出力制御、振動制御、または所定アプリケーションの起動制御の少なくともいずれかである、請求項3に記載の情報処理装置。
- 前記行動特徴量は、学習データに基づいて分析された強度、分散、または相関の値である、請求項1に記載の情報処理装置。
- 前記検知部は、ジャイロセンサ、加速度センサ、または地磁気センサの少なくともいずれかである、請求項1に記載の情報処理装置。
- 前記情報処理装置は、
前記情報処理装置が前記ユーザの頭部に装着された際に、前記ユーザの眼前に配置される透過型の表示部をさらに備える、請求項1に記載の情報処理装置。 - 前記制御部は、前記推定部により推定された行動に応じて、前記情報処理装置が前記ユーザの頭部に装着されるウェアラブルデバイスである場合に、前記ユーザの眼前に配置される透過型の表示部の表示内容を制御する、請求項1に記載の情報処理装置。
- 前記制御部は、前記推定部により推定された行動に予め対応付けられたアプリケーションを起動するよう制御する、請求項1に記載の情報処理装置。
- ユーザの頭部に装着され頭部の動きを検知する検知部により検知されたデータと、学習データに基づいて算出された行動特徴量とを比較し、前記ユーザの行動を推定するステップと、
前記推定された行動に応じた制御を行うステップと、
を含む、制御方法。 - コンピュータを、
ユーザの頭部に装着され頭部の動きを検知する検知部により検知されたデータと、学習データに基づいて算出された行動特徴量とを比較し、前記ユーザの行動を推定する推定部と、
前記推定部により推定された行動に応じた制御を行う制御部と、
として機能させるための、プログラム。
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/025,512 US10359839B2 (en) | 2013-11-08 | 2014-08-05 | Performing output control based on user behaviour |
JP2015546313A JP6402718B2 (ja) | 2013-11-08 | 2014-08-05 | 情報処理装置、制御方法およびプログラム |
EP14860594.2A EP3067782B1 (en) | 2013-11-08 | 2014-08-05 | Information processing apparatus, control method, and program |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2013231711 | 2013-11-08 | ||
JP2013-231711 | 2013-11-08 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2015068440A1 true WO2015068440A1 (ja) | 2015-05-14 |
Family
ID=53041225
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2014/070652 WO2015068440A1 (ja) | 2013-11-08 | 2014-08-05 | 情報処理装置、制御方法およびプログラム |
Country Status (4)
Country | Link |
---|---|
US (1) | US10359839B2 (ja) |
EP (1) | EP3067782B1 (ja) |
JP (1) | JP6402718B2 (ja) |
WO (1) | WO2015068440A1 (ja) |
Cited By (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2017058969A (ja) * | 2015-09-16 | 2017-03-23 | グリー株式会社 | 仮想画像表示プログラム、仮想画像表示装置及び仮想画像表示方法 |
JP2019101811A (ja) * | 2017-12-04 | 2019-06-24 | 富士通株式会社 | 処理プログラム、処理方法及び処理装置、並びに表示プログラム、表示方法及び表示制御装置 |
JPWO2018047433A1 (ja) * | 2016-09-08 | 2019-06-24 | ソニー株式会社 | 情報処理装置 |
WO2019244670A1 (ja) * | 2018-06-18 | 2019-12-26 | ソニー株式会社 | 情報処理装置、情報処理方法、及びプログラム |
JP2020077271A (ja) * | 2018-11-09 | 2020-05-21 | セイコーエプソン株式会社 | 表示装置、学習装置、及び、表示装置の制御方法 |
WO2021192673A1 (ja) * | 2020-03-27 | 2021-09-30 | ソニーグループ株式会社 | 表示装置及び表示方法 |
JP2022015543A (ja) * | 2020-07-09 | 2022-01-21 | アルプスアルパイン株式会社 | 監視装置 |
Families Citing this family (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10973440B1 (en) * | 2014-10-26 | 2021-04-13 | David Martin | Mobile control using gait velocity |
JP2017136142A (ja) * | 2016-02-02 | 2017-08-10 | セイコーエプソン株式会社 | 情報端末、動作評価システム、動作評価方法、動作評価プログラム、及び記録媒体 |
WO2018173404A1 (ja) | 2017-03-24 | 2018-09-27 | ソニー株式会社 | 情報処理装置および情報処理方法 |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2006333355A (ja) * | 2005-05-30 | 2006-12-07 | Toshiba Corp | 携帯情報処理装置およびその操作方法 |
JP2011197992A (ja) * | 2010-03-19 | 2011-10-06 | Fujitsu Ltd | モーション判定装置、モーション判定方法、およびコンピュータプログラム |
JP2013003649A (ja) | 2011-06-13 | 2013-01-07 | Sony Corp | 情報処理装置、情報処理方法およびコンピュータプログラム |
JP2013093705A (ja) * | 2011-10-25 | 2013-05-16 | Olympus Corp | 頭部装着型表示装置、プログラム及び画像処理システム |
Family Cites Families (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH0944297A (ja) * | 1995-07-25 | 1997-02-14 | Canon Inc | データ入力方法及びその装置 |
JPH09269889A (ja) * | 1996-04-02 | 1997-10-14 | Sharp Corp | 対話装置 |
JP5036177B2 (ja) * | 2005-12-12 | 2012-09-26 | オリンパス株式会社 | 情報表示装置 |
JP2008067219A (ja) * | 2006-09-08 | 2008-03-21 | Sony Corp | 撮像装置、撮像方法 |
JP5236423B2 (ja) * | 2008-10-17 | 2013-07-17 | Kddi株式会社 | 表示システム、表示方法およびプログラム |
JP5481890B2 (ja) * | 2009-03-12 | 2014-04-23 | ブラザー工業株式会社 | ヘッドマウントディスプレイ装置、画像制御方法および画像制御プログラム |
US20120206334A1 (en) * | 2010-02-28 | 2012-08-16 | Osterhout Group, Inc. | Ar glasses with event and user action capture device control of external applications |
US8964298B2 (en) * | 2010-02-28 | 2015-02-24 | Microsoft Corporation | Video display modification based on sensor input for a see-through near-to-eye display |
JP2012203128A (ja) * | 2011-03-24 | 2012-10-22 | Seiko Epson Corp | 頭部装着型表示装置および頭部装着型表示装置の制御方法 |
US8911087B2 (en) * | 2011-05-20 | 2014-12-16 | Eyefluence, Inc. | Systems and methods for measuring reactions of head, eyes, eyelids and pupils |
US8184070B1 (en) * | 2011-07-06 | 2012-05-22 | Google Inc. | Method and system for selecting a user interface for a wearable computing device |
US9445172B2 (en) * | 2012-08-02 | 2016-09-13 | Ronald Pong | Headphones with interactive display |
US9507426B2 (en) * | 2013-03-27 | 2016-11-29 | Google Inc. | Using the Z-axis in user interfaces for head mountable displays |
-
2014
- 2014-08-05 EP EP14860594.2A patent/EP3067782B1/en active Active
- 2014-08-05 JP JP2015546313A patent/JP6402718B2/ja active Active
- 2014-08-05 WO PCT/JP2014/070652 patent/WO2015068440A1/ja active Application Filing
- 2014-08-05 US US15/025,512 patent/US10359839B2/en active Active
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2006333355A (ja) * | 2005-05-30 | 2006-12-07 | Toshiba Corp | 携帯情報処理装置およびその操作方法 |
JP2011197992A (ja) * | 2010-03-19 | 2011-10-06 | Fujitsu Ltd | モーション判定装置、モーション判定方法、およびコンピュータプログラム |
JP2013003649A (ja) | 2011-06-13 | 2013-01-07 | Sony Corp | 情報処理装置、情報処理方法およびコンピュータプログラム |
JP2013093705A (ja) * | 2011-10-25 | 2013-05-16 | Olympus Corp | 頭部装着型表示装置、プログラム及び画像処理システム |
Non-Patent Citations (1)
Title |
---|
See also references of EP3067782A4 * |
Cited By (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2017058969A (ja) * | 2015-09-16 | 2017-03-23 | グリー株式会社 | 仮想画像表示プログラム、仮想画像表示装置及び仮想画像表示方法 |
JPWO2018047433A1 (ja) * | 2016-09-08 | 2019-06-24 | ソニー株式会社 | 情報処理装置 |
JP7074343B2 (ja) | 2016-09-08 | 2022-05-24 | ソニーグループ株式会社 | 情報処理装置 |
JP2019101811A (ja) * | 2017-12-04 | 2019-06-24 | 富士通株式会社 | 処理プログラム、処理方法及び処理装置、並びに表示プログラム、表示方法及び表示制御装置 |
WO2019244670A1 (ja) * | 2018-06-18 | 2019-12-26 | ソニー株式会社 | 情報処理装置、情報処理方法、及びプログラム |
JP2020077271A (ja) * | 2018-11-09 | 2020-05-21 | セイコーエプソン株式会社 | 表示装置、学習装置、及び、表示装置の制御方法 |
JP7271909B2 (ja) | 2018-11-09 | 2023-05-12 | セイコーエプソン株式会社 | 表示装置、及び、表示装置の制御方法 |
WO2021192673A1 (ja) * | 2020-03-27 | 2021-09-30 | ソニーグループ株式会社 | 表示装置及び表示方法 |
JP2022015543A (ja) * | 2020-07-09 | 2022-01-21 | アルプスアルパイン株式会社 | 監視装置 |
JP7460279B2 (ja) | 2020-07-09 | 2024-04-02 | アルプスアルパイン株式会社 | 監視装置 |
Also Published As
Publication number | Publication date |
---|---|
US10359839B2 (en) | 2019-07-23 |
JP6402718B2 (ja) | 2018-10-10 |
EP3067782A1 (en) | 2016-09-14 |
US20160231808A1 (en) | 2016-08-11 |
EP3067782B1 (en) | 2021-05-05 |
EP3067782A4 (en) | 2017-07-26 |
JPWO2015068440A1 (ja) | 2017-03-09 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP6402718B2 (ja) | 情報処理装置、制御方法およびプログラム | |
US11153431B2 (en) | Mobile terminal and method of operating the same | |
US10674056B2 (en) | Wearable apparatus and method for capturing image data using multiple image sensors | |
JP6574937B2 (ja) | 通信システム、制御方法、および記憶媒体 | |
KR102316327B1 (ko) | 가상 공간의 캡쳐 방법 및 그 전자장치 | |
JP6361649B2 (ja) | 情報処理装置、通知状態制御方法及びプログラム | |
WO2014156389A1 (ja) | 情報処理装置、提示状態制御方法及びプログラム | |
KR20160000741A (ko) | 글래스 타입 단말기 및 이의 제어방법 | |
JP6750697B2 (ja) | 情報処理装置、情報処理方法及びプログラム | |
US10778826B1 (en) | System to facilitate communication | |
US11929087B2 (en) | Systems and methods for selectively attenuating a voice | |
US20230005471A1 (en) | Responding to a user query based on captured images and audio | |
WO2016158003A1 (ja) | 情報処理装置、情報処理方法及びコンピュータプログラム | |
JP6969577B2 (ja) | 情報処理装置、情報処理方法、及びプログラム | |
KR20200031098A (ko) | 정보 처리 장치, 정보 처리 방법 및 프로그램 | |
WO2019044135A1 (ja) | 情報処理装置、情報処理方法、およびプログラム | |
US20220311979A1 (en) | Wearable apparatus for projecting information | |
CN114531582B (zh) | 一种增强现实功能控制方法和电子设备 | |
CN107872619B (zh) | 一种拍照处理方法、装置及设备 | |
WO2018155128A1 (ja) | 表示装置、制御装置および車両 | |
US20240205614A1 (en) | Integrated camera and hearing interface device | |
US20220417677A1 (en) | Audio feedback for correcting sound degradation |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 14860594 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 2015546313 Country of ref document: JP Kind code of ref document: A |
|
WWE | Wipo information: entry into national phase |
Ref document number: 15025512 Country of ref document: US |
|
REEP | Request for entry into the european phase |
Ref document number: 2014860594 Country of ref document: EP |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2014860594 Country of ref document: EP |
|
NENP | Non-entry into the national phase |
Ref country code: DE |