JP2019191947A - Information processing device - Google Patents

Abstract

To provide an input device capable of easily operating an image.SOLUTION: In a control unit 4A, a motion recognition unit 44 acquires distance information from a lidar 7, and recognizes the operation such as a click operation on an operation image C displayed on a display surface D on the basis of a secular change of a distance from the lidar 7 to a person's finger. Then, an input control unit 45 performs processing corresponding to the click operation or the like on the operation image C displayed on the display surface D on the basis of the recognition result obtained by the motion recognition unit 44.SELECTED DRAWING: Figure 5

Description

  The present invention relates to an information processing apparatus that performs predetermined processing.

  For example, Patent Document 1 describes that an image such as a movie is projected from a projector onto a ceiling surface in a vehicle interior. Patent Document 1 describes that an automatic driving system is operating normally on a ceiling surface together with an entertainment image.

JP, 2006-53622, A

  In an image such as a movie described in Patent Document 1, there are cases where operations such as fast-forward and pause are performed in order to view a desired scene, for example.

  Since Patent Document 1 is in a state where the vehicle is in a vehicle, the user's hand seated on the seat may not reach the ceiling surface, and the display surface D is not a touch panel in the first place, so the user performs a touch operation. Cannot be operated directly. In addition, although it can be operated from an operation means such as a button arranged within the reach of the user's hand, it is often inferior in terms of operability as compared with touch operation on the display surface.

  An example of a problem to be solved by the present invention is to easily manipulate an image.

  In order to solve the above problem, the invention according to claim 1 is capable of measuring a distance to the object by emitting an electromagnetic wave in the predetermined space and receiving the electromagnetic wave reflected by the object in the predetermined space. An acquisition unit that acquires distance information from a sensor, a recognition unit that recognizes a predetermined operation on the image based on a change over time of the distance from the sensor to a part of a person's body, and And a processing unit that performs a predetermined process on the image displayed on the display unit based on the recognition result.

  The invention according to claim 4 is an information processing method executed by an information processing apparatus that performs a predetermined process, and emits an electromagnetic wave in a predetermined space and receives an electromagnetic wave reflected by an object in the predetermined space. An acquisition step of acquiring distance information from a sensor capable of measuring a distance to the object, and a predetermined operation on the image based on a change over time of the distance from the sensor to a part of a person's body. And a processing step of performing a predetermined process on the image displayed on the display unit based on the recognition result of the recognition step.

  The invention according to claim 5 is characterized in that the information processing method according to claim 4 is executed by a computer.

It is a functional lineblock diagram of an acoustic system which has an information processor concerning the 1st example of the present invention. It is a figure in which persons, such as a user, are indicating the position with an index finger. 2 is a flowchart of the operation of the information processing apparatus shown in FIG. It is a figure of the other method of indicating a position. It is a functional block diagram of the input device which has an information processing apparatus concerning the 2nd Example of this invention. It is explanatory drawing of the detection of input operation with respect to the operation image displayed on the display surface. It is explanatory drawing of operation which performs the function concerning the highlighted icon. It is a flowchart of operation | movement of the information processing apparatus shown by FIG. It is explanatory drawing of other operation which performs the function concerning the highlighted icon.

  Hereinafter, an information processing apparatus according to an embodiment of the present invention will be described. An information processing apparatus according to an embodiment of the present invention includes a sensor capable of measuring a distance to an object by an acquisition unit emitting an electromagnetic wave in a predetermined space and receiving the electromagnetic wave reflected by the object in the predetermined space. The distance information is acquired, and the recognition unit recognizes a predetermined operation on the image based on a change with time of the distance from the sensor to a part of the human body. And a process part performs a predetermined process with respect to the image currently displayed on the display part based on the recognition result of a recognition part. This makes it easy to remotely input an image based on changes over time in the distance of a part of the body, such as a finger, without using a direct input device such as a touch panel or a push button. It can be carried out. That is, the image can be easily manipulated.

  A sensor may be installed at a position where the distance changes as a person's fingertip approaches. By doing so, it is possible to easily detect an action that makes it difficult to detect an image captured by a camera or the like, such as a part of the body such as a fingertip approaching or moving away from the sensor.

  The recognizing unit may recognize an operation on a specific position of the image based on a temporal change in the distance from the sensor to the person's fingertip. By doing so, for example, an operation of clicking a specific icon displayed on the image can be recognized.

  In addition, the information processing method according to an embodiment of the present invention can measure the distance to an object by emitting an electromagnetic wave in a predetermined space in the acquisition step and receiving the electromagnetic wave reflected by the object in the predetermined space. Distance information is acquired from the sensor, and a predetermined operation on the image is recognized based on a change over time in the distance from the sensor to a part of the human body in the recognition step. And a predetermined process is performed with respect to the image currently displayed on the display part based on the recognition result in a recognition process at a process process. This makes it easy to remotely input an image based on changes in the distance of a part of the body such as a finger, for example, without using a direct input device such as a touch panel or a push button. It can be carried out. That is, the image can be easily manipulated.

  The information processing method described above may be executed by a computer. In this way, using a computer, for example, based on the change over time of the distance of a part of the body such as a finger without using a direct input device such as a touch panel or a push button. Remote input can be performed easily. That is, the image can be easily manipulated.

  An acoustic system having an information processing apparatus according to a first embodiment of the present invention will be described with reference to FIGS. The acoustic system is mounted on a vehicle such as an automobile. A functional configuration of an acoustic system having an information processing apparatus is shown in FIG. The acoustic system includes an audio device 1, a delay adjustment unit 2 (2 R, 2 L), a speaker 3 (3 R, 3 L), a control unit 4, a microphone 5, and a lidar 7. In this embodiment, the sound system is mounted on a vehicle and controls sound in the vehicle interior as a predetermined space.

  The audio device 1 is a device that outputs an audio signal that is intended to be listened to by a user, such as a CD player, memory audio, radio tuner, or streaming distribution receiving device.

  In this embodiment, the delay adjustment unit 2 includes a left channel delay adjustment unit 2L and a right channel delay adjustment unit 2R. The delay adjustment unit 2L delays the left channel audio signal output from the audio device 1 to the left channel speaker 3L by a set delay time. The delay adjusting unit 2R delays the right channel audio signal output from the audio device 1 to the right channel speaker 3R by a set delay time.

  The control unit 4 includes a position recognition unit 41, a sound image control unit 42, and a voice recognition unit 43. The control part 4 is comprised with the microcomputer which has CPU, for example. The position recognition unit 41, the sound image control unit 42, and the voice recognition unit 43 may be realized as functions of a computer program executed by the CPU. The control unit 4 functions as an information processing apparatus according to the present embodiment.

  The position recognition unit 41 recognizes the position indicated by a part of the body (for example, a fingertip) based on the point cloud information output by the lidar 7. An example of position recognition by the position recognition unit 41 will be described with reference to FIG. FIG. 2 is a diagram in which a person such as a user indicates a position with the index finger IF.

  In FIG. 2, the user's hand H including the index finger IF is scanned by the lidar 7 and acquired as point cloud information. In the acquired point group information, the position recognition unit 41 recognizes the hand H and the index finger IF by well-known object recognition. When the index finger IF is recognized, the point cloud information includes information on the distance and direction from the recognized fingertip of the index finger IF to the lidar 7, so that the position of the fingertip in the vehicle compartment space can be recognized.

  Note that the present invention is not limited to point cloud information recognition alone. For example, a separate camera is provided, the index finger IF is recognized from an image captured by the camera, and the point cloud image is compared with the camera image to correspond to the index finger IF. A point cloud may be extracted.

  Based on the position of the fingertip recognized by the position recognition unit 41 and the voice information recognized by the voice recognition unit 43, the sound image control unit 42 is located in the vehicle interior formed by the sound emitted from the speakers 3L and 3R. The delay amounts of the delay adjustment units 2L and 2R are adjusted and set so that the sound image is localized at a designated position (specific position). That is, the control unit 4 (sound image control unit 42) functions as a localization control unit that localizes a sound image in the vehicle interior formed by sounds emitted from the speakers 3L and 3R to a specific position.

  Adjustment of the localization position in the sound image control unit 42 will be described. First, the distance between the fingertip position recognized by the position recognition unit 41 and the speakers 3L and 3R is obtained. Since the speakers 3L and 3R are fixed in the vehicle interior, the distance and direction from the speakers 3L and 3R to the rider 7 are known. Therefore, the distance and direction from the fingertip of the index finger IF to the lidar 7, the distance and direction from the speakers 3L, 3R to the lidar 7, and the distance from the speaker 3L to the lidar 7 and the distance from the speaker 3R to the lidar 7 are obtained. be able to. Then, the output from the speaker 3L (3R) having the shorter distance is delayed according to the difference in distance from the speaker 3R (3L) having the longer distance.

  The voice recognition unit 43 recognizes a voice signal (utterance content) collected by the microphone 5 using a known voice recognition algorithm, and outputs the recognized speech content to the sound image control unit 42 as voice information.

  The microphone 5 is installed at a position where the voice of the user in the passenger compartment such as a dashboard or a ceiling can be collected. The microphone 5 collects the content uttered by the user or the like as voice, converts it into an electrical signal, and outputs it as an audio signal to the control unit 4 (voice recognition unit 43).

  The lidar 7 is installed as a predetermined space at a position where the vehicle interior or the vehicle interior can be scanned. The lidar 7 is a sensor for recognizing an object existing in the periphery scanned by the lidar 7 and is also expressed as LiDAR (Light Detection And Ranging). The lidar 7 irradiates an electromagnetic wave such as a laser beam and discretely measures the direction and distance to the object existing in the scanning range by the reflected wave (reflected light) of the electromagnetic wave, and determines the position and shape of the object. This is a known sensor that recognizes as a three-dimensional point group. Therefore, the point cloud recognized by the lidar 7 is output as point cloud information as information relating to an object existing in the predetermined space.

  The number of riders 7 is not limited to one, and a plurality of riders 7 may be installed. For example, in the case of a vehicle, the number and installation positions of the lidar 7 may be determined as appropriate according to the size of the target passenger compartment space.

  Although not shown in FIG. 1, the acoustic system may include a display unit for confirming operations and various displays.

  Next, the operation (information processing method) of the control unit 4 configured as described above will be described with reference to the flowchart of FIG. Moreover, it can be set as an information processing program by comprising the flowchart shown in FIG. 3 as a program performed with CPU which the control part 4 has.

  First, in step S <b> 11, the position recognition unit 41 acquires point cloud information from the lidar 7. That is, the position recognition unit 41 functions as a first acquisition unit that acquires information about an object existing in a predetermined space.

  Next, in step S <b> 12, the position recognition unit 41 recognizes the position of the fingertip of the index finger IF based on the point cloud information acquired from the lidar 7 as described above. That is, the position recognizing unit 41 functions as a first recognizing unit that recognizes a position in a predetermined space indicated by a part of the body of a person existing in the predetermined space based on information on the object.

  Next, in step S <b> 13, the voice recognition unit 43 performs voice recognition based on the voice signal acquired from the microphone 5. Note that step S13 may be performed in parallel with step S12. That is, the voice recognition unit 43 functions as a second acquisition unit that acquires uttered speech and a second recognition unit that recognizes the content of the uttered speech.

  Next, in step S14, the sound image control unit 42, based on the position recognized in step S13 and the sound recognized in step S14, the delay adjustment unit 2L, so that the sound image is localized at the specified localization position. Adjust 2R respectively. This step is executed only when a specific sound such as “here” or “localize here” is input from the microphone 5, for example. That is, it is possible to reliably recognize the designation of the position where the sound image is localized not only by pointing with a finger but also by giving a voice instruction. That is, the sound image control unit 42 functions as a processing unit that performs processing based on the position recognized by the first recognition unit and the content recognized by the second recognition unit. Such specific audio content may be instructed to the user from a display unit (not shown).

  Next, in step S15, the sound image control unit 42 displays, for example, a message about whether or not readjustment is necessary on a display unit or the like (not shown) and inputs that readjustment is necessary (in the case of YES). Returns to step S11 to acquire the point cloud information again. On the other hand, when an input indicating that readjustment is unnecessary is made (in the case of NO), the flowchart ends.

  As is clear from the above description, step S11 functions as a first acquisition step, step S12 functions as a first recognition step, and step S14 functions as a processing step.

  In the above description, the localization position is specified by the operation pointed by the finger, but the present invention is not limited to this. For example, as shown in FIG. 4, a designation method may be used in which the finger is rotated so as to surround a range that is a finger. In this case, a range in which the finger is rotating is identified from the change in the finger position over time, and the center of the range may be regarded as the localization position.

  Further, the processing based on the position may be used not only for the localization position of the sound image but also for correcting the frequency characteristic at the specific position. That is, the equalizer or the like is adjusted so that the frequency characteristic at the position designated by the finger or the like is flat. By doing so, it is possible to listen to music or the like with a suitable frequency characteristic at a position designated by the user.

  In addition, for example, when a mobile directional speaker is used in a hands-free call, if the directionality is directed toward the driver and the direction is indicated by hand along with the voice "turn the voice away" It is possible to control the directivity of the speaker in that direction.

  Further, when the image displayed on the windshield, for example, by a projector or the like is reclined and turned upside down, the image can be moved to the interior ceiling surface by showing the ceiling by hand. Alternatively, it can be used for changing the spot position of a directional light for reading or the like. In specifying these positions, not only the index finger IF but also other fingers may be used. For example, when the hand is swung from the right to the left, the position where the hand is detected as the left end may be set as the designated position.

  According to the present embodiment, the control unit 4 acquires the point cloud information by the position recognition unit 41, and recognizes the position in the vehicle interior indicated by the person, for example, with the fingertip based on the point cloud information. Then, the sound image control unit 42 localizes the sound image at the position recognized by the position recognition unit 41. In this way, the sound image can be localized based on the position indicated by the finger or the like of the person existing in the vehicle compartment recognized by the position recognition unit 41. Therefore, since the position can be directly specified in a three-dimensional space such as a passenger compartment, the position where the sound image is localized can be specified accurately and easily.

  The position recognition unit 41 acquires point cloud information from the lidar 7. By doing so, the point cloud information can be easily acquired using the lidar 7. Further, since the distance to the object can be measured, it is easy to specify an accurate position.

  In addition, the voice recognition unit 43 that acquires the uttered voice and recognizes the content thereof is provided, and the sound image control unit 42 detects the sound image based on the position recognized by the position recognition unit 41 and the content recognized by the voice recognition unit 43. Is localized. By doing so, the processing can be executed in consideration of the content of the uttered voice, so that the position can be specified with higher accuracy.

  Next, an information processing apparatus according to a second embodiment of the present invention will be described with reference to FIGS. The same parts as those in the first embodiment described above are denoted by the same reference numerals and description thereof is omitted.

  FIG. 5 shows a functional configuration of an input device including a control unit 4A as an information processing device according to the present embodiment. The input device includes a control unit 4A, a lidar 7, and a projector 8. The lidar 7 is the same as that of the first embodiment.

  The projector 8 is installed, for example, in a vehicle interior, and displays images for various operations using a windshield, a ceiling surface, or the like as a display surface.

  The control unit 4A includes an operation recognition unit 44 and an input control unit 45. The motion recognition unit 44 recognizes a predetermined motion such as a gesture based on a change over time in the distance from the lidar 7 to a part of the user's body (for example, a fingertip) based on the point cloud information output by the lidar 7. .

  The input control unit 45 changes the image displayed by the projector 8 based on the operation recognized by the operation recognition unit 44. Further, the input control unit 45 outputs a command or the like to a device or the like outside the input device in order to execute a function corresponding to the operation based on the operation recognized by the operation recognition unit 44.

  An example of motion recognition by the motion recognition unit 44 and image change by the input control unit 45 will be described with reference to FIG. FIG. 6 is an explanatory diagram of detection of an input operation on the operation image C displayed on the display surface D by the projector 8.

  As shown in FIG. 6, the operation image C is displayed on the display surface D. In the operation image C, a plurality of icons I1 to I15 are arranged. Each of the icons I1 to I15 is operated to execute a function assigned to the operated icon. However, when the display surface D is a windshield, a ceiling surface, or the like, the user's hand may not reach, and since the display surface D is not a touch panel in the first place, a user sitting on the seat may perform a touch operation. Can not.

  Therefore, in this embodiment, the selection operation of the icons I1 to I15 shown in FIG. 6 is performed by recognizing the gesture performed by the user's hand H by the change over time of the point cloud information acquired by the rider 7. The image C is remotely operated. First, in the operation image C, the input control unit 45 highlights an icon at a predetermined default position, such as the center of the screen (icon I8 in FIG. 6), and selects it. For example, when the user wants to select the icon I7 on the left side of the icon I8, the hand H is shaken to the left. At this time, the movement recognizing unit 44 detects the temporal change in the position of the hand H in a plurality of frames of the point cloud information detected by the lidar 7, thereby determining the moving direction (the shake direction) of the hand H and making a gesture or the like. Can be recognized. Note that the hand H is recognized by object recognition from the point cloud information as in the first embodiment.

  When the motion recognition unit 44 recognizes a motion such as a gesture of shaking the hand H to the left, the input control unit 45 shifts the highlighted icon to the left (highlights the icon I7). In addition, although FIG. 6 demonstrated operation which highlights the icon on the left side, operation which highlights the icon on the right side and the upper and lower sides can be performed similarly.

  Next, an operation for executing the function assigned to the highlighted icon will be described with reference to FIG. In this embodiment, the operation assigned to the highlighted icon is performed by recognizing an operation corresponding to a click operation or a tap operation.

  When the click operation or the tap operation is performed, for example, the index finger IF is bent from the base of the finger toward the display surface D as shown in FIG. That is, it is common that the belly portion of the finger approaches the display surface D. In the present embodiment, this operation is detected by the lidar 7.

  In FIG. 7, the lidar 7 is assumed to be installed in the vicinity of the display surface D. That is, it is installed at a position where the distance between the rider 7 and the fingertip changes as the fingertip of the person approaches.

  In the case of FIG. 7, the motion recognition unit 44 clicks by detecting a change between the distance before bending the index finger IF (solid line d in FIG. 7) and the distance after bending the index finger IF (broken line d ′ in FIG. 7). Recognize an operation corresponding to an operation or a tap operation. That is, in the case of a click operation or a tap operation, since d> d ′ as a change with time, it is possible to recognize an operation corresponding to the click operation or the tap operation by detecting such a change. it can.

  Although the description of FIG. 7 is a click operation and a tap operation, a double click operation and a double tap operation can be recognized in the same manner. That is, when a change with time of d> d ′ is detected twice in succession, it may be recognized as a double click operation or a double tap operation.

  Next, the operation (information processing method) of the control unit 4A configured as described above will be described with reference to the flowchart of FIG. Moreover, it can be set as an information processing program by configuring the flowchart shown in FIG. 8 as a program executed by the CPU of the control unit 4A.

  First, in step S <b> 21, the motion recognition unit 44 acquires point cloud information from the lidar 7. That is, the motion recognition unit 44 obtains distance information from a sensor (lider 7) that can measure the distance to an object by emitting an electromagnetic wave in the predetermined space and receiving the electromagnetic wave reflected by the object in the predetermined space. Functions as an acquisition unit.

  Next, in step S22, as described above, the motion recognition unit 44 is performed by a hand H (finger) such as a change of an icon to be highlighted or a click operation based on the point cloud information acquired from the lidar 7. Recognize gestures and other actions. That is, the motion recognition unit 44 functions as a recognition unit that recognizes an operation on a specific position (a specific icon) of an image based on a change with time of a distance from a sensor to a part of a person's body (a fingertip).

  Next, in step S23, the input control unit 45 executes processing corresponding to the operation recognized by the operation recognition unit 44 in step S22. For example, the icon to be highlighted is changed in accordance with the waving operation as shown in FIG. Alternatively, the function assigned to the clicked icon is executed in response to a click operation or the like as shown in FIG. That is, the input control unit 45 functions as a processing unit that performs a predetermined process on the image displayed on the display unit based on the recognition result of the recognition unit.

  As is clear from the above description, step S21 functions as an acquisition process, step S22 functions as a recognition process, and step S23 functions as a processing process.

  According to the present embodiment, in the control unit 4A, the motion recognition unit 44 acquires the distance information from the rider 7, and the display surface D is displayed based on the change over time of the distance from the rider 7 to the person's finger. An operation such as a click operation on the operation image C is recognized. Then, the input control unit 45 performs processing corresponding to a click operation or the like on the operation image C displayed on the display surface D based on the recognition result of the motion recognition unit 44. By doing so, remote input can be performed on the operation image C based on the change over time of the distance of a part of the body such as a finger without using a direct input device such as a touch panel or a push button. The operation image C can be easily operated.

  In addition, since the operation can be performed by a gesture or the like, an intuitive operation can be performed on the image, and the operability is improved. In addition, since the motion can be detected in any posture within the detection range of the lidar 7, the operation can be performed regardless of the posture.

  A rider 7 is installed at a position where the distance changes as the fingertip of a person approaches. By doing so, it is possible to easily detect a click operation or the like that is difficult to detect in an image captured by a camera or the like. When a sensor is installed at a position where a person's fingertip approaches, if the sensor is a camera, there are many cases in which detection is difficult because there is little change in the image. On the other hand, in the case of the lidar 7, detection is performed based on a change in distance, so that detection is possible even when the lidar 7 is installed at a position where a fingertip of a person approaches. In addition, since the finger movement of the click operation may be small, the movement may not be captured in an image captured by a camera or the like. On the other hand, if the lidar 7 is used, information on the distance to the finger can be obtained, so that even a small movement can be detected.

  Further, when the lidar 7 is used, recognition is possible even in a dark state. Further, since the laser beam of the lidar 7 is transmitted, it is not erroneously recognized for what is reflected on the glass.

  In the second embodiment, the change of the icon to be selected, the click operation, and the like are recognized by the method shown in FIGS. 6 and 7, but the method shown in FIG. 9 may be used. FIG. 9 is an explanatory diagram of a method of selecting an icon to be selected on the display surface D by pointing with the index finger IF.

  The method shown in FIG. 9 first specifies the tip position IF1 and the root position IF2 of the index finger IF from the point cloud information acquired by the lidar 7. Then, a point P where the tip position IF1 and the root position IF2 are connected and the line L extended toward the display surface D and the display surface D intersect is recognized as the position indicated by the index finger IF. Since the display surface D is fixed in the passenger compartment, the relative position to the rider 7 can be obtained in advance. Accordingly, if the position coordinates of the four corners of the display surface D are obtained in advance with the position of the lidar 7 as the origin, it is possible to obtain where the point P is located in the display surface D from the relative positions of the point P and the four corners. Is possible. And what is necessary is just to highlight-display that the icon corresponding to the position in the display surface D of the point P was selected.

  After the icon to be selected is determined, the click operation or the like may be recognized by the method shown in FIG. 7, or the operation of bringing the index finger IF closer to the display surface D while the hand H is shown in FIG. It is good also as clicking operation.

  Further, the present invention is not limited to the above embodiment. That is, those skilled in the art can implement various modifications in accordance with conventionally known knowledge without departing from the scope of the present invention. Of course, such modifications are included in the scope of the present invention as long as the information processing apparatus of the present invention is provided.

4 control unit 41 position recognition unit (first acquisition unit, first recognition unit)
42 Sound image control unit (processing unit)
43 Voice recognition unit (second acquisition unit, second recognition unit)
4A control unit 44 motion recognition unit (acquisition unit, recognition unit)
45 Input control unit (processing unit)

Claims (5)

An acquisition unit that obtains distance information from a sensor that can measure the distance to the object by emitting an electromagnetic wave in the predetermined space and receiving the electromagnetic wave reflected by the object in the predetermined space;
A recognition unit for recognizing a predetermined operation on the image based on a change over time of the distance from the sensor to a part of a person's body;
A processing unit that performs a predetermined process on the image displayed on the display unit based on the recognition result of the recognition unit;
An information processing apparatus comprising:   The information processing apparatus according to claim 1, wherein the sensor is installed at a position where the distance changes as the fingertip of the person approaches.   The information processing apparatus according to claim 1, wherein the recognition unit recognizes an operation on a specific position of the image based on a change with time of a distance from the sensor to the fingertip of the person. An information processing method executed by an information processing apparatus that performs predetermined processing,
An acquisition step of obtaining distance information from a sensor capable of measuring the distance to the object by emitting an electromagnetic wave in the predetermined space and receiving the electromagnetic wave reflected by the object in the predetermined space;
A recognition step for recognizing a predetermined action on the image based on a change over time of the distance from the sensor to a part of a person's body;
A processing step of performing a predetermined process on the image displayed on the display unit based on the recognition result of the recognition step;
An information processing method comprising:   An information processing program that causes a computer to execute the information processing method according to claim 4.

Images