JP2015130582A - Image providing apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To detect a detection target in a wider range.SOLUTION: An image providing apparatus comprises: a measurement unit which acquires space information by using plural sensors; an overlap detector which detects an overlapping area having undergone overlapped measurement by the sensors; a target detector which detects a detection target on the basis of the information acquired by the sensors; a matching unit which matches the detection target detected in the overlapping area and detects an identical detection target which the sensors measure; an information acquisition unit which acquires information expressing the position and state of the identical detection target by the sensors; and a generator which generates an image on the basis of the information expressing the position and state of the identical detection target.

Description

  The present invention relates to a technique for providing an image.

  2. Description of the Related Art In recent years, a technology for realizing video communication represented by a video conference and digital signage (electronic signage) using a video display device and a sensor device has been put into practical use. In these technologies, technologies that enable more interactive information provision according to the state of the user such as position and posture have been proposed. For example, a technique for expressing motion parallax (for example, see Non-Patent Document 1), a technique for performing face recognition (for example, see Non-Patent Document 2), and the like have been proposed.

MoPaCo: Motion parallax video communication system using a monocular camera, Ryo Ishii, Shiro Ozawa, Kei Noto, Takafumi Mukai, Nobuhiko Matsuura, IEICE Technical Report, IEICE Technical Report MVE2010-65 (2010-10) Evolving interactive signage "Mitene", http://www.ar-mitene.jp/mob/index.php?site=mitene&ima=1947 SURF: Speeded Up Robust Features, Herbert Bay, Tinne Tuytelaars, Luc Van Gool, Computer vision ECCV2006,404-417,2006 High-Resolution Multi-View Projection Display With a Quantized-Diffusion-Angle Screen, Tohru Kawakami, Baku Katagiri, Takahiro Ishinabe, Tatsuo Uchida, JOURNAL OF DISPLAY TECHNOLOGY, VOL.8, NO.9, SEPTEMBER 2012

Video display devices and sensor devices are used in a one-to-one correspondence. Moreover, the area | region which each sensor apparatus measures is narrow. For this reason, each video display device has a problem that it is impossible to detect a user in a wide range.
In view of the above circumstances, an object of the present invention is to provide a video providing apparatus that can detect a detection target in a wider range.

  One aspect of the present invention is a measurement unit that acquires spatial information using a plurality of sensor devices, an overlap detection unit that detects overlapping regions that are measured by the plurality of sensor devices, and the plurality of sensors. A target detection unit that detects a detection target based on information acquired by the device, and a same detection target measured by the plurality of sensor devices by performing a matching process on the detection target detected in the overlapping region Based on the matching unit, the information acquisition unit that acquires information indicating the position and state of the same detection target by the plurality of sensor devices, and the information that indicates the position and state of the same detection target And a generation unit.

  One aspect of the present invention is the video providing apparatus described above, wherein the generation unit divides the video according to the position of the detection target, and displays the video having directivity for the divided video. This is a video providing device that outputs to a display device.

  One aspect of the present invention is the video providing apparatus described above, wherein the processing unit processes a motion parallax video according to a position of the detection target, and outputs the processed motion parallax video to the display device. , A video providing device.

  According to the present invention, it is possible to detect a detection target in a wider range.

It is a functional block diagram showing the function structure of the image | video provision apparatus 10 of embodiment. It is the figure which showed the specific example of a mode that an observer observes the image | video displayed on a some display apparatus. It is the figure which showed the specific example of the observation area | region of a display apparatus. It is the figure which showed the specific example of the detection area | region of a sensor apparatus. It is the figure which showed the specific example of a mode that an observer observes the image | video displayed on a multi-directional image | video output apparatus. It is the figure which showed the specific example of the observation area | region of a multidirectional video output device. It is the figure which showed the specific example of the detection area | region of the sensor apparatus in a multi-directional video output device. 3 is a functional block diagram illustrating details of a functional configuration of a matching unit 12. FIG. It is the figure which represented typically the display area and observation area | region of an image | video in a multi-directional video output device. It is a figure which shows the specific example of the image | video of the divided | segmented multi-directional image | video output apparatus. It is a flowchart showing the flow of a process of the image | video provision apparatus 10 of embodiment.

[Embodiment]
FIG. 1 is a functional block diagram illustrating a functional configuration of the video providing apparatus 10 according to the embodiment.
The video providing apparatus 10 acquires information measured by the sensor group 20 via an interface such as a LAN (Local Area Network) or a serial interface. The video providing device 10 outputs a video to the display device group 30 via an interface such as a LAN or a serial interface.
The sensor group 20 is a sensor device group including a plurality of sensor devices. A sensor device is a device that measures information by replacing information in space with an electrical signal. The sensor device included in the sensor group 20 is a device such as a camera, a microphone, or a depth sensor, for example. In FIG. 1, as an example, three sensor devices including a sensor device 21, a sensor device 22, and a sensor device 23 are shown as sensor devices included in the sensor group 20. The sensor group 20 outputs information acquired by measurement (hereinafter referred to as “measurement information”) to the video providing apparatus 10.

  The display device group 30 is a video display device group including a plurality of video display devices. The display device group 30 is a video display device such as a liquid crystal display, a CRT (Cathode Ray Tube) display, or a touch panel. In FIG. 1, as an example, three display devices including a display device 31, a display device 32, and a display device 33 are shown as display devices included in the display device group 30. The display device group 30 displays the video output by the video providing device 10.

FIG. 2 is a diagram illustrating a specific example of how an observer observes images displayed on a plurality of display devices.
In the example of FIG. 2, three observers (observer 1, observer 2 and observer 3) observe the images displayed on three display devices (display device A, display device B, and display device C). The state of doing is shown. In the example of FIG. 2, the sensor device A, the sensor device B, and the sensor device C are installed above the display device A, the display device B, and the display device C, respectively.

FIG. 3 is a diagram showing a specific example of the observation area of the display device.
The observation area is an area where an observer can observe an image displayed on the display device. In the example of FIG. 3, the observation area of the display device (display device A, display device B, and display device C) shown in the example of FIG. 2 is shown. Observation region A, observation region B, and observation region C represent the observation regions of display device A, display device B, and display device C, respectively. The observation area A, the observation area B, and the observation area C have areas that overlap with other observation areas depending on the position and orientation of the corresponding display device.

FIG. 4 is a diagram illustrating a specific example of the detection region of the sensor device.
In the example of FIG. 4, the detection area of the sensor device installed in the display device (display device A, display device B, and display device C) shown in the example of FIG. 2 is shown. Detection area A, detection area B, and detection area C represent detection areas of sensor device A, sensor device B, and sensor device C, respectively. The sensor device A, the sensor device B, and the sensor device C are installed such that the observation region A, the observation region B, and the observation region C are measurement target regions (hereinafter referred to as “detection regions”), respectively. The detection area A, the detection area B, and the detection area C have areas that overlap with other detection areas (hereinafter referred to as “overlap areas”) depending on the position and orientation of the corresponding display device. That is, the object located in the overlapping area is measured by a plurality of sensor devices that measure the overlapping area.

FIG. 5 is a diagram showing a specific example of how the observer observes the video displayed on the multidirectional video output device.
The multi-directional video output device is a video display device that displays a video having a plurality of directivities. An image having directivity is an image that is observed only in a specific area of the observation area of the display device. The multidirectional video output device is configured using, for example, a QDA (Quantized-Diffusion-Angle) screen described in Non-Patent Document 4.
In the example of FIG. 5, images having three types of directivity displayed on one multi-directional image output device (display device A) are displayed by four observers (observer 1, observer 2, observer 3 and It shows how the observer 4) observes. In the example of FIG. 5, the sensor device A, the sensor device B, and the sensor device C are installed above the display device A. Each sensor device is installed such that the observation region corresponding to the directivity of the image displayed on the display device A becomes the detection region.

FIG. 6 is a diagram showing a specific example of the observation area of the multidirectional video output device.
In the example of FIG. 6, the observation region of the display device A shown in the example of FIG. 5 is shown. The observation area A, the observation area B, and the observation area C represent areas in which three types of images having directivity are observed among the observation areas of the display device A.

FIG. 7 is a diagram illustrating a specific example of the detection region of the sensor device in the multidirectional video output device.
In the example of FIG. 7, the detection region of the sensor device installed in the display device A shown in the example of FIG. 5 is shown. Detection area A, detection area B, and detection area C represent detection areas of sensor device A, sensor device B, and sensor device C, respectively. The sensor device A, the sensor device B, and the sensor device C are installed such that the observation region A, the observation region B, and the observation region C are detection regions, respectively.

  Returning to the description of FIG. The video providing apparatus 10 includes a CPU (Central Processing Unit) connected via a bus, a memory, an auxiliary storage device, and the like, and executes a control program, thereby an input unit 11, a matching unit 12, a data processing unit 13, and an output unit. 14 functions as a device. Note that all or a part of the functions of the input unit 11, the matching unit 12, the data processing unit 13, and the output unit 14 are ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), and FPGA (Field Programmable Gate Array). It may be realized using hardware such as. The control program may be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, or a storage device such as a hard disk built in the computer system. The control program may be transmitted via a telecommunication line.

The input unit 11 is configured using a communication interface such as a LAN or a serial interface. The input unit 11 acquires measurement information from the sensor devices included in the sensor group 20. The input unit 11 outputs the acquired measurement information to the matching unit 12.
Based on the acquired measurement information, the matching unit 12 detects an object to be detected (hereinafter referred to as “detection target”) that exists in the measurement target space. Specifically, the detection target is an observer who observes an image displayed on the display device in the observation area. When the detection target is detected in the overlapping region, the matching unit 12 performs a detection target matching process among the plurality of sensor devices. When the same detection target is detected among a plurality of sensor devices by the matching process, the matching unit 12 acquires information indicating the position and state of the detection target. The matching unit 12 outputs information indicating the acquired position and state of the detection target to the data processing unit 13.

FIG. 8 is a functional block diagram showing details of the functional configuration of the matching unit 12.
The matching unit 12 includes an overlapping area detection unit 121, a detection target recognition unit 122, and an information acquisition unit 123.

  The overlapping area detection unit 121 acquires information (hereinafter referred to as “sensor information”) representing the position and orientation of the sensor devices included in the sensor group 20 in the three-dimensional space. The sensor information may be stored in advance in the video providing apparatus 10 as known information. Sensor information may be acquired using a sensor device such as a position sensor. The sensor information may be acquired based on the position and orientation of the display device in which each sensor device is installed.

  The overlapping region detection unit 121 calculates the position of the detection region of each sensor device in the three-dimensional space based on the acquired sensor information and the detection range of the sensor device. The detection range of the sensor device is given by information such as the angle of view of the camera and the sound collection angle of the microphone. The overlapping area detection unit 121 calculates an overlapping area in the detection area of each sensor device based on the position of the detection area of each sensor device in the three-dimensional space.

The detection target recognition unit 122 detects a detection target based on the measurement information, and acquires information indicating the position and state of the detection target.
For example, when an imaging device such as a camera is used as the sensor device included in the sensor group 20, the detection target recognition unit 122 calculates the feature amount of the image based on the image captured by each camera. The detection target recognition unit 122 detects the detection target based on the calculated feature amount of the image. When the detection target is detected in the overlap region, the detection target recognition unit 122 performs the detection target matching process based on the feature amount of the detection target and the position of the detection target in the overlap region. By this matching processing, the detection target recognition unit 122 identifies the same detection target from detection targets photographed by a plurality of cameras.

  For example, when a sound collection device such as a microphone is used as the sensor device included in the sensor group 20, a voiceprint may be used as a feature amount for detecting a detection target. The voiceprint is information representing the characteristics of the voice represented by the time when the voice is emitted and the frequency spectrum of the voice. The detection target recognition unit 122 performs detection target matching processing based on the acquired voiceprint. By this matching process, the detection target recognition unit 122 identifies the same detection target from the detection targets that have emitted the sound collected by the plurality of microphones.

  For example, when a measurement device such as a depth sensor is used as the sensor device included in the sensor group 20, a feature value representing the shape of an object may be used as a feature value for detecting a detection target. The detection target recognition unit 122 performs detection target matching processing based on the feature amount representing the shape of the detection target. By this matching process, the detection target recognition unit 122 identifies the same detection target from the detection targets measured by the plurality of depth sensors.

  In this manner, by identifying the same detection target from the detection targets measured by the plurality of sensor devices, the video providing device 10 can continue without losing sight of the detection target even when the detection target moves. Can be detected. Furthermore, even when the detection target moves outside the detection area of one sensor device, it is possible to continuously detect the detection target by another sensor device.

The information acquisition unit 123 acquires information indicating the state of the detection target detected in the overlapping area.
For example, when an imaging device such as a camera is used as the sensor device included in the sensor group 20, the information acquisition unit 123 detects the position of the face of the observer (detection target) located in the overlapping region. As a method for detecting the position of the face based on the image, the method described in Non-Patent Document 2 may be used. Based on the relationship between the detected face position and the camera position and orientation (sensor information), the information acquisition unit 123 obtains information representing the state of the observer in the three-dimensional space, such as the position, orientation, and posture. get. The relationship between the face position and the sensor information may be stored in advance in the video providing device 10 or may be learned by accumulating training data.
As described above, since the measurement information is acquired by the plurality of sensor devices for the detection target located in the overlapping region, it is possible to acquire the information representing the state of the detection target with high accuracy.

  Returning to the description of FIG. The data processing unit 13 generates an image corresponding to the position and state of the detection target based on the information representing the position and state of the detection target acquired by the matching unit 12. The data processing unit 13 outputs the generated video to the output unit 14.

Next, a specific example of an image corresponding to the state of a detection target generated by the data processing unit 13 when an image is provided to an observer using a plurality of display devices as in the example of FIG. 2 will be described. To do.
For example, consider a case where it is determined that a plurality of observers are observing the same display device based on information indicating the position and state of the observer (detection target). In this case, the data processing unit 13 may generate an image that is estimated to be of interest to many observers who observe the same display device.

Further, the data processing unit 13 may calculate a centroid position based on position information of a plurality of observers, and generate an image having the centroid position as an observation position. The generated video is, for example, a video such as a three-dimensional CG model.
The data processing unit 13 may generate an image provided for each of a plurality of observers. The data processing unit 13 may divide the display area of the display device in accordance with the number of observers, and display images for each of the plurality of observers in the divided display areas. The display area may be equally divided in order from the display area close to the observer based on the position information of the observer.

Next, when an image is provided to an observer using a multidirectional image output device as in the example of FIG. 5, a specific example of an image corresponding to the position of a detection target generated by the data processing unit 13 is provided. explain.
For example, a case where a QDA screen is used in a multidirectional video output device will be described. On the QDA screen, the observation area of each image having directivity is fixed in advance. Therefore, the user cannot dynamically change the observation area. In addition, between the observation areas of the plurality of videos displayed on the QDA screen, there is an area inappropriate for observation in which the videos of the respective observation areas are mixed (hereinafter referred to as “observation inappropriate area”).

  When the observer is located in the observation inappropriate area, the data processing unit 13 divides the display area of each image having directivity according to the position of the observer. The data processing unit 13 generates two images having different directivities, which are images that can be observed by being observed in a mixed state in the observation inappropriate region. The data processing unit 13 outputs the generated two videos to two display areas corresponding to the observation inappropriate area among the divided display areas.

By the above processing of the data processing unit 13, the video providing device 10 can provide an observable video to an observer located in the observation inappropriate region.
Next, the content of the processing of the data processing unit 13 described above will be described using a specific example.

FIG. 9 is a diagram schematically showing a video display area and an observation area in the multidirectional video output apparatus.
In the example of FIG. 9, the multidirectional video output device displays three videos having different directivities. Three videos having different directivities are displayed in the display area A, the display area B, and the display area C, respectively. The observation areas of the images displayed on the display area A, the display area B, and the display area C are the observation area A, the observation area B, and the observation area C, respectively. The observation inappropriate region AB is an observation inappropriate region between the observation region A and the observation region B. The observation inappropriate region BC is an observation inappropriate region between the observation region B and the observation region C.

FIG. 10 is a diagram illustrating a specific example of the video of the divided multidirectional video output device.
In the example of FIG. 10, a specific example of the video divided by the data processing unit 13 in the multidirectional video output device having the display area and the observation area shown in the example of FIG. 9 is shown.
When the observer (detection target) of the multidirectional video output device is located as in the example of FIG. 10, the data processing unit 13 classifies the observer into a predetermined number of groups based on the position information of the observer. To do. The predetermined number may be preset as a default value. The predetermined number may be dynamically determined based on arbitrary information such as the size of the display area and the number of observers. In the example of FIG. 9, the observer is classified into four groups, group 1, group 2, group 3, and group 4.

  The data processing unit 13 determines the division position of the display area of each image having directivity based on the position information of each group. In the example of FIG. 9, the division positions B and C are determined based on the position of the group 3 located in the viewing inappropriate area BC. For the position information of each group, an average of the positions of the observers belonging to each group may be used. Further, the position of the center of gravity of the observer belonging to each group may be used as the position information of each group. Since the division position of the display area is determined as the division position B and the division position C, the video 1 is displayed in the display area A. In the display area B, the video 2 and the video 3B are displayed with the division position B as a boundary. In the display area C, the video 3C and the video 4 are displayed with the division position C as a boundary.

  The data processing unit 13 generates a video displayed in the divided areas of the display areas. In the example of FIG. 9, the data processing unit 13 generates video 1, video 2, video 3B, video 3C, and video 4. The data processing unit 13 generates an image 3B and an image 3C that can be observed by being observed in a mixed state in the observation inappropriate region BC.

  The data processing unit 13 outputs the generated video to the output unit 14. In the example of FIG. 10, the data processing unit 13 outputs the video 1 to the output unit 14 as a video displayed in the display area A. The data processing unit 13 outputs the video 2 and the video 3B to the output unit 14 as video displayed in the display area B. The data processing unit 13 outputs the video 3C and the video 4 to the output unit 14 as video displayed in the display area C.

  As described above, when the multidirectional video output device is used as the display device, the video providing device 10 can observe the video by dividing the display area of each video having directivity according to the position of the observer (detection target). It is possible to provide an observable image for an observer located in the inappropriate area.

  Returning to the description of FIG. The output unit 14 outputs the video output from the data processing unit 13 to display devices included in the display device group 30.

FIG. 11 is a flowchart illustrating a process flow of the video providing apparatus 10 according to the embodiment.
First, the video providing device 10 acquires measurement information from the sensor devices included in the sensor group 20 (step S101). The video providing device 10 acquires information (sensor information) indicating the position and orientation of each sensor device in the three-dimensional space (step S102). The video providing device 10 calculates the position of the detection area of each sensor device in the three-dimensional space based on the acquired sensor information (step S103).

  The video providing device 10 calculates a region (overlapping region) that is measured by each sensor device in an overlapping manner based on the position of the detection region of each sensor device in the three-dimensional space (step S104). The video providing apparatus 10 calculates the feature amount of the detection area based on the measurement information (Step S105). The video providing apparatus 10 detects a detection target (observer) based on the calculated feature amount of the detection region (step S106).

  The video providing apparatus 10 determines whether or not the detection target detected in step S106 is located in the overlapping area (step S107). When the detection target is not located in the overlapping area (step S107—NO), the process proceeds to step S110, and the video providing apparatus 10 generates a video to be provided to the observer (detection target). On the other hand, when the detection target is located in the overlap region (step S107—YES), the video providing apparatus 10 detects the detection target located in the overlap region based on the feature amount and the position information of the detection target located in the overlap region. A matching process is performed (step S108).

  The video providing apparatus 10 acquires information indicating the state of the detection target (observer) located in the overlapping area (step S109). The information indicating the state is information indicating the state of the detection target such as the orientation and the posture. The video providing apparatus 10 generates a video corresponding to the position and state of the detection target (observer) based on information representing the position and state of the detection target (observer) (step S110). The video providing device 10 outputs the generated video to the display devices included in the display device group 30 (Step S111).

  The video providing apparatus 10 according to the embodiment configured as described above performs detection target matching processing based on the feature amount of the overlapping region. By this matching processing, the video providing apparatus 10 can detect detection targets in a wider detection range by combining the detection ranges of a plurality of sensor devices.

  In addition, the video providing apparatus 10 according to the embodiment detects the same detection target located in the overlapping area by using a plurality of sensor devices. Therefore, the video providing apparatus 10 can accurately detect the detection target located in the overlapping area. As a result, the video providing apparatus 10 can provide a video corresponding to the position and state of the observer to the observer (detection target) located in the overlapping region.

  These functions can also be applied when a multidirectional video output device is used for the display device group 30. The video providing apparatus 10 according to the embodiment generates each video having directivity according to the position of the observer. With this function, the video providing apparatus 10 can provide a video that can be observed even in an observation inappropriate region. In addition, the video providing device 10 is configured using a multidirectional video output device, so that motion parallax video of the same object can be simultaneously provided to a plurality of observers.

<Modification>
The detection target need not be limited to an observer who is observing an image displayed on the display device in the observation area. For example, the detection target may be all persons located in the observation area, may be a person satisfying a predetermined condition among persons located in the observation area, or may be another person or object. Also good.

You may make it implement | achieve the image provision apparatus in embodiment mentioned above with a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system serving as a server or a client in that case may be included and a program held for a certain period of time. Further, the program may be a program for realizing a part of the above-described functions, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system. You may implement | achieve using programmable logic devices, such as FPGA (Field Programmable Gate Array).
The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

DESCRIPTION OF SYMBOLS 10 ... Video provision apparatus, 11 ... Input part, 12 ... Matching part, 13 ... Data processing part, 14 ... Output part, 141 ... Overlapping area | region detection part, 142 ... Detection object recognition part, 143 ... Detection object information acquisition part, 20 ... Sensor group, 21, 22, 23 ... Sensor device, 30 ... Display device group, 31, 32, 33 ... Display device

Claims (3)

A measurement unit that obtains spatial information using a plurality of sensor devices;
An overlap detection unit for detecting an overlap region measured in an overlapping manner by the plurality of sensor devices;
A target detection unit that detects a detection target based on information acquired by the plurality of sensor devices;
A matching unit that detects the same detection target measured by the plurality of sensor devices by performing a matching process on the detection target detected in the overlapping region;
An information acquisition unit for acquiring information representing the position and state of the same detection target by the plurality of sensor devices;
A generating unit that generates a video based on information indicating the position and state of the same detection target;
A video providing apparatus comprising:   The video providing apparatus according to claim 1, wherein the generation unit divides the video according to the position of the detection target, and outputs the divided video to a display device that displays a video having directivity.   The video providing apparatus according to claim 2, wherein the processing unit processes a motion parallax video according to a position of the detection target, and outputs the processed motion parallax video to the display device.

Images