JP2021125789A - Image processing device, image processing system, image processing method, and computer program - Google Patents

Abstract

To provide an image processing device capable of generating an image related to a user without forcing an excessive burden on the user.SOLUTION: The image processing device comprises: an acquisition unit which acquires a plurality of images captured by a plurality of cameras; an orientation detection unit which detects the orientation of an object person; a position identification unit which identifies the position of the object person; and an image generation unit which on the basis of the detected orientation of the object person and the identified position of the object person, generates an image related to the object person from the images.SELECTED DRAWING: Figure 3

Description

The present disclosure relates to a video processing apparatus, a video processing system, a video processing method, and a computer program.

Conventionally, a hands-free video phone capable of making a videophone call by hand has been proposed (see, for example, Non-Patent Document 1).

The hands-free videophone described in Non-Patent Document 1 makes a videophone call using a glasses-type wearable terminal device. Five ultra-wide-angle cameras are arranged in the terminal device toward the face, and a face image is generated by synthesizing partial images of the face taken by those cameras and transmitted to the terminal device on the other side. ing.

"[CEATEC JAPAN 2012] NTT DoCoMo booth, reference exhibit of" hands-free video phone "that allows you to make video calls empty-handed", [online], October 3, 2012, Livedoor News, Gadget Tsushin, [Reiwa 1st year 11th Search on 22nd of March], Internet <URL: https: //getnews.jp/archives/257957> "Basic Principle of Line-of-sight Detection Technology", [online], April 23, 2013, Fujitsu Laboratories, [Search on January 6, 2nd year of Reiwa], Internet <URL: https: // www. Fujitsu. com / jp / group / labs / resources / tech / techguide / list / eye-movements / p03. html> Kazuki Okami, Kota Takeuchi, Ai Isogai, Hideaki Kizen, "Free-viewpoint video synthesis technology that realizes a new video viewing experience", NTT Technology Journal, October 2017, p. 10-14

However, the conventional eyeglass-type wearable terminal device is fixed at a position where the user's face can be photographed. Therefore, it is not possible to transmit an image other than the face of the user wearing the terminal device to the terminal device on the other side.

Further, it is not realistic for the user to wear a terminal device equipped with a plurality of cameras because the weight and the like impose a burden on the user. In addition, since it is necessary to shoot a plurality of images, it is difficult to drive the battery for a long time, and there is a concern about heat generation.

The present disclosure has been made in view of such circumstances, and is a video processing device, a video processing system, a video processing method, and a video processing method capable of generating a video related to the user without imposing an excessive burden on the user. The purpose is to provide computer programs.

The video processing device according to one aspect of the present disclosure is an orientation of a target person based on an acquisition unit that acquires a plurality of images captured by a plurality of cameras and at least one of the acquired plurality of images. Based on the orientation detection unit for detecting the target person, the position specifying unit for specifying the position of the target person, the detected orientation of the target person, and the identified position of the target person, the target person is selected from the plurality of images. It is provided with a video generation unit that generates video related to.

The image processing system according to another embodiment of the present disclosure includes a plurality of cameras and the above-mentioned image processing device that generates an image based on a plurality of images taken by the plurality of cameras.

The image processing method according to another embodiment of the present disclosure is based on a step of acquiring a plurality of images captured by a plurality of cameras and at least one of the acquired images of the subject. Based on the step of detecting the orientation, the step of specifying the position of the target person, the detected orientation of the target person, and the identified position of the target person, the plurality of images are related to the target person. Includes steps to generate video.

A computer program according to another embodiment of the present disclosure is based on a computer based on an acquisition unit that acquires a plurality of images captured by a plurality of cameras, and at least one of the acquired plurality of images. The plurality of images are based on a direction detecting unit that detects the orientation of the target person, a position specifying unit that specifies the position of the target person, the detected orientation of the target person, and the identified position of the target person. To function as an image generation unit that generates an image related to the target person.

Needless to say, the computer program can be distributed via a computer-readable non-temporary recording medium such as a CD-ROM (Compact Disc-Read Only Memory) or a communication network such as the Internet. Further, the present disclosure can also be realized as a semiconductor integrated circuit that realizes a part or all of the video processing apparatus.

According to the present disclosure, it is possible to generate a video related to a user without imposing an excessive burden on the user.

FIG. 1 is a diagram showing a configuration of a video processing system according to the first embodiment of the present disclosure. FIG. 2 is a block diagram showing a functional configuration of the video processing device. FIG. 3 is a block diagram showing a detailed configuration of a video processing unit included in the video processing device. FIG. 4 is a diagram for explaining an example of the target person image selection process by the target person image selection unit. FIG. 5 is a diagram for explaining an example of the ambient image selection process by the ambient image selection unit. FIG. 6 is a diagram for explaining an example of the ambient image selection process by the ambient image selection unit. FIG. 7 is a diagram for explaining an example of the ambient image selection process by the ambient image selection unit. FIG. 8 is a flowchart showing an example of a processing procedure of the video processing apparatus. FIG. 9 is a flowchart showing details of the self-shooting mode video processing (step S4 of FIG. 8). FIG. 10 is a flowchart showing details of the viewpoint sharing mode video processing (step S6 of FIG. 8). FIG. 11 is a block diagram showing a detailed configuration of a video processing unit included in the video processing device. FIG. 12 is a diagram for explaining an example of a self-portrait video generation process by the self-portrait video generation unit. FIG. 13 is a diagram for explaining an example of the generation process of the target person's viewpoint image by the target person's viewpoint image generation unit. FIG. 14 is a flowchart showing details of the self-shooting mode video processing (step S4 of FIG. 8). FIG. 15 is a flowchart showing details of the viewpoint sharing mode video processing (step S6 of FIG. 8).

[Summary of Embodiments of the present disclosure]
First, the outlines of the embodiments of the present disclosure will be listed and described.

(1) The image processing apparatus according to the embodiment of the present disclosure includes an acquisition unit that acquires a plurality of images taken by a plurality of cameras, an orientation detection unit that detects the orientation of the target person, and a position of the target person. A position specifying unit for specifying the target person, and an image generating unit for generating an image related to the target person from the plurality of images based on the detected orientation of the target person and the identified position of the target person. ..

According to this configuration, for example, from a plurality of images acquired from a plurality of cameras via a network, images related to the target person can be generated based on the orientation and position of the target person. Therefore, it is possible to generate an image related to the target person without the target person wearing a dedicated terminal device on which the camera is arranged. As a result, it is possible to generate a video related to the user without imposing an excessive burden on the user.

(2) Preferably, the plurality of cameras include at least one of an outdoor-installed camera, an indoor-installed camera, and an in-vehicle camera.

According to this configuration, it is possible to generate an image related to the target person by using a camera existing around the target person without using a dedicated camera. That is, the target person can generate an image related to the target person without having to worry about the remaining battery level of the camera, heat generation, and the like.

(3) More preferably, the orientation detection unit detects the orientation of the target person based on at least one of the acquired plurality of images.

According to this configuration, the orientation of the target person can be detected based on the image. Therefore, the subject does not need to carry a device for detecting the orientation of the subject.

(4) More preferably, the image generation unit generates an image obtained by shooting the target person in the direction of the target person from a position facing the target person as an image related to the target person.

According to this configuration, it is possible to generate a so-called self-portrait image in which the target person is photographed from a position facing the target person.

(5) Further, the image generation unit may extract an image taken from the direction closest to the direction of the target person from the plurality of images as an image related to the target person.

According to this configuration, it is possible to generate a so-called self-portrait image from a plurality of images in which the target person is photographed from the direction closest to the target person's direction. Therefore, it is possible to generate a self-portrait image at high speed.

(6) Further, the image generation unit detects an image including the image of the target person from the plurality of images, and synthesizes the detected image including the image of the target person to make the target person. An image obtained by photographing the target person in the direction of the target person from the opposite position may be generated.

According to this configuration, a so-called self-portrait image can be generated by synthesizing an image including an image of a target person. Therefore, even if the image obtained by shooting the target person in the direction of the target person cannot be obtained from the camera, it is possible to generate the image obtained by shooting the target person in the direction.

(7) Further, the image generation unit may generate an image of the surroundings of the target person in the direction of the target person from the position of the target person as an image related to the target person.

According to this configuration, for example, it is possible to generate an image of an object in front of the line of sight of the object, that is, an image of the same object as the object is seeing.

(8) Further, the image generation unit may select an image of the surroundings of the target person from the position of the target person in the direction of the target person from the plurality of images.

According to this configuration, it is possible to select, for example, a target image in front of the target person's line of sight from a plurality of images. Therefore, it is possible to generate an image of the surroundings from the target person at high speed.

(9) Further, the image generation unit may generate an image in which the surroundings of the target person are viewed from the position of the target person in the direction of the target person by synthesizing the plurality of images.

According to this configuration, by synthesizing a plurality of images, for example, it is possible to generate an image of an object in front of the line of sight of the object. Therefore, even if it is not possible to generate such an image from one image, it is possible to generate the same image of the target as the subject is viewing.

(10) Further, the orientation of the subject may include at least one of the orientation of the subject's line of sight, the orientation of the subject's face, and the orientation of the subject's body.

According to this configuration, it is possible to generate an image of the target person viewed from the direction of the target person's line of sight, face or body, or an image of the surroundings viewed from the position of the target person in that direction.

(11) Further, the video generation unit may further perform privacy protection processing on an image of a person other than the target person included in the video related to the target person.

According to this configuration, it is possible to generate an image in which the privacy of a person other than the target person is protected.

(12) Further, the above-mentioned video processing device may further include a video transmission unit that transmits the video generated by the video generation unit and related to the target person to the terminal device.

According to this configuration, the generated video can be transmitted to the terminal device. Therefore, the subject can make a video call or have a video conference with an interlocutor who uses the terminal device without being aware of the camera.

(13) The image processing system according to another embodiment of the present disclosure includes a plurality of cameras and the above-mentioned image processing device that generates an image based on a plurality of images taken by the plurality of cameras.

This configuration has the same configuration as the above-mentioned video processing apparatus. Therefore, according to this configuration, it is possible to obtain the same operations and effects as those of the above-mentioned video processing apparatus.

(14) The image processing method according to another embodiment of the present disclosure specifies a step of acquiring a plurality of images taken by a plurality of cameras, a step of detecting the orientation of the target person, and a position of the target person. A step of generating an image related to the object from the plurality of images based on the detected orientation of the object and the identified position of the object.

This configuration includes a step corresponding to a characteristic processing unit included in the above-mentioned video processing apparatus. Therefore, according to this configuration, it is possible to obtain the same operations and effects as those of the above-mentioned video processing apparatus.

(15) A computer program according to another embodiment of the present disclosure includes a computer, an acquisition unit that acquires a plurality of images taken by a plurality of cameras, an orientation detection unit that detects the orientation of a target person, and the target. A position specifying unit that specifies the position of a person, and an image generation unit that generates an image related to the target person from the plurality of images based on the detected orientation of the target person and the identified position of the target person. To function as.

According to this configuration, the computer can function as the above-mentioned video processing device. Therefore, the same operations and effects as those of the above-mentioned video processing apparatus can be obtained.

[Details of Embodiments of the present disclosure]
Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. It should be noted that all of the embodiments described below show a specific example of the present disclosure. Numerical values, shapes, materials, components, arrangement positions and connection forms of components, steps, step order, and the like shown in the following embodiments are examples, and do not limit the present disclosure. Further, among the components in the following embodiments, the components not described in the independent claims are components that can be arbitrarily added. Further, each figure is a schematic view and is not necessarily exactly illustrated.

Further, the same components are designated by the same reference numerals. Since their functions and names are the same, their description will be omitted as appropriate.

<Embodiment 1>
[Overall configuration of video processing system]
FIG. 1 is a diagram showing a configuration of a video processing system according to the first embodiment of the present disclosure.

The video processing system 100 is a system that transmits the video of the target person 3 or the video that the target person 3 will be viewing to the user 4 other than the target person 3. The user 4 is, for example, an interlocutor who makes a call with the target person 3.

The video processing system 100 includes a plurality of cameras 2, a video processing device 1, and a terminal device 5.

The camera 2 is, for example, a surveillance camera installed in the city or inside or outside a building, a camera installed indoors, an in-vehicle camera installed in a vehicle, or the like. The camera 2 is directly or indirectly connected to a network 7 such as the Internet or a 5G (fifth generation mobile communication system) communication network by wire or wirelessly.

The image processing device 1 is connected to the network 7, acquires images from a plurality of cameras 2, and generates an image of the target person 3 or an image that the target person 3 will be viewing based on the acquired images. The video processing device 1 transmits the generated video to the terminal device 5 such as a smartphone used by the user 4 via the network 7.

It is assumed that the target person 3 is wearing, for example, a hands-free phone 6 and is having a conversation with a user 4 who has a terminal device 5. The user 4 can have a conversation with the target person 3 while watching the image of the target person 3 or the image that the target person 3 will be seeing through the screen of the terminal device 5.

[Configuration of video processing device 1]
FIG. 2 is a block diagram showing a functional configuration of the video processing device 1. The video processing device 1 is composed of, for example, a computer including a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), an HDD (Hard Disk Drive), a communication interface, an input / output interface, and the like. Can be done.

The video processing device 1 includes a video acquisition unit 10, a target person information acquisition unit 20, a mode reception unit 30, a video processing unit 40, and a video transmission unit 50. For example, the functions of the processing units 10 to 50 are realized by expanding the computer program stored in the HDD of the computer on the RAM and executing the computer program on the CPU.

The video acquisition unit 10 functions as an acquisition unit, receives video data (hereinafter referred to as “video”) taken by the camera 2 from the camera 2 via the network 7, and outputs the video to the video processing unit 40. .. It is assumed that information indicating the identifier of the camera 2, the position of the camera 2, and the direction of the optical axis of the camera 2 is added to the video. However, these information may be stored in a storage device such as an HDD. In this case, if the information of the identifier of the camera 2 is added to the video, the position of the camera 2 and the direction of the optical axis of the camera 2 can be specified from this information.

The target person information acquisition unit 20 acquires the target person information which is the information of the target person 3 whose image is to be generated. The target person information is information necessary for identifying the image of the target person 3, and is, for example, a face image of the target person 3. When the target person 3 starts a conversation with the user 4, the target person 3 takes a picture (selfie) of his / her own face image using a terminal device such as a smartphone owned by the target person 3, transmits it to the video processing device 1, and performs video processing. The device 1 receives the face image of the target person 3 as the target person information. The target person information acquisition unit 20 outputs the received face image of the target person 3 to the video processing unit 40.

The terminal device possessed by the subject 3 may extract an image feature amount such as SIFT (Scale-Invariant Features Transfer) or SURF (Speeded-Up Robust Features) from the self-portrait face image. The terminal device transmits the extracted image feature amount to the video processing device 1 as the target person information, and the video processing device 1 receives the image feature amount as the target person information. The target person information acquisition unit 20 outputs the received image feature amount to the video processing unit 40.

The identifier of the target person 3 and the face image or image feature amount of the target person 3 may be stored in advance in a storage device such as an HDD of the image processing device 1. In this case, the target person information acquisition unit 20 acquires the identifier of the target person 3 from the terminal device possessed by the target person 3, and acquires the face image or the image feature amount of the target person 3 from the storage device based on the identifier. .. In this case, the identifier of the target person 3 can be, for example, the telephone number of the target person 3. The video processing device 1 acquires a telephone number at the start of a call of the target person 3, and acquires a face image or an image feature amount of the target person 3 based on the acquired telephone number.

The mode reception unit 30 receives the mode of the video to be generated. The mode includes a selfie mode and a viewpoint sharing mode.

The self-portrait mode is a mode for generating an image of the subject 3. More specifically, the self-shooting mode is a mode for generating an image of the target person 3 taken in the direction of the target person 3 from a position facing the target person 3. Here, the orientation of the subject 3 includes, for example, at least one of the orientation of the line of sight of the subject 3, the orientation of the face of the subject 3, and the orientation of the body of the subject 3.

The viewpoint sharing mode is a mode for generating an image that the subject 3 will be watching. More specifically, the viewpoint sharing mode is a mode for generating an image of the surroundings of the target person 3 in the direction of the target person 3 from the position of the target person 3. The orientation of the subject 3 is the same as in the selfie mode.

When starting a conversation with the user 4, the target person 3 selects a mode using a terminal device such as a smartphone possessed by the target person 3, and transmits the selected mode to the mode reception unit 30. The mode reception unit 30 receives the mode from the terminal device and outputs the received mode to the video processing unit 40.

The image processing unit 40 receives images taken by a plurality of cameras 2 from the image acquisition unit 10, target person information of the target person 3 from the target person information acquisition unit 20, and a mode from the mode reception unit 30. The video processing unit 40 generates a video according to the mode based on the plurality of videos and the target person information of the target person 3. That is, the image processing unit 40 generates the image of the target person 3 when the mode is the self-shooting mode, and generates the image that the target person 3 will be watching when the mode is the viewpoint sharing mode. The video processing unit 40 outputs the generated video to the video transmission unit 50. The details of the video processing unit 40 will be described later.

The video transmission unit 50 receives the video from the video processing unit 40 and transmits the video to the terminal device 5 used by the user 4 who is the other party of the target person 3.

[Structure of video processing unit 40]
FIG. 3 is a block diagram showing a detailed configuration of the image processing unit 40 included in the image processing device 1 shown in FIG.

The image processing unit 40 includes a target person image detection unit 41, a target person orientation detection unit 42, a target person image selection unit 43, a privacy protection processing unit 44, a target person position identification unit 45, and a surrounding image detection unit 46. And the surrounding image selection unit 47.

The target person video detection unit 41 is based on the target person information received from the target person information acquisition unit 20, and is a video showing the target person 3 among a plurality of videos received from the video acquisition unit 10 (hereinafter, “target”. Detects "personal video").

Specifically, the target person image detection unit 41 extracts an image feature amount such as SIFT or SURF from each image received from the image acquisition unit 10. Further, when the target person information received from the target person information acquisition unit 20 is a face image of the target person 3, the target person image detection unit 41 extracts the image feature amount of the target person 3 from the face image. If the target person information received from the target person information acquisition unit 20 is the image feature amount of the target person 3, the target person image detection unit 41 does not perform processing on the image feature amount.

The target person image detection unit 41 searches for a region having an image feature amount similar to the image feature amount of the target person 3 from a plurality of images, and detects the image having the area as the target person image. That is, the target person image detection unit 41 detects as the target person image an image having an image feature amount region whose similarity with the image feature amount of the target person 3 is equal to or more than a predetermined threshold value from a plurality of images. The target person image detection unit 41 outputs the detected target person image to the target person orientation detection unit 42, the target person image selection unit 43, and the target person position identification unit 45. Further, the target person image detection unit 41 sets the position of the region having the image feature amount whose similarity is equal to or higher than a predetermined threshold value as the position of the target person 3 in the target person video, and sets the privacy protection processing unit 44 and the target person position identification unit. Output to 45.

The target person orientation detection unit 42 functions as an orientation detection unit, receives a target person image from the target person image detection unit 41, and detects the direction of the target person 3 based on the target person image. A known technique can be used to detect the orientation of the subject 3. For example, when the direction of the target person 3 is the direction of the line of sight of the target person 3, the target person orientation detection unit 42 has a portion where the eyes do not move (reference point) and a portion (moving point) where the eyes do not move from the subject image. And detect. Here, the reference point is the inner corner of the subject 3 and the moving point is the iris of the subject 3. The target person orientation detection unit 42 detects the direction of the target person 3's line of sight when the direction of the optical axis of the camera 2 is used as a reference based on the position of the moving point with respect to the reference point (see, for example, Non-Patent Document 2). ).

The target person orientation detection unit 42 in the three-dimensional space based on the information indicating the position of the camera 2 and the direction of the optical axis of the camera 2 added to the target person image and the detected orientation of the target person 3. The orientation of the subject 3 is detected. That is, since the position of the camera 2 and the direction of the optical axis of the camera 2 are known, and the direction of the line of sight of the target person 3 with respect to the direction of the optical axis of the camera 2 is known, the target person orientation detection unit 42 calculates the orientation of the subject 3 in the three-dimensional space from these. The target person orientation detection unit 42 outputs information on the orientation of the target person 3 to the target person image selection unit 43 and the surrounding image selection unit 47.

The target person image selection unit 43 receives the target person image from the target person image detection unit 41, and receives information on the orientation of the target person 3 from the target person orientation detection unit 42. The target person image selection unit 43 functions as an image generation unit, and when the mode is the self-shooting mode, the target person 3 is photographed from the direction closest to the direction of the target person 3 from the target person images. Select as the video related to the subject 3. That is, the target person image selection unit 43 calculates the difference between the direction of the optical axis of the camera 2 and the direction of the target person 3 in the three-dimensional space for each target person image. The target person video selection unit 43 selects the target person video having the smallest difference as the video related to the target person 3. The target person video selection unit 43 outputs the video related to the selected target person 3 to the privacy protection processing unit 44.

FIG. 4 is a diagram for explaining an example of the target person image selection process by the target person image selection unit 43. (A) to (C) of FIG. 4 show the subject image received from the subject image detection unit 41. Here, when the orientation of the target person 3 is, for example, the orientation of the face of the target person 3, the orientation of the face of the target person 3 shown by the target person image shown in FIG. 4B is the target person image. It is closest to the direction of the optical axis of the camera 2 that took the picture. That is, the above difference is the smallest. Therefore, the target person image selection unit 43 selects the target person image shown in FIG. 4B as an image related to the target person 3. FIG. 4D shows an image related to the subject 3.

The target person position identification unit 45 receives information on the target person image and the position of the target person 3 from the target person image detection unit 41. The target person position specifying unit 45 functions as a position specifying unit, and includes information indicating the position of the camera 2 added to the target person image and the direction of the optical axis of the camera 2, and the position of the target person 3 in the target person image. Therefore, the position of the target person 3 in the three-dimensional space is specified. The target person position specifying unit 45 can specify the optical axis direction of the camera 2 in the three-dimensional space from the information indicating the position of the camera 2 and the direction of the optical axis of the camera 2 added to the target person image. .. Further, since the position of the target person 3 in the target person image is known, the target person position specifying unit 45 indicates in which direction the target person 3 exists with respect to the position of the camera 2 in the three-dimensional space. Can be calculated. When there are a plurality of target person images, the target person position specifying unit 45 calculates the intersection of the directions of the target person 3 calculated from each target person image, and thereby, the position of the target person 3 in the three-dimensional space. Can be identified.

The target person position specifying unit 45 considers the distance from the camera 2 to the target person 3 as a predetermined predetermined distance even if there is only one target person image, so that the target person 3 in the three-dimensional space The position of can be specified.

The target person position specifying unit 45 outputs information on the position of the target person 3 in the specified three-dimensional space to the surrounding image detection unit 46.

The surrounding image detection unit 46 receives images taken by a plurality of cameras 2 from the image acquisition unit 10, and receives information on the position of the target person 3 in the three-dimensional space from the target person position identification unit 45. The surrounding image detection unit 46 detects an image (hereinafter, referred to as “surrounding image”) obtained by photographing the surroundings of the object 3 from a plurality of images based on the position of the object 3 in the three-dimensional space. For example, the ambient image detection unit 46 detects an image captured by the camera 2 whose distance to the object 3 is equal to or less than a predetermined distance threshold value as an ambient image based on the position of the target person 3 and the position of the camera 2. .. The ambient image detection unit 46 outputs the detected ambient image to the ambient image selection unit 47.

The peripheral image selection unit 47 receives the ambient image from the ambient image detection unit 46, and receives the information on the orientation of the target person 3 from the target person orientation detection unit 42. The surrounding image selection unit 47 functions as an image generation unit, and when the mode is the viewpoint sharing mode, an image of the surroundings of the target person 3 viewed from the position of the target person 3 in the direction of the target person 3 from the surrounding images. Is selected as the video related to the subject 3. That is, the surrounding image selection unit 47 calculates the difference between the direction of the optical axis of the camera 2 and the direction of the target person 3 in the three-dimensional space for each surrounding image. The surrounding image selection unit 47 selects the target person image having the smallest difference as the image related to the target person 3. The surrounding image selection unit 47 outputs the image related to the selected target person 3 to the privacy protection processing unit 44.

5 to 7 are diagrams for explaining an example of the ambient image selection process by the ambient image selection unit 47.

FIG. 5 is a diagram showing an object located in a three-dimensional space. For example, it is assumed that the cone 81 and the sphere 82 are arranged in the three-dimensional space. The arrow 70 indicates the direction of the subject 3, and it is assumed that the subject 3 is looking at the cone 81 and the sphere 82 in the direction of the arrow 70. Further, arrows 71 to 73 indicate the directions of the optical axes of the three cameras 2 that capture the surrounding image, and the three cameras 2 have the cone 81 and the cone 81 in any direction from the arrow 71 to the arrow 73. It is assumed that the ball 82 is photographed.

FIG. 6 is a diagram showing an example of a video viewed by the subject 3. Subject 3 is looking at the cone 81 arranged on the left side and the sphere 82 arranged on the right side with a space between the cone 81.

FIG. 7A shows an image taken by the camera 2 whose optical axis direction is arrow 71, and the sphere 82 is located on the back surface of the cone 81 in the image. FIG. 7B shows an image taken by the camera 2 whose optical axis direction is arrow 72, and in the image, the cone 81 and the sphere 82 are arranged with a space open. FIG. 7C shows an image taken by the camera 2 whose optical axis direction is arrow 73, and the cone 81 is located on the back surface of the sphere 82 in the image.

As shown in FIG. 5, the direction of the optical axis closest to the direction of the subject 3 (arrow 70) is arrow 72. Therefore, the ambient image selection unit 47 selects the ambient image shown in FIG. 7 (B) as the image related to the target person 3. FIG. 7 (D) shows an image related to the subject 3.

The privacy protection processing unit 44 functions as a video generation unit. The privacy protection processing unit 44 receives information on the position of the target person 3 in the target person video from the target person video detection unit 41, and receives the target person video as a video related to the target person 3 from the target person video selection unit 43. , The surrounding image is received from the surrounding image selection unit 47 as an image related to the target person 3.

When the mode is the self-portrait mode, the privacy protection processing unit 44 detects an image of a person from the target person image received from the target person image selection unit 43. For the detection of the image of a person, for example, a face image detection technique for detecting a face image from a video can be used.

The privacy protection processing unit 44 leaves the image of the person existing at the position of the target person 3, that is, the image of the target person 3, and applies a mosaic to the images of other people, or the image in a predetermined image. Perform privacy protection processing such as mask processing. This makes it difficult to identify a person other than the target person 3.

When the mode is the viewpoint sharing mode, the privacy protection processing unit 44 detects an image of a person from the surrounding images received from the surrounding image selection unit 47. The privacy protection processing unit 44 performs privacy protection processing on the detected image of the person. This makes it difficult to identify a person other than the target person 3.

The privacy protection processing unit 44 outputs a video related to the subject 3 to which the privacy protection processing has been performed to the video transmission unit 50.

The video transmission unit 50 receives the video related to the target person 3 from the privacy protection processing unit 44 and transmits the video to the terminal device 5.

[Processing procedure of video processing device 1]
FIG. 8 is a flowchart showing an example of the processing procedure of the video processing device 1.

The target person information acquisition unit 20 acquires the face image of the target person 3 as the target person information from the terminal device possessed by the target person 3 (step S1).

The mode reception unit 30 receives the mode selected by the target person 3 from the terminal device possessed by the target person 3 (step S2).

When the received mode is the self-shooting mode (YES in step S3), the video processing unit 40 executes the self-shooting mode video processing (step S4). The details of the self-shooting mode video processing (step S4) will be described later.

When the received mode is the viewpoint sharing mode (NO in step S3, YES in step S5), the video processing unit 40 executes the viewpoint sharing mode video processing (step S6). The viewpoint sharing mode video processing (step S6) will be described later.

If an erroneous mode other than the self-shooting mode and the viewpoint sharing mode is accepted, or if the mode is not accepted (NO in step S3, NO in step S5), the image processing apparatus 1 ends the process.

[Selfie mode video processing]
FIG. 9 is a flowchart showing details of the self-shooting mode video processing (step S4 of FIG. 8).

The image acquisition unit 10 acquires images captured by the cameras 2 from the plurality of cameras 2 (step S11).

The target person image detection unit 41 is among a plurality of images acquired in step S11 based on the face image of the target person 3 as the target person information acquired in the target person information acquisition process (step S1 in FIG. 8). Therefore, the target person image showing the target person 3 is detected (step S12).

The image processing unit 40 determines the number of target person images detected in step S12 (step S13).

When the number of target person images is 0 (0 in step S13), the processing after step S14 is not executed.

When the number of target person images is 2 or more (2 or more in step S13), the target person orientation detection unit 42 detects the orientation of the target person 3 for each target person image detected in step S12 (step S12). S14).

The target person image selection unit 43 calculates the difference between the direction of the target person 3 and the direction of the optical axis of the camera 2 for each direction of the target person 3 detected in step S14 (step S15).

The target person image selection unit 43 selects the target person image having the smallest difference calculated in step S15 from the target person images detected in step S12 (step S16).

The privacy protection processing unit 44 performs privacy protection processing on the image of a person other than the target person 3 with respect to the target person video selected in step S16 (step S17).

The video transmission unit 50 transmits the target person video to which the privacy protection processing has been performed in step S17 to the terminal device 5 used by the user 4 (step S18).

When the number of target person videos detected in step S12 is 1 (2 or more in step S13), privacy protection processing (step S17) and video transmission processing (step S18) are performed on the target person video. ..

The video processing unit 40 repeatedly executes the processes from step S11 to step S18 until the call between the target person 3 and the user 4 is completed.

[Viewpoint sharing mode video processing]
FIG. 10 is a flowchart showing details of the viewpoint sharing mode video processing (step S6 of FIG. 8).

The image acquisition unit 10 acquires images captured by the cameras 2 from the plurality of cameras 2 (step S21).

The target person image detection unit 41 is among a plurality of images acquired in step S21 based on the face image of the target person 3 as the target person information acquired in the target person information acquisition process (step S1 in FIG. 8). From, the target person image in which the target person 3 is shown and the position of the target person 3 in the target person image are detected (step S22).

The image processing unit 40 determines whether or not the target person image is detected in step S22 (step S23).

If the subject image is not detected (NO in step S23), the processing after step S24 is not executed.

When the target person image is detected (YES in step S23), the target person orientation detection unit 42 detects the orientation of the target person 3 for each target person image detected in step S22 (step S24).

The target person position specifying unit 45 identifies the position of the target person 3 in the three-dimensional space based on the target person image detected in step S22 and the position information of the target person 3 (step S25).

The surrounding image detection unit 46 captures the surrounding image of the target person 3 from the images acquired in the step S21 based on the position of the target person 3 in the three-dimensional space specified in the step S25. Detect (step S26).

The image processing unit 40 determines the number of ambient images detected in step S26 (step S27).

When the number of surrounding images is 0 (0 in step S27), the processing after step S28 is not executed.

When the number of ambient images is 2 or more (2 or more in step S27), the ambient image selection unit 47 directs the optical axis of the camera 2 in the three-dimensional space for each ambient image detected in step S26. The difference between the orientation of the subject 3 and the orientation of the subject 3 is calculated (step S28).

The ambient image selection unit 47 selects the ambient image having the smallest difference calculated in step S28 from the ambient images detected in step S26 (step S29).

The privacy protection processing unit 44 applies privacy protection processing to the image of a person with respect to the surrounding image selected in step S29 (step S30).

The video transmission unit 50 transmits the surrounding video to which the privacy protection processing has been performed in step S30 to the terminal device 5 used by the user 4 (step S31).

When the number of ambient images detected in step S26 is 1 (1 in step S27), privacy protection processing (step S30) and video transmission processing (step S31) are performed on the ambient video.

The video processing unit 40 repeatedly executes the processes from step S21 to step S31 until the call between the target person 3 and the user 4 is completed.

[Effect of Embodiment 1]
As described above, according to the first embodiment of the present disclosure, the image processing device 1 is based on the orientation and position of the target person 3 from a plurality of images acquired from a plurality of cameras 2 via the network 7. An image related to the target person 3 can be generated. Therefore, even if the target person 3 does not wear the dedicated terminal device on which the camera 2 is arranged, the image related to the target person 3 can be generated. As a result, it is possible to generate an image related to the target person 3 without imposing an excessive burden on the target person 3.

Further, the image processing device 1 can generate an image related to the target person 3 by using the cameras 2 existing around the target person 3 without using the dedicated camera 2. That is, the target person 3 can generate the image related to the target person 3 without having to worry about the remaining battery level of the camera 2, the heat generation, and the like.

Further, when the mode is the self-shooting mode, it is possible to generate a so-called self-shooting image in which the target person 3 is photographed from a position facing the target person 3.

Further, from the images including the image of the target person 3, an image obtained by shooting the target person 3 from the direction closest to the direction of the target person 3 can be generated as a so-called self-portrait image. Therefore, it is possible to generate a self-portrait image at high speed.

Further, when the mode is the viewpoint sharing mode, for example, it is possible to generate an image of the target in front of the line of sight of the target person 3, that is, an image of the same target as the target person 3 is watching.

Further, from a plurality of images, for example, an image of the target located in front of the line of sight of the target person 3 can be selected. Therefore, it is possible to generate an image of the surroundings viewed from the subject 3 at high speed.

The orientation of the subject 3 includes at least one of the orientation of the subject 3's line of sight, the orientation of the subject 3's face, and the orientation of the subject 3's body. Therefore, it is possible to generate an image of the target person 3 viewed from the direction of the line of sight, the direction of the face, or the direction of the body of the target person 3, or an image of the surroundings viewed from the position of the target person 3 in that direction.

In addition, privacy protection processing is applied to the image of a person other than the target person 3. Therefore, it is possible to generate an image in which the privacy of the person concerned is protected.

Further, the video processing device 1 can transmit the generated video to the terminal device 5. Therefore, the target person 3 can make a video call or have a video conference with the user 4 who uses the terminal device 5 without being aware of the camera 2.

<Embodiment 2>
In the first embodiment, an image related to the target person 3 is generated by selecting one image from the images taken by the camera 2. In the second embodiment, an example of generating an image related to the target person 3 by synthesizing a plurality of images taken by the camera 2 will be described.

The overall configuration of the video processing system according to the second embodiment of the present disclosure is the same as that shown in FIG.

Further, the functional configuration of the video processing apparatus 1 according to the second embodiment of the present disclosure is the same as that shown in FIG. However, the detailed configuration of the video processing unit 40 is different from that of the first embodiment.

[Structure of video processing unit 40]
FIG. 11 is a block diagram showing a detailed configuration of the image processing unit 40 included in the image processing device 1 shown in FIG.

The image processing unit 40 includes a target person image detection unit 41, a target person orientation detection unit 42, a self-portrait video generation unit 48, a privacy protection processing unit 44, a target person position identification unit 45, and a surrounding image detection unit 46. And a target person's viewpoint image generation unit 49.

The processes executed by the target person image detection unit 41, the target person orientation detection unit 42, the target person position identification unit 45, and the surrounding image detection unit 46 are the same as those in the first embodiment.

However, the target person image detection unit 41 outputs the detected target person image to the self-portrait image generation unit 48. Further, the target person orientation detection unit 42 outputs the orientation information of the target person 3 to the self-portrait image generation unit 48 and the target person viewpoint image generation unit 49. Further, the ambient image detection unit 46 outputs the detected ambient image to the target person viewpoint image generation unit 49.

The self-portrait image generation unit 48 receives the target person image from the target person image detection unit 41, and receives the information on the orientation of the target person 3 from the target person orientation detection unit 42. The self-portrait video generation unit 48 functions as a video generation unit, and when the mode is the self-portrait mode, the self-portrait image generation unit 48 synthesizes the target person video so that the target person 3 faces the target person 3 from a position facing the target person 3. Generates a video of a selfie. The video is a so-called self-portrait video. That is, the self-portrait image generation unit 48 synthesizes the self-portrait image by interpolation interpolation by performing the viewpoint conversion process based on the plurality of target person images. The viewpoint conversion process is disclosed in, for example, Non-Patent Document 3. The self-portrait image generation unit 48 outputs the generated self-portrait image to the privacy protection processing unit 44 as an image related to the target person 3.

FIG. 12 is a diagram for explaining an example of a self-portrait image generation process by the self-portrait image generation unit 48. (A) to (C) of FIG. 12 show the target person image received from the target person image detection unit 41. Here, when the orientation of the subject 3 is, for example, the orientation of the face of the subject 3, the self-portrait video generation unit 48 has three subjects shown in FIGS. 12A to 12C. A self-portrait video is generated by synthesizing a video shot of the subject 3 from the front by interpolation interpolation from the video. FIG. 12D shows the generated self-portrait image.

The target person viewpoint image generation unit 49 receives the surrounding image from the surrounding image detection unit 46, and receives the information on the orientation of the target person 3 from the target person orientation detection unit 42. The target person viewpoint image generation unit 49 functions as an image generation unit, and when the mode is the viewpoint sharing mode, the surroundings of the target person 3 are oriented from the position of the target person 3 to the direction of the target person 3 by synthesizing the surrounding images. Generates a video of what you see (hereinafter referred to as "target person's viewpoint video"). That is, the target person viewpoint image generation unit 49 synthesizes the target person viewpoint image by interpolation interpolation by performing the viewpoint conversion process based on the plurality of surrounding images. The viewpoint conversion process is disclosed in, for example, Non-Patent Document 3. The target person viewpoint image generation unit 49 outputs the generated target person viewpoint image to the privacy protection processing unit 44 as an image related to the target person 3.

FIG. 13 is a diagram for explaining an example of the generation process of the target person's viewpoint image by the target person's viewpoint image generation unit 49. (A) to (C) of FIG. 13 show the same surrounding images as shown in (A) to (C) of FIG. 7, respectively. That is, with reference to FIG. 5, FIG. 13A shows an ambient image taken by the camera 2 whose optical axis direction is arrow 71, and FIG. 13B shows the direction of the optical axis. Shows the ambient image taken by the camera 2 of the arrow 72, and FIG. 13C shows the ambient image taken by the camera 2 whose optical axis direction is the arrow 73. The self-portrait image generation unit 48 synthesizes the target person's viewpoint image by interpolation interpolation from the three surrounding images shown in FIGS. 13 (A) to 13 (C). FIG. 13 (D) shows the generated target person's viewpoint image. The subject viewpoint image is an image similar to the image seen by the subject 3 shown in FIG.

The privacy protection processing unit 44 receives information on the position of the target person 3 in the target person video from the target person video detection unit 41, and receives the self-portrait video as a video related to the target person 3 from the self-portrait video generation unit 48. , The target person's viewpoint image is received from the target person's viewpoint image generation unit 49 as an image related to the target person 3.

When the mode is the self-portrait mode, the privacy protection processing unit 44 detects an image of a person from the self-portrait image received from the self-portrait image generation unit 48. The privacy protection processing unit 44 leaves the image of the person existing at the position of the target person 3, that is, the image of the target person 3, and performs the privacy protection processing on the images of other people. This makes it difficult to identify a person other than the target person 3.

When the mode is the viewpoint sharing mode, the privacy protection processing unit 44 detects an image of a person from the target person's viewpoint image received from the target person's viewpoint image generation unit 49. The privacy protection processing unit 44 performs privacy protection processing on the detected image of the person. This makes it difficult to identify a person other than the target person 3.

The privacy protection processing unit 44 outputs a video related to the subject 3 to which the privacy protection processing has been performed to the video transmission unit 50.

[Processing procedure of video processing device 1]
The processing procedure of the video processing device 1 is the same as the processing procedure of the video processing device 1 according to the first embodiment shown in FIG. However, the details of the self-shooting mode video processing (S4) and the viewpoint sharing mode video processing (S6) are different from those of the first embodiment.

[Selfie mode video processing]
FIG. 14 is a flowchart showing details of the self-shooting mode video processing (step S4 of FIG. 8).

The processes from step S11 to step S14, step S17 and step S18 are the same as those shown in FIG.

In the second embodiment, the process of step S101 is executed instead of the processes of steps S15 and S16 shown in FIG.

That is, the self-portrait video generation unit 48 generates a self-portrait video in which the target person 3 is photographed in the direction of the target person 3 from a position facing the target person 3 by synthesizing the target person video (step S101).

In the privacy protection process (step S17), the privacy protection process for the self-portrait video is performed, and in the video transmission process (step S18), the self-portrait video with the privacy protection process is transmitted to the terminal device 5. NS.

[Viewpoint sharing mode video processing]
FIG. 15 is a flowchart showing details of the viewpoint sharing mode video processing (step S6 of FIG. 8).

The processes from step S21 to step S27, step S30 and step S31 are the same as those shown in FIG.

In the second embodiment, the process of step S102 is executed instead of the processes of steps S28 and S29 shown in FIG.

That is, the target person viewpoint image generation unit 49 generates the target person viewpoint image by synthesizing the surrounding images (step S102).

In the privacy protection process (step S30), the privacy protection process for the target person's viewpoint video is performed, and in the video transmission process (step S31), the target person's viewpoint video to which the privacy protection process is applied is sent to the terminal device 5. Will be sent.

[Effect of Embodiment 2]
As described above, according to the second embodiment of the present disclosure, a so-called self-portrait video can be generated by synthesizing a video including an image of the subject 3. Therefore, even if the image obtained by shooting the target person 3 in the direction of the target person 3 cannot be obtained from the camera 2, it is possible to generate the image obtained by shooting the target person in the direction.

Further, by synthesizing a plurality of surrounding images, for example, it is possible to generate an image of an object in front of the line of sight of the object 3. Therefore, even if it is not possible to generate such an image from one ambient image, it is possible to generate the same target image as the target person 3 is viewing.

[Additional Notes]
Although the video processing apparatus 1 according to the embodiment of the present disclosure has been described above, the present disclosure is not limited to this embodiment.

For example, in the above-described embodiment, the video related to the target person 3 is transmitted to the terminal device 5 used by the user 4, but the video processing device 1 transmits the video related to the user 4 to the target person 3. It may be generated in the same manner as the related video and transmitted to the terminal device used by the target person 3. As a result, a two-way videophone can be realized between the target person 3 and the user 4.

Further, the application target of the video processing device 1 is not limited to the videophone. For example, the video processing device 1 can be applied to a video conference. For example, a plurality of cameras 2 installed in the first conference room at the first point (hereinafter referred to as "first camera") photograph the first conference room, and a plurality of cameras installed in the second conference room at the second point. Camera 2 (hereinafter referred to as "second camera") will be used to photograph the second conference room. The video processing device 1 generates a video related to the user in the first conference room from the video captured by the plurality of first cameras and transmits the video to the second terminal device for video conferencing installed in the second conference room. do. By receiving the video and displaying it on the display device, the second terminal device can show the video related to the user in the first conference room to the user in the second conference room. Similarly, the video processing device 1 generates a video related to a user in the second conference room from the video captured by the plurality of second cameras, and is installed in the first conference room as a first terminal for video conferencing. Send to the device. By receiving the video and displaying it on the display device, the first terminal device can show the video related to the user in the second conference room to the user in the first conference room.

Further, the target person position specifying unit 45 of the image processing unit 40 determines the position of the target person 3 in the three-dimensional space from the position information of the camera 2 and the position of the target person 3 in the video. If the position of the target person 3 can be known by any other method, the position information may be used. For example, when the terminal device possessed by the target person 3 is provided with the function of a GPS (Global Positioning System) sensor, the target person positioning unit 45 uses the position information of the target person 3 captured by the terminal device. It may be received from the terminal device and used as the position information of the target person 3 in the three-dimensional space.

Further, the target person orientation detection unit 42 of the image processing unit 40 has decided to detect the orientation of the target person 3 based on the target person image, but if the orientation of the target person 3 can be known by any other method, The orientation information may be used. For example, when the subject 3 wears a terminal device such as a geomagnetic sensor (electronic compass) or a smartphone equipped with a pair of a geomagnetic sensor and a gyro sensor in a pocket of clothes or the like, a detection unit suitable for the subject. 42 may detect the orientation of the target person based on the sensor output, and use the detection result as the orientation information of the target person. The terminal device may be a dedicated sensor that can be attached to the target person 3 to detect the orientation of the above-mentioned geomagnetic sensor or the like. Further, the terminal device may be a wearable device (for example, smart glasses) including the above-mentioned geomagnetic sensor or the like.

In addition, some or all of the components constituting the video processing device 1 may be one or more system LSIs (Large Scale Integration), ASICs (Application Specific Integrated Circuits), FPGA (Field-Programmable Gate Array), and the like. It may be composed of.

Further, the above-mentioned computer program may be recorded and distributed on a computer-readable non-temporary recording medium such as an HDD, a CD-ROM, or a semiconductor memory.

Further, the video processing device 1 may be realized by a plurality of computers.

Further, a part or all of the functions of the video processing device 1 may be provided by cloud computing. That is, a part or all the functions of the video processing device 1 may be realized by the cloud server.

Further, at least a part of the above embodiments may be arbitrarily combined.

The embodiments disclosed this time should be considered to be exemplary in all respects and not restrictive. The scope of the present disclosure is indicated by the scope of claims, not the above-mentioned meaning, and is intended to include all modifications within the meaning and scope equivalent to the scope of claims.

1 Video processing device 2 Camera 3 Target person 4 User 5 Terminal device 6 Hands-free phone 7 Network 10 Video acquisition unit 20 Target person information acquisition unit 30 Mode reception unit 40 Video processing unit 41 Target person video detection unit 42 Target person orientation detection unit 42 43 Target person image selection unit 44 Privacy protection processing unit 45 Target person position identification unit 46 Surrounding image detection unit 47 Surrounding image selection unit 48 Self-portrait image generation unit 49 Target person viewpoint image generation unit 50 Video transmission unit 70 Arrow 71 Arrow 72 Arrow 73 Arrow 81 Conical 82 Sphere 100 Video processing system

Claims (15)

An acquisition unit that acquires multiple images taken by multiple cameras,
A direction detection unit that detects the direction of the target person,
The position specifying unit that specifies the position of the target person and
An image processing device including an image generation unit that generates an image related to the object from the plurality of images based on the detected orientation of the object and the identified position of the object. The video processing apparatus according to claim 1, wherein the plurality of cameras include at least one of a camera installed outdoors, a camera installed indoors, and an in-vehicle camera. The video processing device according to claim 1 or 2, wherein the orientation detection unit detects the orientation of the target person based on at least one of the acquired plurality of images. Any one of claims 1 to 3, wherein the image generation unit generates an image of the target person in the direction of the target person from a position facing the target person as an image related to the target person. The video processing apparatus according to claim 1. The video generator
The image processing apparatus according to claim 4, wherein an image taken from the direction closest to the direction of the target person is extracted from the plurality of images as an image related to the target person. The video generator
An image including an image of the target person is detected from the plurality of images, and the image is detected.
The video processing according to claim 4, wherein a video including the detected image of the target person is synthesized to generate a video of the target person taken in the direction of the target person from a position facing the target person. Device. Any one of claims 1 to 6, wherein the image generation unit generates an image of the surroundings of the target person in the direction of the target person from the position of the target person as an image related to the target person. The video processing apparatus according to claim 1. The video processing device according to claim 7, wherein the video generation unit selects a video of the surroundings of the target person in the direction of the target person from the position of the target person from the plurality of video images. The video processing apparatus according to claim 7, wherein the video generation unit generates a video of looking around the target person from the position of the target person in the direction of the target person by synthesizing the plurality of videos. .. Any one of claims 1 to 9, wherein the orientation of the subject includes at least one of the orientation of the subject's line of sight, the orientation of the subject's face, and the orientation of the subject's body. The video processing apparatus described in 1. The image generation unit further performs privacy protection processing on an image of a person other than the target person included in the video related to the target person, according to any one of claims 1 to 10. Video processing equipment. Moreover,
The video processing device according to any one of claims 1 to 11, further comprising a video transmission unit that transmits a video related to the target person generated by the video generation unit to a terminal device. With multiple cameras
An image processing system including the image processing apparatus according to any one of claims 1 to 12, which generates an image based on a plurality of images taken by the plurality of cameras. Steps to acquire multiple images taken by multiple cameras,
Steps to detect the orientation of the subject and
The step of identifying the position of the target person and
An image processing method including a step of generating an image related to the object from the plurality of images based on the detected orientation of the object and the identified position of the object. Computer,
An acquisition unit that acquires multiple images taken by multiple cameras,
A direction detection unit that detects the direction of the target person,
The position specifying unit that specifies the position of the target person and
A computer program for functioning as an image generation unit that generates an image related to the object from the plurality of images based on the detected orientation of the object and the identified position of the object.

Images