JP7828183B2 - Information processing device, image determination method, and program - Google Patents

Description

This disclosure relates to an information processing device, an image assessment method, and a program.

Non-Patent Document 1 discloses a method for removing objects reflected parallel to the glass surface from distant night scene images captured through the glass using a stereo camera. Non-Patent Document 2 discloses a technology that separates the background from the occluding object based on the difference in the relative positions of the background and the occluding object as seen from the camera while slightly moving the camera, and restores the background scene as if the occluding object were not present.

Fumiya Tsurumi, Jun Yamashita, Toru Kaneko, "A Study on Removing Reflections from Distant Night Scene Images Scattered Through Glass Using Stereo Images," Image Recognition and Understanding Symposium (MIRU2011), IS3-12: 942-947 (https://ipsj.ixsq.nii.ac.jp/ej/index.php?action=pages_view_main&active_action=repository_action_common_download&item_id=77776&item_no=1&attribute_id=1&file_no=1&page_id=13&block_id=8) Tianfan Xue, Michael Rubinstein, Ce Liu, William T. Freeman, A Computational Approach for Obstruction-Free Photography, SIGGRAPH 2015 (https://sites.google.com/site/obstructionfreephotography/)

The techniques disclosed in Non-Patent Documents 1 and 2 require the separation of distant and foreground views. However, applying the techniques disclosed in Non-Patent Documents 1 and 2 requires capturing two or more images of a subject from a sufficiently large distance that the distant and foreground views can be separated, and the subject must be photographed while facing the same direction and moving appropriately. Therefore, if the subject is not photographed from a sufficiently large distance and without moving appropriately in the same direction, the techniques disclosed in Non-Patent Documents 1 and 2 may not be able to separate the distant and foreground views. Furthermore, the photographer may be forced to make decisions and perform operations to move appropriately. In other words, when using the techniques disclosed in Non-Patent Documents 1 and 2 to identify the image of a reflected object, the shooting environment may be limited. Therefore, one aspect of the present disclosure aims to provide an information processing device, image assessment method, and program that can identify the image of a reflected object without limiting the shooting environment.

An information processing device according to one embodiment of the present disclosure includes a first imaging unit that captures an image of a first imaging range to acquire a first image, a second imaging unit that captures an image of a second imaging range outside the first imaging range to acquire a second image, and a determination unit that determines whether the second image contains a second object image having features that match features of a first object image contained in the first image at a degree that exceeds a determination threshold.

An image judgment method according to one embodiment of the present disclosure includes the steps of capturing an image of a first imaging range to acquire a first image, capturing an image of a second imaging range outside the first imaging range to acquire a second image, and determining whether the second image contains a second object image having features that match features of a first object image contained in the first image at a degree of similarity that exceeds a judgment threshold.

A program according to one embodiment of the present disclosure causes a computer to perform the following functions: capturing an image of a first imaging range to acquire a first image; capturing an image of a second imaging range outside the first imaging range to acquire a second image; and determining whether the second image contains a second object image having features that match features of a first object image contained in the first image at a degree of similarity that exceeds a determination threshold.

FIG. 1 is a front view illustrating an example of an information processing device. FIG. 2 is a side view taken along an arrow II in FIG. 1 . 10A and 10B are diagrams illustrating an example of a person captured in a first imaging range and the face of the person reflected on a reflective surface in a second imaging range. FIG. 1 is a block diagram illustrating an example of a configuration of an information processing device. FIG. 2 is a block diagram showing an example of the configuration of a control unit according to the first embodiment. 5 is a flowchart illustrating an example of an operation of the information processing device according to the first embodiment. 7 is a flowchart illustrating an example of the operation of the information processing device following FIG. 6 . FIG. 2 is a diagram illustrating an example of a first image. FIG. 10 is a diagram illustrating an example of a second image. FIG. 4 is a diagram illustrating an example of a display image displayed on a display unit. FIG. 10 is a diagram illustrating an example of the configuration of a control unit according to a second embodiment. 10 is a flowchart illustrating an example of an operation of the information processing device according to the second embodiment. FIG. 1 is a diagram showing an example of two facial images with different facial expressions reflected on a reflective surface. 10 shows an example of a second image and a second image including images of faces with different expressions. 10A and 10B are diagrams illustrating an example of a person captured within a first imaging range, an image of the person reflected on a reflective surface within a second imaging range, and a person within the second imaging range. 1 shows an example of a first image including a first object image that is an image of a moving person. 14B shows an example of a second image captured with the person represented by the first object image illustrated in FIG. 14A reflected in the second imaging range. FIG. 4 is a diagram illustrating an example of a display image displayed on a display unit. FIG. 4 is a diagram illustrating an example of a display image displayed on a display unit.

(First embodiment)
The configuration of an information processing device 100 according to this embodiment will be described with reference to FIGS. 1 to 9. FIG. 1 is a front view showing an example of the information processing device 100. FIG. 2 is a side view taken along arrow II in FIG. 1. In the following description, the right direction in FIG. 1 will be referred to as the +X direction, the upward direction as the +Y direction, and the forward direction as the +Z direction. In the following description, the +Z direction will be referred to as the front direction. In addition, the -Z direction, which is the opposite direction to the +Z direction, will be referred to as the rear direction.

The information processing device 100 includes a housing 101, a first imaging unit 102, a display unit 103, a touch panel 104, a second imaging unit 201, etc. The information processing device 100 is, for example, a smartphone, a tablet, a digital camera, a video camera, etc.

The information processing device 100 determines whether the object image captured by the first imaging unit 102 is included in the second image acquired by the second imaging unit 201. In other words, the information processing device 100 determines whether an object present in the imaging range of one imaging unit is reflected in a reflective object such as glass or a mirror within the imaging range of the other imaging unit.

The housing 101 has a front surface 105 and a rear surface 202. The first imaging unit 102 and display unit 103 are arranged on the front surface 105. The second imaging unit 201 is arranged on the rear surface 202.

The first imaging unit 102 captures an image of the first imaging range 211 to obtain a first image 411 (see Figure 4). The first imaging range 211 is in the direction in which the front 105 faces.

The second imaging unit 201 captures an image of the second imaging range 212, which is outside the first imaging range 211, to obtain a second image 412 (see FIG. 4). The second imaging range 212 is in the direction in which the rear surface 202 faces. In the information processing device 100 illustrated in FIG. 2, the second imaging unit 201 captures an image in the opposite direction to the first imaging unit 102.

The display unit 103 displays the second image 412. Furthermore, the display unit 103 may also display the first image 411. The display unit 103 is a liquid crystal display, an organic EL (Electro Luminescence) display, or the like. Also, for example, the display unit 103 is configured with a touch panel 104 superimposed thereon. In other words, the display unit 103 may be a touch panel display having the touch panel 104.

Figure 3 shows an example of a person 301 captured within the first imaging range 211 and the person 301 reflected on a reflective surface 302 within the second imaging range 212. As illustrated in Figure 3, when the person 301 is reflected on the reflective surface 302 within the second imaging range 212, the second image 412 acquired by the second imaging unit 201 includes an image of the person 301. For example, when the person 301 operates the information processing device 100 to capture an image of the second imaging range 212, the person 301 who is taking the image is reflected in the second image 412. Note that Figure 3 does not illustrate the information processing device 100 being reflected on the reflective surface 302.

Figure 4 is a block diagram showing an example of the configuration of the information processing device 100. The information processing device 100 includes a first imaging unit 102, a display unit 103, a touch panel 104, a second imaging unit 201, a control unit 401, a ROM (Read Only Memory) 402, a RAM (Random Access Memory) 403, etc. The first imaging unit 102, the display unit 103, the touch panel 104, and the second imaging unit 201 are as described above, so detailed description will be omitted.

The control unit 401 executes various processes in accordance with the programs and data stored in the ROM 402. For example, the control unit 401 is implemented using a processor such as a CPU (Central Processing Unit).

ROM 402 stores programs that operate the information processing device 100. RAM 403 temporarily stores the programs stored in ROM 402 that are read by the control unit 401, the first image 411, the second image 412, the processing results by the control unit 401, etc.

Figure 5 is a block diagram showing an example of the configuration of the control unit 401 according to this embodiment. The control unit 401 includes a first object detection unit 501, a second object detection unit 502, a first feature detection unit 503, a second feature detection unit 504, a determination unit 505, an image processing unit 506, a notification unit 507, etc.

The first object detection unit 501 detects a first object image 511 included in the first image 411. The second object detection unit 502 detects a second object image 512 included in the second image 412.

The first feature detection unit 503 detects feature amounts 513 from the first object image 511. For example, the feature amounts 513 indicate feature amounts of feature points of the first object image 511. The second feature detection unit 504 detects feature amounts 514 from the second object image 512. For example, the feature amounts 514 indicate feature amounts of feature points of the second object image 512.

The determination unit 505 determines whether the second image 412 includes a second object image 512 having a degree of similarity 515 with the features of the first object image 511 that exceeds a determination threshold. Specifically, the determination unit 505 calculates the degree of similarity 515 based on the comparison result between the feature amount 513 and the feature amount 514.

When the image processing unit 506 determines that the second image 412 includes a second object image 512 that has a degree of similarity 515 with the features of the first object image 511 that exceeds the judgment threshold, the image processing unit 506 performs image processing on the second image 412 to hide the second object image 512. Specifically, when the image processing unit 506 determines that the second image 412 includes a second object image 512 that has a degree of similarity 515 with the features of the first object image 511 that exceeds the judgment threshold, the image processing unit 506 causes the display unit 103 to display an image in which the second object image 512 is hidden in the second image 412.

When it is determined that the second image 412 includes a second object image 512 that has a degree of similarity 515 with the features of the first object image 511 that exceeds the determination threshold, the notification unit 507 outputs a message indicating that the second image 412 includes a second object image 512 that has a degree of similarity 515 with the features of the first object image 511 that exceeds the determination threshold.

FIG. 6 is a flowchart showing an example of the operation of the information processing device 100 according to this embodiment. For example, when the touch panel 104 receives an operation to start imaging, the control unit 401 starts the processing of step S601 illustrated in FIG. 6. For example, the control unit 401 may cause an icon for starting imaging to be displayed on the display unit 103. In this case, when the touch panel 104 detects contact at the position of the icon for starting imaging, the control unit 401 determines that the touch panel 104 has received an operation to start imaging, and starts the processing of step S601 illustrated in FIG. 6.

In step S601, the first imaging unit 102 captures the first imaging range 211 to acquire the first image 411. In step S602, the second imaging unit 201 captures the second imaging range 212 to acquire the second image 412. The processing of step S601 and the processing of step S602 may be executed simultaneously. Alternatively, the first image 411 acquired in step S601 may be stored in RAM 403, and then the second imaging unit 201 may acquire the second image 412 in step S602.

In step S603, the first object detection unit 501 detects a first object image 511 from the first image 411. Specifically, the first object detection unit 501 performs image processing such as pattern matching, edge detection, and region segmentation on the first image 411 to detect the first object image 511. For example, the first object detection unit 501 detects features according to the type of detection target from the first image 411 to detect the first object image 511 included in the first image 411.

For example, suppose the type of detection target is a human face. In this case, the first object detection unit 501 detects feature points indicating the coordinate values of the edges of the eyes, nose, mouth, etc. from the first image 411, and detects the image of the human face included in the first image 411 as the first object image 511.

Alternatively, for example, suppose the type of detection target is text. In this case, the first object detection unit 501 detects text from the first image 411 and detects the image of the text included in the first image 411 as the first object image 511. For example, if the first image 411 includes an image of text such as a vehicle license plate, a sign, or a sign indicating an address, the first object detection unit 501 can detect the image of the vehicle license plate, a sign, or a sign indicating an address included in the first image 411 as the first object image 511 by detecting the text from the first image 411.

Alternatively, the first object detection unit 501 may perform region division in the first image 411 and detect the first object image 511. In this case, the first object detection unit 501 does not limit the type of the first object image 511, but detects multiple types of object images included in the first image 411.

In step S604, the first feature detection unit 503 detects feature quantities 513 from the first object image 511 detected in step S603.

For example, if the type of the first object image 511 is a human face, the first feature detection unit 503 detects, as feature quantities 513, the relative positions between feature points indicating the coordinate values of the ends of the eyes, nose, mouth, etc. contained in the first object image 511, features determined from pixel values, etc.

Furthermore, for example, if the type of the first object image 511 is a human face, the first feature detection unit 503 may detect features indicating a specific facial expression as the feature amount 513. For example, the first feature detection unit 503 detects an evaluation value of the facial expression determined from the relative positions between feature points as the feature amount 513.

Furthermore, for example, if the type of the first object image 511 is a character, the first feature detection unit 503 detects features such as the shape of the character represented by the first object image 511 as the feature amount 513.

In step S605, the second object detection unit 502 detects a second object image 512 from the second image 412. Specifically, the second object detection unit 502 performs the same image processing as the first object detection unit 501 on the second image 412 to detect the second object image 512. As a result, the second object detection unit 502 detects an object image of the same type as the first object image 511 contained in the second image 412 as the second object image 512.

For example, if the type of detection target is a human face, the second object detection unit 502 detects feature points indicating the coordinate values of the edges of the eyes, nose, mouth, etc. from the second image 412, and detects the image of the human face included in the second image 412 as the second object image 512.

Furthermore, for example, if the type of detection target is text, the second object detection unit 502 detects text from the second image 412 and detects the image of the text contained in the second image 412 as the second object image 512.

For example, if the second image 412 includes an image of text such as a vehicle license plate, a sign, or a sign indicating an address, the second object detection unit 502 can detect the image of the vehicle license plate, sign, or sign indicating an address included in the second image 412 as the second object image 512 by detecting the text from the second image 412.

Alternatively, the second object detection unit 502 may perform region division in the second image 412 and detect the second object image 512. In other words, the second object detection unit 502 may not limit the type of the second object image 512, but may detect multiple types of object images included in the second image 412.

In step S606, the second feature detection unit 504 detects feature quantities 514 from the second object image 512 detected in step S605.

For example, if the type of the second object image 512 is a human face, the second feature detection unit 504 detects, as feature quantities 514, the relative positions of feature points indicating the coordinate values of the ends of the eyes, nose, mouth, etc. contained in the second object image 512, features determined from pixel values, etc.

Furthermore, for example, if the type of the second object image 512 is a human face, the second feature detection unit 504 may detect features indicating a specific facial expression as the feature amount 514. For example, the second feature detection unit 504 detects an evaluation value of the facial expression determined from the relative positions between feature points as the feature amount 513.

Furthermore, for example, if the type of the second object image 512 is text, the second feature detection unit 504 detects the text shown by the second object image 512, the shape of the edges of the area showing the text, etc. as feature amounts 514.

In step S607, the determination unit 505 calculates the degree of agreement 515 between the feature 513 detected in step S604 and the feature 514 detected in step S606. For example, the determination unit 505 calculates the degree of agreement 515 between the feature 513 and the feature 514 using pattern matching, deep learning, or the like. The information processing device 100 may learn the feature 513 in advance, and then the determination unit 505 may calculate the degree of agreement 515 between the feature 513 and the feature 514 using pattern matching, deep learning, or the like. Alternatively, a server device different from the information processing device 100 may learn the feature 513 in advance. In this case, the information processing device 100 may connect to the server device via a network and acquire the learned feature 513 from the server device. The control unit 401 then proceeds to step S701, as illustrated in FIG. 7.

Next, we will continue to explain the operation of the information processing device 100 with reference to Figure 7.

In step S701, the determination unit 505 determines whether the degree of match 515 calculated in step S605 illustrated in FIG. 6 exceeds the determination threshold. If the degree of match 515 does not exceed the determination threshold in step S701, the notification unit 507 causes the display unit 103 to display the second image 412 in step S702. The control unit 401 then terminates the process of determining whether the second object image 512 having a degree of match 515 with the features of the first object image 511 that exceeds the determination threshold is included. On the other hand, if the degree of match 515 exceeds the determination threshold in step S701, the determination unit 505 determines in step S703 that the second object image 512 having a degree of match 515 with the features of the first object image 511 that exceeds the determination threshold is included in the second image 412.

In step S704, the image processing unit 506 hides the second object image 512 that has a similarity 515 with the features of the first object image 511 that exceeds the judgment threshold. Specifically, the image processing unit 506 performs image processing on the second image 412 to hide the second object image 512. For example, the image processing unit 506 performs image processing such as stamping to superimpose a predetermined image, filling in with a predetermined color, or retouching to make the second object image 512 invisible by changing the brightness and saturation in the area of the second image 412 that includes the second object image 512. In this way, the image processing unit 506 hides the second object image 512 included in the second image 412. In other words, the image processing unit 506 makes the second object image 512 included in the second image 412 invisible.

In step S705, the notification unit 507 causes the display unit 103 to display an image in which the second object image 512 contained in the second image 412 and having a degree of similarity 515 with the features of the first object image 511 that exceeds the judgment threshold is hidden. This allows the information processing device 100 to prevent the image of the object present in the first imaging range 211 from being displayed on the display unit 103. In other words, the information processing device 100 can prevent the second image 412 including the image of the object present in the first imaging range 211 from being output.

For example, if the feature amount 513 and the feature amount 514 indicate facial features, the information processing device 100 can prevent the second image 412 that includes an image of the face of a person present in the first imaging range 211 from being output. For example, the information processing device 100 can prevent the second image 412 that includes an image of the photographer's face from being output.

Similarly, when feature amount 513 and feature amount 514 indicate the characteristics of a specific facial expression, the information processing device 100 can prevent the second image 412 including an image of a face with that expression from being output when the facial expression of the photographer present in the first imaging range 211 is an expression that the photographer does not intend.

Furthermore, for example, the information processing device 100 can prevent the output of a second image 412 that includes a second object image 512 such as a vehicle license plate, a sign, or a sign indicating an address. This allows the information processing device 100 to prevent the output of a second image 412 that includes information that can identify an individual or information that can identify the shooting location.

Also, for example, suppose that a photographer uses a video distribution service to distribute a second image 412 captured of the second image capture range 212 via the Internet through so-called live streaming. In this case, it is difficult for the photographer to perform editing work to hide images of objects included in the first image capture range 211 from the second image 412. However, the information processing device 100 performs image processing to hide second object images 512 that have a similarity 515 with the characteristics of the first object image 511 that exceeds the judgment threshold without any operation by the photographer. This eliminates the need for the photographer to perform editing work to hide images of objects included in the first image capture range 211 from the second image 412. Therefore, the information processing device 100 can prevent images of inappropriate objects that the photographer did not intend to include in the second image 412, without requiring the photographer to perform complicated work. As a result, the information processing device 100 allows the photographer to safely use video distribution services, etc., without requiring the photographer to perform complicated work.

In step S706, the notification unit 507 outputs a message indicating that the second image 412 contains a second object image 512 that has a degree of similarity 515 with the features of the first object image 511 that exceeds the determination threshold. In other words, the notification unit 507 notifies the photographer that an inappropriate object that was not intended by the photographer is reflected in the second image 412.

For example, the notification unit 507 causes the display unit 103 to display a message indicating that a second object image 512 having a degree of similarity 515 with the features of the first object image 511 that exceeds a determination threshold is included. Alternatively, if the information processing device 100 is equipped with a speaker (not shown), the notification unit 507 may output a sound indicating that the second object image 512 is included.

By notifying the photographer that an unintended inappropriate object has been captured, the information processing device 100 can prompt the photographer to adjust the second imaging range 212 so that the unintended inappropriate object is not captured.

Figure 8A is a diagram showing an example of a first image 411. For example, the first object image 511 included in the first image 411 shown in Figure 8A is an image of the photographer's face. Figure 8B is a diagram showing an example of a second image 412. The second image 412 shown in Figure 8B includes a second object image 512, which is an image of the photographer's face, in the portion of the reflective surface image 801. In other words, the second object image 512 shown in Figure 8B is an image of the photographer's face reflected in a reflective surface such as a mirror included in the second imaging range 212.

For example, suppose the second object image 512 illustrated in FIG. 8B has a feature with a degree of similarity 515 that exceeds the determination threshold with respect to the feature amount of the first object image 511 illustrated in FIG. 8A. In this case, the image processing unit 506 performs image processing to hide the second object image 512 included in the image 801 of the reflecting surface. Note that FIG. 8B does not illustrate the information processing device 100 being reflected on the reflecting surface.

Figure 9 is a diagram showing an example of a display image 901 displayed on the display unit 103. Specifically, the display image 901 includes a mask 902 that fills in the area including the second object image 512, as illustrated in Figure 8B. For example, the image processing unit 506 fills in the area including the second object image 512, as illustrated in the display image 901. The notification unit 507 then causes the display unit 103 to display an image in which the mask 902 is superimposed on the area including the second object image 512, as illustrated in the display image 901. Furthermore, as illustrated in the display image 901, the notification unit 507 may cause the display unit 103 to display an image in which a message 903 stating "Inappropriate material is reflected" is superimposed on the second image 412.

(Modification)
The position where the first imaging unit 102 is arranged is not limited to the front surface 105. For example, the first imaging unit 102 may be arranged on the top surface of the housing 101 facing the +Y direction. Similarly, the position where the second imaging unit 201 is arranged is not limited to the back surface 202. For example, the second imaging unit 201 may be arranged on the side surface of the housing 101 facing the +Z direction.

Second Embodiment
The second embodiment will be described with reference to Figures 10 to 16. In the drawings, the same or equivalent elements are designated by the same reference numerals, and duplicated explanations will be omitted. Configurations and processes having substantially the same functions as those of the first embodiment will be referred to by the same reference numerals, and explanations will be omitted, and differences from the first embodiment will be described.

Figure 10 is a diagram showing an example of the configuration of the control unit 401 according to this embodiment. The control unit 401 illustrated in Figure 10 differs from the control unit 401 illustrated in Figure 5 in that the control unit 401 illustrated in Figure 10 includes a first feature detection unit 1001, a second feature detection unit 1002, and a determination unit 1003 instead of the first feature detection unit 503, the second feature detection unit 504, and the determination unit 505.

The first feature detection unit 1001 calculates the rate of change 1011 of the distance between two or more feature points detected from the first object image 511. The second feature detection unit 1002 calculates the rate of change 1012 of the distance between two or more feature points detected from the second object image 512.

The determination unit 1003 calculates the degree of match 515 based on the result of comparing the movement of the first object image 511 with the movement of the second object image 512. Specifically, the determination unit 1003 calculates the degree of match 515 based on the result of comparing the rate of change 1011 of the distance between two or more feature points detected from the first object image 511 with the rate of change 1012 of the distance between two or more feature points detected from the second object image 512.

When it is determined that the second image 412 includes a second object image 512 having a degree of similarity 515 with the movement characteristics of the first object image 511 that exceeds the determination threshold, the notification unit 507 according to this embodiment outputs a message indicating that the second image 412 includes a second object image 512 having a degree of similarity 515 with the movement characteristics of the first object image 511 that exceeds the determination threshold. Alternatively, the notification unit 507 according to this embodiment may superimpose characters, symbols, etc. indicating the degree of similarity 515 for the movement of each object image included in the second image 412 on the display unit 103 and display them.

Figure 11 is a flowchart showing an example of the operation of the information processing device 100 according to this embodiment. The processing in steps S1101 and S1102 is similar to the processing in steps S601 and S602 illustrated in Figure 6, so a detailed description will be omitted.

In step S1103, the first object detection unit 501 detects a first object image 511 from the first image 411 acquired in step S1101. For example, the first object detection unit 501 detects an object image of a predetermined type included in the first image 411 as the first object image 511. For example, the predetermined type may be a human face, a human body, etc. Alternatively, for example, the first feature detection unit 1001 may perform region division in the first image 411 and detect the first object image 511. In this case, the first feature detection unit 1001 does not limit the type of the first object image 511, but detects multiple types of object images included in the first image 411.

In step S1104, the first feature detection unit 1001 detects two or more feature points from the first object image 511. For example, if the type of the first object image 511 is a human face, the first feature detection unit 1001 detects the coordinate values of the edges of the eyes, nose, mouth, etc. included in the first object image 511 as feature points. Also, for example, if the type of the first object image 511 is a human body, the first feature detection unit 1001 detects the coordinate values of the arms, legs, etc. included in the first object image 511 as feature points. Alternatively, if the type of the first object image 511 is an image of a human body, the first feature detection unit 1001 may detect representative values of the coordinate values of the arms, legs, etc. included in the first object image 511 as feature points.

In step S1105, the first feature detection unit 1001 calculates the rate of change 1011 of the distance between the feature points detected in step S1104. For example, the first feature detection unit 1001 calculates the rate of change 1011 from the rate of change over time of the length of each side for a polygon having two or more feature points as vertices detected in step S1104. For example, the first feature detection unit 1001 calculates the representative value of the rate of change over time of the length of each side for a polygon having two or more feature points as the rate of change 1011.

In step S1106, the second object detection unit 502 detects a second object image 512 from the second image 412 acquired in step S1102. In step S1107, the second feature detection unit 1002 detects two or more feature points from the second object image 512. For example, if the type of the second object image 512 is a human face, the second feature detection unit 1002 detects the coordinate values of the edges of the eyes, nose, mouth, etc. included in the second object image 512 as feature points. Also, for example, if the type of the second object image 512 is a human body, the second feature detection unit 1002 detects the coordinate values of the arms, legs, etc. included in the second object image 512 as feature points. Alternatively, if the type of the second object image 512 is a human body image, the second feature detection unit 1002 may detect representative coordinate values of the arms, legs, etc. as feature points.

In step S1108, the second feature detection unit 1002 calculates the rate of change 1012 of the distance between the feature points detected in step S1107. For example, the second feature detection unit 1002 may calculate the rate of change 1012 from the rate of change over time of the length of each side for a polygon having vertices at two or more feature points detected in step S1107. For example, the second feature detection unit 1002 calculates, as the rate of change 1012, a representative value of the rate of change over time of the length of each side for a polygon having vertices at two or more feature points detected in step S1107.

In step S1109, the determination unit 505 calculates the degree of match 515 based on the comparison result between the rate of change 1011 calculated in step S1105 and the rate of change 1012 calculated in step S1108. The control unit 401 then proceeds to S701, as shown in FIG. 7.

Figure 12A is a diagram showing an example of a person 1205 in the first imaging range 1203, reflected on the reflecting surface 1201. Person images 1202a and 1202b shown in Figure 12A show images of the person 1205 with different facial expressions, reflected on the reflecting surface 1201 in the second imaging range 212. As shown in Figure 12A, the facial expression shown by person image 1202a changes to the facial expression shown by person image 1202b.

Figure 12B shows examples of second images 412a and 412b captured in the second imaging range 1204 shown in Figure 12A. The second image 412a shown in Figure 12B includes a second object image 512a showing the face of the person image 1202a. The second image 412b shown in Figure 12B includes a second object image 512b showing the face of the person image 1202b. The rate of change 1011 of the distance between feature points in the first object image 511, which is the image of the person 1205, is equivalent to the rate of change 1012 of the distance between feature points from the second object image 512a to the second object image 512b. For example, the rate of change 1011 of the distance between feature points at the edges of the mouth in the first object image 511 is equivalent to the rate of change 1012 of the distance between feature points at the edges of the mouth from the second object image 512a to the second object image 512b.

The determination unit 505 therefore compares the time-series rate of change 1011 of the distance between feature points in the first object image 511 with the time-series rate of change 1012 of the distance between feature points from the second object image 512a to the second object image 512b, and determines that the person 1205 present in the first imaging range 1203 is reflected in the second imaging range 212.

Furthermore, by comparing the time-series rate of change 1011 of the distance between feature points in the first object image 511 with the time-series rate of change 1012 of the distance between feature points in the second object image 512, the information processing device 100 can determine whether an object included in the first image capture range 211 is reflected in the second image capture range 212, even if the angle of the first object image 511 and the angle of the second object image 512 are different. For example, by comparing the time-series rate of change 1011 of the distance between feature points in the first object image 511 with the time-series rate of change 1012 of the distance between feature points in the second object image 512, the information processing device 100 can identify the image of the photographer's face from the images of multiple objects reflected in the second image capture range 212, even if the angle of the image of the photographer's face, which is the first object image 511, is different from the angle of the image of the photographer's face, which is the second object image 512. In other words, if the information processing device 100 can detect global features from the first image 411 and the second image 412, it can identify the image of the photographer's face reflected in the second imaging range 212 even if the angle of the first object image 511 and the angle of the second object image 512 are different.

Similarly, if the information processing device 100 can detect global features from the first image 411 and the second image 412, it can identify an object reflected in the second imaging range 212 even if the brightness of the first object image 511 and the brightness of the second object image 512 are different.

As described above, the information processing device 100 can identify an object present in the first image capture range 211 that is reflected in the second image capture range 212 by suppressing the effects of differences in the angle and color tone of the object image between the first image 411 and the second image 412. In other words, the information processing device 100 can identify an object present in the first image capture range 211 that is reflected in the second image capture range 212 by suppressing the effects of differences in the angle and color tone of the object image between the first image 411 and the second image 412.

Figure 13 is a diagram showing an example of a person 1301 present within the first imaging range 211, an image of the person 1301 reflected on a reflective surface 1302 within the second imaging range 212, and people 1303 and 1304 present within the second imaging range 212. When the person 1301 shown in Figure 13 moves, the first imaging unit 102 captures an image of the moving person 1301, and the second imaging unit 201 captures an image of the moving person 1301 reflected on the reflective surface 1302. Similarly, when the people 1303 and 1304 shown in Figure 13 move, the second imaging unit 201 captures images of the moving people 1303 and 1304. Note that Figure 13 does not show the information processing device 100 being reflected on the reflective surface 1302.

Figure 14A shows an example of a first image 411 including a first object image 1411, which is an image of the person 1301 shown in Figure 13. Figure 14B shows an example of a second image 412 captured with the person 1301 reflected in the second imaging range 212. The second image 412 shown in Figure 14B includes a second object image 512, which is an image of the person 1301. Furthermore, the second image 412 shown in Figure 14B includes object images 1401 and 1402, which are images of people present in the second imaging range 212. Note that, as with Figure 13, Figure 14B does not illustrate the information processing device 100 being reflected in the reflective surface 1302.

The change in movement of the first object image 511 illustrated in FIG. 14A is equivalent to the change in movement of the second object image 512 illustrated in FIG. 14B. Therefore, the rate of change 1011 of the distance between feature points in the first object image 511 illustrated in FIG. 14A is equivalent to the rate of change 1012 of the distance between feature points in the second object image 512 illustrated in FIG. 14B.

For example, the determination unit 505 calculates the degree of match 515 based on the comparison result between the rate of change 1011 of the distance between the feature points of the arms in the first object image 511 and the rate of change 1012 of the distance between the feature points of the arms in the second object image 512. If the degree of match 515 exceeds the determination threshold, the determination unit 505 determines that the second image 412 includes an image of the person represented by the first object image 511.

Figure 15 is a diagram showing an example of a display image 1501 displayed on a display unit. Specifically, the second object image 512 shown in Figure 14B is not displayed in the display image 1501. Specifically, a mask 1502 is superimposed on the area including the second object image 512 in the display image 1501. Therefore, the second object image 512 is not displayed in the display image 1501. This enables the information processing device 100 to prevent the output of an image including an object image whose degree of similarity 515 with the movement of the first object image 511 exceeds the determination threshold.

FIG. 16 is a diagram showing an example of a display image 1601 displayed on the display unit 103. Specifically, in the display image 1601, characters indicating the degree of correspondence 515 between the movement of each of the second object image 512, object image 1401, and object image 1402 and the movement of the first object image 511 are superimposed on the second image 412 shown in FIG. 14B. The degree of correspondence 515 between the movement of the first object image 511 and the movement of the second object image 512 shown in FIG. 14A is 100%. The degree of correspondence 515 between the movement of the first object image 511 and the movement of the object image 1401 shown in FIG. 14A is 60%. The degree of correspondence 515 between the movement of the first object image 511 and the movement of the object image 1402 shown in FIG. 14A is 50%.

In this way, by displaying the display image 1601 on the display unit 103, the information processing device 100 can present the degree of similarity 515 between the movement of an object present in the first imaging range 211 and the movement of an object present in the second imaging range 212. This allows the information processing device 100 to allow the photographer to recognize which of the multiple object images included in the second image 412 is an image of an object present in the first imaging range 211.

This disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. Furthermore, new technical features can be created by combining the technical means disclosed in each embodiment.

100 Information processing device, 101 Housing, 102 First imaging unit, 103 Display unit, 104 Touch panel, 105 Front, 201 Second imaging unit, 202 Rear, 211 First imaging range, 212 Second imaging range, 301 Person, 302 Reflective surface, 401 Control unit, 402 ROM, 403 RAM, 411 First image, 412 Second image, 412a Second image, 412b Second image, 501 First object detection unit, 502 Second object detection unit, 503 First feature detection unit, 504 Second feature detection unit, 505 Determination unit, 506 Image processing unit, 507 Notification unit, 511 First object image, 512 Second object image, 512a Second object image, 512b Second object image, 513 Feature amount, 514 Feature amount, 515 Matching ratio, 801 Image, 901 Display image, 902 Mask, 903 Message, 1001 First feature detection unit, 1002 Second feature detection unit, 1003 Determination unit, 1011 Change rate, 1012 Change rate, 1201 Reflecting surface, 1202a Image, 1202b Image, 1204 First imaging range, 1205 Person, 1301 Person, 1302 Reflecting surface, 1303 Person, 1304 Person, 1401 Object image, 1402 Object image, 1501 Display image, 1502 Mask, 1601 Display image

Claims (11)

a first imaging unit that captures an image of a first imaging range to obtain a first image;
a second imaging unit that captures an image of a second imaging range that is outside the first imaging range and acquires a second image;
a determination unit that determines whether or not a second object image having a feature that matches a feature of a first object image included in the first image and that exceeds a determination threshold,
the determination unit calculates the degree of coincidence based on a comparison result between a time-series rate of change in the distance between two or more feature points in the first object image and a time-series rate of change in the distance between two or more feature points in the second object image. a notification unit that outputs, when it is determined that the second image includes the second object image, a message indicating that the second image includes the second object image.
The information processing device according to claim 1 . an image processing unit that performs image processing on the second image to hide the second object image when it is determined that the second image includes the second object image;
3. The information processing device according to claim 1. further comprising a display unit that displays the second image;
when it is determined that the second image includes the second object image, the image processing unit causes the display unit to display an image in which the second image does not include the second object image.
The information processing device according to claim 3 . the determination unit calculates the degree of coincidence for each of the object images based on a comparison result between the movement of the first object image and the movement of each of the object images included in the second image;
a display unit that displays the second image;
The information processing apparatus according to claim 1 , further comprising: a notification unit that causes information indicating the degree of coincidence for each of the object images to be superimposed on the second image and displayed on the display unit. further comprising a housing having a front surface and a back surface;
the first imaging range is in a direction in which the front faces,
The information processing device according to claim 1 , wherein the second imaging range is in a direction in which the rear surface faces. capturing an image of a first imaging range to obtain a first image;
capturing an image of a second imaging range outside the first imaging range to acquire a second image;
determining whether or not a second object image having a feature that matches a feature of a first object image included in the first image to a degree that exceeds a determination threshold is included in the second image;
The image determination method includes a step of calculating the degree of match based on a comparison result between a time-series rate of change in the distance between two or more feature points in the first object image and a time-series rate of change in the distance between two or more feature points in the second object image. On the computer,
a function of capturing an image of a first imaging range to acquire a first image;
a function of capturing an image of a second imaging range outside the first imaging range to acquire a second image;
a function of determining whether or not a second object image having a feature that matches a feature of a first object image included in the first image to a degree exceeding a determination threshold is included in the second image;
The determining function is a program that executes a function of calculating the degree of coincidence based on a comparison result between a time-series rate of change in the distance between two or more feature points in the first object image and a time-series rate of change in the distance between two or more feature points in the second object image. a first imaging unit that captures an image of a first imaging range to obtain a first image;
a second imaging unit that captures an image of a second imaging range that is outside the first imaging range and acquires a second image;
a determination unit that determines whether a second object image having a feature that matches a feature of a first object image included in the first image and that exceeds a determination threshold is included in the second image;
a display unit that displays the second image ;
Notification department and preparation,
the determination unit calculates the degree of coincidence for each of the object images based on a comparison result between the movement of the first object image and the movement of each of the object images included in the second image ;
The notification unit is an information processing device that causes the display unit to display information indicating the degree of coincidence for each of the object images superimposed on the second image . capturing an image of a first imaging range to obtain a first image;
capturing an image of a second imaging range outside the first imaging range to acquire a second image;
determining whether or not a second object image having a feature that matches a feature of a first object image included in the first image and that exceeds a determination threshold is included in the second image;
displaying the second image on a display unit ;
the determining step calculates the degree of coincidence for each of the object images based on a comparison result between the movement of the first object image and the movement of each of the object images included in the second image ;
In the displaying step, information indicating the degree of coincidence for each of the object images is superimposed on the second image and displayed on the display unit . On the computer,
a function of capturing an image of a first imaging range to acquire a first image;
a function of capturing an image of a second imaging range outside the first imaging range to acquire a second image;
a function of determining whether a second object image having a feature that matches a feature of a first object image included in the first image and that exceeds a determination threshold is included in the second image;
and a function of displaying the second image on a display unit.
the function of determining includes a function of calculating the degree of coincidence for each of the object images based on a comparison result between the movement of the first object image and the movement of each of the object images included in the second image,
a program for displaying, in the display function, information indicating the degree of coincidence for each of the object images superimposed on the second image on the display unit;