JP7214092B2 - Image control device and its program - Google Patents

Description

The present invention relates to an image control device and its program.

2. Description of the Related Art In recent years, with the development of mobile terminals, there are many techniques for synthesizing and displaying image contents such as characters and stamps with images captured by cameras installed in mobile terminals. Many of these are widely used as applications for mobile terminals that allow image content to be arranged in images captured by cameras of mobile terminals. In such an application, image content whose size and position are adjusted by the user is superimposed on the image captured by the camera and displayed on the display screen.

Another technique is disclosed in Patent Document 1. Japanese Patent Laid-Open No. 2005-100001 describes a technique of displaying image content on a display screen and photographing the user together with the user when the distance between the faces of the users displayed on the display screen is closer than a predetermined distance. .

JP 2018-23056 A

For example, if the user takes a picture of the image content of the character and a two-shot picture, the user needs to adjust the size and position so that the user's image fits in the application for mobile terminals described above.

Moreover, when using the technique of Patent Document 1, the user himself/herself needs to adjust the standing position so that the user's size and standing position match the character.

Therefore, the problem to be solved by the present invention is to solve the above-mentioned problems, and to provide a technique for generating an image in which the user appears together with the image content without the user adjusting the image content. .

An image control apparatus according to the present invention for solving the above problems includes an estimation means for estimating a posture of a person based on a skeleton of the person detected from a captured image, and acquiring a character image in a pose corresponding to the estimated posture. an acquiring means for converting the size of the acquired character image in accordance with the vertical ratio of the detected skeleton on the captured image, and displaying a superimposed image in which the converted character image is superimposed on the captured image. and display control means for causing the display to be displayed.

According to the present application, an image can be generated in which the user is shown with the image content without the user adjusting the image content.

1 is a schematic diagram of an image control device; FIG. 1 is an example of a block diagram of an image control device; FIG. It is an example of a character database. It is an example of a posture model database. It is an example of a character image database. It is a figure for demonstrating a key point. FIG. 4 is a diagram showing an example of a hierarchical structure of keypoints; It is a figure for demonstrating the operation|movement of this invention.

In order to explain the features of the present invention, specific description will be made below with reference to the drawings.

The image control device 10 of the present invention is a device for providing a photograph in which a user is photographed together with a character. The image control device 1 prepares images of characters in a plurality of poses in advance, adjusts the image of the character in a pose that matches the posture of the user, and adjusts the image of the character with the user. Generate photos. It should be noted that the user's posture described in the present invention is a general concept including the user's posture, pose, and attitude.

FIG. 1 is an image diagram showing an example in which an image control device 1 of the present invention is used. In the example shown in FIG. 1, a device having a display screen of approximately the same size as a human being, such as digital signage equipped with a camera, is used for explanation, but a smartphone is used as the image control device of the present invention. be good.

As shown in the block diagram of the image control device 10 in FIG. , an image capture unit 17 and a display unit 20 . The display unit 20 has a display screen 21 equipped with, for example, a touch panel.

The storage unit 11 has a character database 111 , posture model data 112 , a character image database 113 and a screen database 114 .

In the character database 111, as shown in FIG. 3, a character ID that uniquely identifies a character and a character name are associated with each other and stored.

The posture model database 112 stores model data of various postures of a person, and is used to estimate the posture of a person (user) in an image captured by the imaging unit 11 . As shown in FIG. 4, the posture model database 112 stores posture IDs and model data in association with each other. The posture ID of the posture model database 112 is an identification number that uniquely identifies model data. The model data contains a list of conditions for estimating the user's posture.

In the character image database 113, as shown in FIG. 5, character images in different poses are associated with posture IDs and stored for each character.

The display screen database 114 stores screen data of menu screens necessary for the user to shoot.

The imaging unit 12 may be provided with a special device for photographing the user, or may be a general digital camera equipped with a CCD, CMOS, or the like. In the present invention, the file format of the image captured by the imaging unit 11 may be JPEG (Joint Photographic Experts Group) or the like. The imaging unit 12 is preferably arranged along the outer periphery of the display screen of the display device 20 .

The estimation unit 13 estimates the posture of the person from the image captured by the imaging unit 12 . The posture estimated by the estimating unit 13 is predetermined, and the estimating unit 13 estimates the posture based on an estimation condition defined for each posture to be estimated. A method for estimating a person's posture from an image is not particularly specified in the present invention, but an example will be described below.

The estimation unit 13 first recognizes the skeleton of the person when estimating the posture of the person from the image captured by the imaging unit 12 . The estimating unit 13 detects key points that can be feature points from the image of the person when recognizing the skeleton. The keypoints include, for example, at least one of the centers of body parts, connection points between body parts, and joints. FIG. 6 shows an example in which the following (1) to (19) including the center of body parts, connection points between body parts, and joints are detected as key points.
(1) JOINT HEAD (center of head)
(2) JOINT NECK (neck center)
(3) JOINT LEFT AND RIGHT COLLAR (sternoclavicular joint)
(4) JOINT TRSO (central trunk)
(5) JOINT WAIST (center of waist)
(6) JOINT RIGHT SHOLDER (right shoulder joint)
(7) JOINT LEFT SHOLDER (left shoulder joint)
(8) JOINT RIGHT ELBOW (right elbow joint)
(9) JOINT LEFT ELBOW (left elbow joint)
(10) JOINT RIGHT WRIST (right wrist joint)
(11) JOINT LEFT WRIST (left wrist joint)
(12) JOINT RIGHT HAND (Right hand joint)
(13) JOINT LEFT HAND (left hand joint)
(14) JOINT RIGHT HIP (right hip joint)
(15) JOINT LEFT HIP (left hip joint)
(16) JOINT RIGHT KNEE (right knee joint)
(17) JOINT LEFT KNEE (left knee joint)
(18) JOINT RIGHT ANKLE (right ankle joint)
(19) JOINT LEFT ANKLE (left leg joint)
Each keypoint has a hierarchical structure in which a parent-child relationship is established in advance with other keypoints. An example of the hierarchical structure of keypoints is shown in FIG. JOINT HEAD(1) is the top of the hierarchical structure. JOINT HEAD (1) is at the top, JOINT NECK (2), JOINT LEFT AND RIGHT COLLAR (3), and the parent-child relationship is connected to the lower layer in order. In JOINT LEFT AND RIGHT COLLAR (3), the child key points are JOINT TRSO (4), JOINT RIGHT SHOLDER (6), and JOINT LEFT SHOLDER (7).

JOINT TRSO (4) has a parent-child relationship with JOINT WAIST (5), and JOINT WAIST (5) has a parent-child relationship with JOINT RIGHT HIP (14) and JOINT LEFT HIP (15). JOINT RIGHT HIP (14) has itself as the upper layer, and the parent-child relationship is connected to the lower layers in the order of JOINT RIGHT KNEE (16) and JOINT RIGHT ANKLE (18). JOINT LEFT HIP, like JOINT RIGHT HIP (14), has itself as the upper layer, and the parent-child relationship is connected to the lower layers in the order of JOINT LEFT KNEE (17) and JOINT LEFT ANKLE (19).

JOINT RIGHT SHOLDER (6) has itself as a higher layer, and the parent-child relationship is connected to the lower layers in the order of JOINT RIGHT ELBOW (8), JOINT RIGHT WRIST (10), and JOINT RIGHT HAND (12).

JOINT LEFT SHOLDER (7), like JOINT RIGHT SHOLDER (6), has a parent-child relationship in the order of JOINT LEFT ELBOW (9), JOINT LEFT WRIST (11), and JOINT LEFT HAND (13).

FIG. 7 is a diagram showing a tree structure in which the key points having the parent-child relationship as described above are connected with straight lines. In FIG. 6 as well, the key points having a parent-child relationship are shown by connecting them with straight lines. The keypoints and tree structure shown in FIGS. 6 and 7 are examples, and the present invention can be applied to any location and structure of the keypoints and tree structure.

The estimation unit 13 recognizes straight lines between key points in a parent-child relationship as skeletons, and determines whether a predetermined angle between skeletons satisfies the conditions described in the model data of the posture model database 112 . The estimating unit 13 calculates the ratio of the predetermined interskeletal angles matching the condition list, and estimates that the posture associated with the highest ratio is the user's posture. That is, the degree of approximation between the recognized skeleton and the model data is calculated, and the posture of the closest model data is estimated to be the posture of the user.

Although the method by which the estimating unit 13 detects the human joints from the image is not specified, for example, a machine learning device may be used. Even if the machine learning device detects the joints by using the relationship between the person and the joint obtained by specifying the joint of the person in the image as training data, the machine learning device acquires the relationship between the person and the joint by deep learning and uses it. Joints may be detected using this as teacher data.

The acquiring unit 14 acquires a character image of a pose associated with the posture estimated by the estimating unit 13 from the character image database 113 .

The display control unit 15 reads the menu screen from the display image database 114 and displays it on the display screen 21 . The display control unit 15 reads the character ID of the character database 111 when displaying the character selection menu screen. A character name associated with the read character ID is read from the character database 111 . Further, the display control unit 15 reads out the image of the read character ID from the character image database 113 and displays it on the display screen 21 side by side with the character name read out previously. The image read out from the character image database 113 may be predetermined or random.

The display control unit 15 superimposes (combines) the character image selected by the acquisition unit 14 on the image captured by the image capturing unit 12 and displays it on the display screen 21 . When superimposing the character image, the display control unit 15 converts the size of the selected character image according to the vertical ratio of the skeleton recognized by the estimation unit 13 on the captured image. The display control unit 15 arranges the character image whose size has been converted in the captured image at a position corresponding to at least one of the horizontal and vertical (depth) positions on the image of the skeleton recognized by the estimation unit 13. do. An example of character image position determination performed by the display control unit 15 will be described.

For example, the image is divided into a plurality of areas such as a right area and a left area, and the display control unit 15 arranges the character image in an area other than the area where the estimated skeleton is located. In addition, the posture ID of the posture model database 112 is previously associated with placement information indicating the position on the screen to be stored, and the display control unit 15 places the character image based on this placement information.

As another example, a skeleton is also defined for the character image of each pose, and the relative position of the skeleton of the character image with respect to the skeleton of the recognized user is associated with the posture ID of the character image database 113 and stored as arrangement information. Then, the display control unit 15 arranges the character image based on this arrangement information, the skeleton of the character, and the skeleton of the user.

The display control unit 15 of the present invention arranges the character image in the captured image using any of the above methods. The display control unit 15 may change the angle at which the character image is arranged according to the recognized skeleton.

The depth sensor 16 uses the captured image from the imaging unit 12 to detect that the user performs a predetermined action. For example, the depth sensor 16 periodically measures the distance to the object to generate a depth image, detects the shape of the user corresponding to the person from the depth image, and detects the user's predetermined action according to the time-series change of the shape. Monitor what you've done.

When the depth sensor detects a predetermined action of the user, the image capture unit 17 captures a composite image in which the display control unit 15 superimposes the character image on the captured image. The image capture unit 17 stores the captured image in the storage unit 11 as a photographed image. Note that the self-timer may be activated when the depth sensor detects a predetermined action of the user, and the image capture unit 17 may capture an image after the countdown.

Next, the operation of this embodiment will be described. FIG. 8 is a flow chart for explaining the operation of the present invention.

The display control unit 15 generates a menu screen for allowing the user to select a character from the character name and the image of the character read from the character database 111 and the character image database 113, and the menu screen read from the display screen database 114. to display it on the display screen 21 (step 1501).

When the user selects a character using the menu screen displayed on the display screen 21, the input section notifies the acquisition section 14 of the character ID of the selected character (step 1502).

When the character is selected, the camera is activated and the captured image captured by the imaging section 12 is displayed on the display screen 21 (step 1503).

The estimation unit 13 recognizes the user's skeleton from the captured image (step 1504).

The estimating unit 13 calculates the rate at which the angles between predetermined skeletons of the recognized skeletons match the conditions described in the model data of the posture model database 112 (step 1505).

The estimation unit 13 reads the posture ID associated with the model data with the highest ratio (step 1506).

The acquiring unit 14 acquires a character image from the character image database 113 based on the character ID of the character selected in step 1502 and the posture ID read by the estimating unit 13 (step 1507).

The display control unit 15 changes the size of the character image selected in step 1507 according to the vertical ratio of the skeleton on the captured image recognized in step 1504 (step 1508).

The display control unit 15 arranges the character image whose size has been converted in step 1508 in the captured image at a position corresponding to at least one of the horizontal and depth positions on the skeleton image recognized in step 1504. It is displayed on the display screen 21 (step 1509).

The depth sensor 16 starts monitoring and detects that the user performs a predetermined action (step 1510).

When the depth sensor 16 detects a predetermined action of the user, the image capture unit 17 captures the image displayed on the display screen 21 (step 1511).

The image capture unit 17 stores the captured image in the storage unit 11 (step 1522).

When the present invention described above is used, the character image is arranged according to the posture of the user, so an image in which the user is shown together with the character image can be taken without the user adjusting the image content.

Further, according to the present invention, since the size of the character image is changed according to the screen size of the user, it is possible to take an image of the user together with the character image without adjusting the standing position of the user. .

Moreover, when the present invention is used, the screen is captured when a predetermined action of the user is detected, so the user can take an image together with the character image without moving to the touch panel of the display screen.

In the above description, the case of estimating the posture is described using the case where the angle between the predetermined skeletons is estimated based on the ratio of the conditions that meet the angle condition list described in the model data, but there are other methods. can be For example, the relative positional relationship of the recognized skeleton may be estimated based on the rate of similarity to the positional relationship described in the model data.

Also, in the above description, the acquisition unit acquires the character image of the pose associated with the estimated posture from the character image database 113, but other methods may be used. For example, the obtaining unit may change the image of the character to a pose corresponding to the estimated posture.

Also, in the above description, an example in which the depth sensor 16 is used to detect the user performing a predetermined action has been used, but other methods may be used. For example, instead of the depth sensor, an optical sensor may be used to detect that the user performs a predetermined action. Alternatively, it may be detected whether or not the user has performed a predetermined action based on a time-series change in the skeleton recognized by the estimation unit 13 .

The apparatus of the present invention described above can be configured with hardware, but can also be realized with a computer program. In this case, the image control device 10 has hardware resources such as a processor, memory (ROM or RAM), storage device (hard disk, semiconductor disk, etc.), input device (keyboard, mouse, touch panel), display device, and communication unit. It can also be realized by a general-purpose computer. It is also possible to realize only part of the functions by a computer program.

Preferred embodiments and examples have been described above. However, the present invention is not necessarily limited to the above-described embodiments and examples, and can be variously modified and implemented within the scope of its technical ideas.

Although the present invention has been described above with reference to the embodiments and examples, the present invention is not necessarily limited to the above-described embodiments and examples, and can be implemented in various modifications within the scope of the technical idea thereof. can do

10 image control device 10
11 storage unit 12 imaging unit 13 estimation unit 14 acquisition unit 15 display control unit 16 depth sensor 17 image capture unit 20 display unit 21 display screen

Claims (9)

selection means for selecting model data that approximates the posture of the person from model data of a plurality of types of postures that are predetermined to be estimated as the posture of the person, based on the skeleton of the person detected from the captured image; ,
acquisition means for acquiring a character image in a pose stored in association with the selected model data;
The size of the obtained character image is converted according to the vertical ratio of the detected skeleton on the captured image, and the converted character image is superimposed on an area of the captured image other than the area where the skeleton is located . display control means for displaying a superimposed image;
and image capturing means for capturing the superimposed image when a predetermined motion of the person is recognized. 2. An image control apparatus according to claim 1, wherein said selection means detects said skeleton based on feature points including positions of joints of said person from said captured image. 3. An image control apparatus according to claim 1, wherein said obtaining means changes the image of the character to a pose corresponding to the selected posture. 3. The image control apparatus according to claim 1, wherein said display control means determines a position where said acquired character image is superimposed according to a horizontal position of said detected skeleton on said captured image. 3. The image control apparatus according to claim 1, wherein said display control means determines a position where said acquired character image is superimposed based on layout information stored in association with said model data. 6. The display control means determines the vertical position of superimposing the acquired character image according to the vertical position of the detected skeleton on the captured image. image controller. 7. The image control apparatus according to claim 1, wherein said predetermined motion is recognized by time-series motion of the skeleton detected by said selection means. 7. The image control device according to any one of claims 1 to 6, wherein said predetermined motion is recognized by a depth sensor or an optical sensor. A program for an image control device, the program causing the image control device to:
selection means for selecting model data approximating the posture of the person from among a plurality of types of model data predetermined as the posture of the person to be estimated based on the skeleton of the person detected from the captured image;
acquisition means for acquiring a character image in a pose stored in association with the selected model data;
The size of the obtained character image is converted according to the vertical ratio of the detected skeleton on the captured image, and the converted character image is superimposed on an area of the captured image other than the area where the skeleton is located . display control means for displaying a superimposed image;
A program functioning as image capture means for capturing the superimposed image when the predetermined motion of the person is recognized.

Images