KR20210106763A - Electronic device and method for tracking movements of an object - Google Patents

Abstract

The present invention provides an electronic device and a method which can identify all exercise motions without posture limitations and obtain movement information of a joint by scanning a body area of a user in various postures. According to one embodiment of the present invention, the electronic device comprises at least one camera module, a memory, and a processor operatively connected to the at least one camera module and the memory. The processor can photograph an image of a first object, acquire color information of the photographed first object, start video recording for the first object and a second object moving on the first object in response to a video recording command, determine a first object image corresponding to the first object in data acquired by the video recording based on the color information of the first object, change three-dimensional coordinates of an area determined by the first object image in the data based on a designated distance value, and track a movement of the second object based on the data in which the three-dimensional coordinates of the first object image are changed.

Description

ELECTRONIC DEVICE AND METHOD FOR TRACKING MOVEMENTS OF AN OBJECT}

Various embodiments of the present document relate to a technique for tracking the motion of an object moving on the ground by independently extracting the object.

The sports industry and health care industry are gradually expanding into areas that take into account various age groups and various sports, centering on individual activities. As the sports and health care markets grow rapidly, related technology industries are also developing, and cutting-edge technologies are gradually being grafted together. This is due to the diversification of consumer demand for sporting goods, facilities, and services.

In particular, technologies related to motion tracking that can maximize the effect of movement through better movement or posture by realizing a person's movement or posture as an image are emerging.

As a motion tracking or motion tracking technology, there are a method of extracting an object on the screen as 3D information by analyzing the phase difference with a stereo camera, a method of tracking the user's motion by extracting the user's area with a general RGB camera, etc. have.

According to the prior art, when the user is attached to a wall or the ground or the phase difference is not large, it is difficult to extract the user's human body information. In particular, it was difficult to provide content as content because only extremely limited motions could be identified in exercises in which a significant portion of motions such as yoga and Pilates were performed in a prone, supine, or sitting position.

In addition, when the user's area is extracted with a general RGB camera according to the prior art, the left/right hand is not distinguished when the user stands sideways because there is not enough information on the x-axis, y-axis, and z-axis, so movement motion tracking There was something lacking as a system.

According to various embodiments of the present disclosure, it is possible to provide an electronic device and method capable of detecting all movement motions without a posture limitation and obtaining joint motion information by scanning user body regions in various postures.

The electronic device according to an embodiment of the present document includes at least one camera module, a memory, and a processor operatively connected to the at least one camera module and the memory, wherein the processor receives an image of a first object. photographing, obtaining color information of the photographed first object, and in response to a video recording command, starting video recording of the first object and a second object moving on the first object, Based on the color information, a first object image corresponding to the first object is determined from the data obtained by photographing the moving picture, and three-dimensional coordinates of an area determined as the first object image in the data are designated as a distance value and an electronic device for tracking the movement of the second object based on the data in which the three-dimensional coordinates of the first object image are changed.

The method of operating an electronic device according to an exemplary embodiment of the present document includes an operation of photographing an image of a first object, an operation of obtaining color (RGB) information of the photographed first object, and in response to a video recording command, the Initiating video recording of a first object and a second object moving on the first object, based on the color information of the first object, corresponding to the first object in data obtained by photographing the video An operation of determining a first object image, an operation of changing the three-dimensional coordinates of an area determined as the first object image in the data based on a specified distance value, an operation of changing the three-dimensional coordinates of the first object image based on the changed data It may include an operation of tracking the movement of the second object.

According to various embodiments disclosed in this document, information about motion may be extracted through a simple image pre-processing process while maintaining a motion tracking solution used in the past.

According to various embodiments disclosed in this document, by independently extracting an object, it is possible to extract an object without difficulty even in an operation in yoga or Pilates performed in close contact with an exercise mattress or the like.

According to various embodiments disclosed in this document, movement tracking or motion tracking is possible without limitation of movement motion, and thus various and effective programs and contents can be configured.

1 is a diagram illustrating a system for tracking a motion of an object according to an exemplary embodiment.
2 is a flowchart illustrating an overall flow of tracking a motion of an object in an electronic device, according to an exemplary embodiment.
3 is a block diagram of an electronic device 301 in a network environment 300 according to various embodiments of the present disclosure.
4 is a block diagram 400 illustrating a camera module 380, according to various embodiments.
5 is a flowchart illustrating a flow of pre-processing an image of a first object in an electronic device according to an exemplary embodiment.
6A is a diagram illustrating a state in which an image of a first object is pre-processed in an electronic device according to an exemplary embodiment;
6B is a diagram illustrating a state in which an image of a first object is pre-processed in an electronic device according to an exemplary embodiment.
7 is a flowchart illustrating a flow of determining a first object image in an electronic device according to an exemplary embodiment.
8 is a flowchart illustrating a flow of adjusting a distance value d of a first object image and tracking a motion of a second object based on location information in an electronic device, according to an exemplary embodiment.
9A is a diagram illustrating a concept of tracking a motion of a second object based on location information in an electronic device according to an exemplary embodiment.
9B is a diagram illustrating a concept of tracking a motion of a second object based on location information in an electronic device according to an exemplary embodiment.
9C is a diagram illustrating a concept of tracking a motion of a second object based on location information in an electronic device according to an exemplary embodiment.
10 is a diagram illustrating a concept of tracking a motion of a second object by using a change in a distance value d according to various embodiments of the present disclosure;

1 is a diagram illustrating a system for tracking a motion of an object according to an exemplary embodiment.

In an embodiment, the electronic device 110 may photograph objects including at least the first object 130 and the second object 140 . For example, the first object 130 may be a yoga mattress, an exercise mattress, a blanket, or the like. The second object 140 may be a moving subject (eg, a person, an animal, etc.). As a specific example, the electronic device 110 may photograph a person practicing yoga on a yoga mattress. The electronic device 110 may photograph surrounding objects, such as a desk, a vase, and a lamp, in addition to a yoga mattress and a person.

In an embodiment, the electronic device 110 may obtain color (RGB) information and location information for the first object 130 and the second object 140 . For example, the electronic device 110 may acquire color information of the first object (eg, yoga mattress, exercise mattress, blanket, etc.) by taking a picture using the camera module 380 of FIG. 4 , which will be described later. For another example, the electronic device 110 obtains color information and location information of a first object and color information and location information of a second object (eg, people, animals, etc.) by shooting a video based on the camera module 380 . can do.

In an embodiment, the electronic device 110 may perform pre-processing for motion tracking using color information and location information of the acquired objects (eg, the first object 130, the second object 140, etc.). have. The electronic device 110 may determine the first object image by using the obtained color information of the first object. Also, the electronic device 110 may change the distance value of the first object image into data by using the obtained location information. In various embodiments, the electronic device 110 may change the distance value of the first object image to be separated from the second object in various ways.

In an embodiment, the electronic device 110 may transmit data related to the motion tracking of the second object to the display device 120 , and the display device 120 may track the motion of the second object received from the electronic device 110 . A motion image of the second object may be displayed based on the data related to the . For example, the electronic device 110 may track a movement of a person performing a yoga movement, and may transmit data on the person's yoga movement to the display device 120 . The display device 120 may implement a figure of a person doing yoga in 3D based on the data on the movement.

In various embodiments, the electronic device 110 or the display device 120 may have the same or similar structure to the electronic device 301 of FIG. 3 to be described later.

In various embodiments, the display device 120 may be a computer as shown in FIG. 1 or various electronic devices such as a TV and a tablet PC not shown in FIG. 1 .

2 is a flowchart illustrating an overall flow of independently extracting an object from an electronic device to track a motion, according to an exemplary embodiment.

According to an embodiment, in operation 210, the electronic device 110 may execute an application.

According to an embodiment, the electronic device 110 may execute the application upon obtaining an input for executing the application from the user. For example, the application may be an application for photographing and tracking the movement of a moving subject. In addition, an input for executing the application may be various, such as an input for touching an icon, a voice input, and the like.

According to an embodiment, in operation 220, the electronic device 110 may photograph the first object. The first object may be, for example, a yoga mattress, an exercise mattress, a blanket, a living room rug, or the like. Also, the shooting may be a photo shooting or a video shooting.

In another embodiment, the electronic device 110 may photograph the ground of a living room floor, a room floor, a gym floor, and the like. Even in the case of photographing the ground, all operations disclosed in this document may be applicable to the electronic device 110 .

According to an embodiment, in operation 230 , the electronic device 110 may detect and store color information of the first object. For example, the color information may include RGB information. As another example, when the color of the photographed yoga mattress is white, RGB information regarding white (eg, (R, G, B)=(255, 253, 244), etc.) may be detected. The RGB information may include information such as R, G, and B color values. Also, the electronic device 100 may store data related to the color information in a memory. In this regard, it will be described in detail with reference to FIG. 5 .

According to an embodiment, in operation 240 , the electronic device 110 may photograph objects.

In an embodiment, the objects may include a first object and a second object moving on the first object. For example, when the user wants to take a picture of himself doing yoga on a yoga mattress, he or she may take a picture of the figure using the electronic device 110 . In this case, the electronic device 110 may photograph the yoga mattress (eg, the first object) and the user (eg, the second object).

In an embodiment, the shooting may be moving picture shooting. For example, the electronic device 110 may capture a video of the user doing yoga on a yoga mattress in real time according to the composition captured by the user. In this case, it is possible to check the state of doing yoga in real time through the screen displayed on the display device 120 . For another example, the electronic device 110 may capture a video of the user's yoga movement, and the display device 120 may implement the movement in 3D.

In an embodiment, the electronic device 110 may photograph objects based on an RGB camera and a stereo camera. For example, the electronic device 110 may include an RGB camera and a stereo camera. The electronic device 110 may obtain color information (eg, RGB information) of objects from the RGB camera, and may acquire position information (eg, 3D information) of the objects from the stereo camera.

In another embodiment, there may be a difference in the positions of the RGB camera and the stereo camera disposed in the electronic device 110 . In this case, there may be some difference between the position of the image captured by the RGB camera and the stereo camera, the composition of the image, and the like. The electronic device 110 may correct the image or image acquired by the RGB camera and the stereo camera to match or correspond to each other based on the predefined differences for each configuration (eg, mounting position, mounting angle, distance between configurations, etc.). have. For example, when there is a difference in positions or compositions of objects in an image or image obtained by the RGB camera and the stereo camera, the electronic device 110 may process the positions or compositions of the objects to match or correspond. For another example, the electronic device 110 may process an image or image acquired by the RGB camera and the stereo camera so that pixel positions of the objects match or correspond.

According to an embodiment, the electronic device 100 may acquire color information and location information of the objects by photographing the objects. For example, the electronic device 100 may obtain color information of objects from an RGB camera and may obtain position information of objects from a stereo camera. The color information may include at least RGB information. The location information may include location information of a first object (eg, a yoga mattress), location information of a second object (eg, a person), and location information of other objects (eg, a vase, a lamp, etc.).

According to an embodiment, in operation 250 , the electronic device 110 may track the movement of the second object.

In an embodiment, the electronic device 110 may acquire joint information based on location information of a second object (eg, a person). For example, the electronic device 110 may obtain joint information including information on a person's neck joint, shoulder joint, wrist joint, wrist joint, pelvic joint, knee joint, and ankle joint. In an embodiment, the electronic device 110 spatially compares the first object among the photographed objects with the other objects (eg, the second object) based on the color information of the first object obtained through operations 220 and 230 . can be separated. For example, the electronic device 110 may change the value of any one axis (eg, the Z axis) among the spatial coordinates of the first object so that the first object and the second object are sufficiently spaced apart from the Z axis. have. The electronic device 110 separates the first object and the second object on spatial coordinates so that the second object (eg, a person) actually approaches the first object (eg, a mat) to perform an exercise such as a push-up or yoga, and Even if there is, it is possible to effectively track the movement of the second object.

In an embodiment, the electronic device 110 may obtain position information including the joint information by using a stereo camera. For example, the camera module (eg, 380 ) of the electronic device 110 may include a stereo camera as a configuration. The electronic device 110 may obtain location information of the second object (eg, a person) through the stereo camera. The location information may include coordinates of a neck joint, a shoulder joint, a wrist joint, a wrist joint, a pelvic joint, a knee joint, an ankle joint, and a change in the coordinates.

In an embodiment, the electronic device 110 may track the motion of the second object based on location information including joint information. For example, the electronic device 110 may set the positions of various joints included in the joint information as coordinates based on the x-axis, the y-axis, and the z-axis. Also, the electronic device 110 may detect a change in the coordinates of the joints, and may track the movement by tracking the coordinates.

Detailed descriptions of operations according to various embodiments described with reference to FIG. 2 will be described later in detail in corresponding parts of FIGS. 5, 7, and 8 .

3 is a block diagram of an electronic device 301 in a network environment 300 according to various embodiments of the present disclosure. Referring to FIG. 3 , in a network environment 300 , the electronic device 301 communicates with the electronic device 302 through a first network 398 (eg, a short-range wireless communication network) or a second network 399 . It may communicate with the electronic device 304 or the server 308 through (eg, a long-distance wireless communication network). According to an embodiment, the electronic device 301 may communicate with the electronic device 304 through the server 308 . According to an embodiment, the electronic device 301 includes a processor 320 , a memory 330 , an input device 350 , a sound output device 355 , a display device 360 , an audio module 370 , and a sensor module ( 376 , interface 377 , haptic module 379 , camera module 380 , power management module 388 , battery 389 , communication module 390 , subscriber identification module 396 , or antenna module 397 . ) may be included. In some embodiments, at least one of these components (eg, the display device 360 or the camera module 380 ) may be omitted or one or more other components may be added to the electronic device 301 . In some embodiments, some of these components may be implemented as a single integrated circuit. For example, the sensor module 376 (eg, a fingerprint sensor, an iris sensor, or an illuminance sensor) may be implemented while being embedded in the display device 360 (eg, a display).

The processor 320, for example, executes software (eg, a program 340) to execute at least one other component (eg, a hardware or software component) of the electronic device 301 connected to the processor 320 . It can control and perform various data processing or operations. According to an embodiment, as at least part of data processing or operation, the processor 320 stores commands or data received from other components (eg, the sensor module 376 or the communication module 390 ) into the volatile memory 332 . may be loaded into the volatile memory 332 , may process commands or data stored in the volatile memory 332 , and may store the resulting data in the non-volatile memory 334 . According to an embodiment, the processor 320 includes a main processor 321 (eg, a central processing unit or an application processor), and an auxiliary processor 323 (eg, a graphic processing unit, an image signal processor) that can be operated independently or together with the main processor 321 . , a sensor hub processor, or a communication processor). Additionally or alternatively, the auxiliary processor 323 may be configured to use less power than the main processor 321 or to specialize in a designated function. The auxiliary processor 323 may be implemented separately from or as part of the main processor 321 .

The coprocessor 323 may be, for example, on behalf of the main processor 321 while the main processor 321 is in an inactive (eg, sleep) state, or the main processor 321 is active (eg, executing an application). ), together with the main processor 321, at least one of the components of the electronic device 301 (eg, the display device 360, the sensor module 376, or the communication module 390) It is possible to control at least some of the related functions or states. According to an embodiment, the coprocessor 323 (eg, an image signal processor or a communication processor) may be implemented as part of another functionally related component (eg, the camera module 380 or the communication module 390). have.

The memory 330 may store various data used by at least one component (eg, the processor 320 or the sensor module 376 ) of the electronic device 301 . The data may include, for example, input data or output data for software (eg, the program 340 ) and instructions related thereto. The memory 330 may include a volatile memory 332 or a non-volatile memory 334 .

The program 340 may be stored as software in the memory 330 , and may include, for example, an operating system 342 , middleware 344 , or an application 346 .

The input device 350 may receive a command or data to be used in a component (eg, the processor 320 ) of the electronic device 301 from the outside (eg, a user) of the electronic device 301 . The input device 350 may include, for example, a microphone, a mouse, a keyboard, or a digital pen (eg, a stylus pen).

The sound output device 355 may output a sound signal to the outside of the electronic device 301 . The sound output device 355 may include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback, and the receiver can be used to receive incoming calls. According to an embodiment, the receiver may be implemented separately from or as a part of the speaker.

The display device 360 may visually provide information to the outside (eg, a user) of the electronic device 301 . The display device 360 may include, for example, a display, a hologram device, or a projector and a control circuit for controlling the corresponding device. According to an embodiment, the display device 360 may include a touch circuitry configured to sense a touch or a sensor circuit (eg, a pressure sensor) configured to measure the intensity of a force generated by the touch. have.

The audio module 370 may convert a sound into an electric signal or, conversely, convert an electric signal into a sound. According to an embodiment, the audio module 370 acquires a sound through the input device 350 or an external electronic device (eg, a sound output device 355 ) directly or wirelessly connected to the electronic device 301 . The electronic device 302) (eg, a speaker or headphones) may output a sound.

The sensor module 376 detects an operating state (eg, power or temperature) of the electronic device 301 or an external environmental state (eg, user state), and generates an electrical signal or data value corresponding to the sensed state. can do. According to an embodiment, the sensor module 376 may include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, It may include a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface 377 may support one or more designated protocols that may be used for the electronic device 301 to directly or wirelessly connect with an external electronic device (eg, the electronic device 302 ). According to an embodiment, the interface 377 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 378 may include a connector through which the electronic device 301 can be physically connected to an external electronic device (eg, the electronic device 302 ). According to an embodiment, the connection terminal 378 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 379 may convert an electrical signal into a mechanical stimulus (eg, vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic sense. According to an embodiment, the haptic module 379 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 380 may capture still images and moving images. According to an embodiment, the camera module 380 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 388 may manage power supplied to the electronic device 301 . According to an embodiment, the power management module 388 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

The battery 389 may supply power to at least one component of the electronic device 301 . According to an embodiment, the battery 389 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 390 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 301 and an external electronic device (eg, the electronic device 302, the electronic device 304, or the server 308). It can support establishment and communication through the established communication channel. The communication module 390 may include one or more communication processors that operate independently of the processor 320 (eg, an application processor) and support direct (eg, wired) communication or wireless communication. According to an embodiment, the communication module 390 is a wireless communication module 392 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 394 (eg, : It may include a local area network (LAN) communication module, or a power line communication module). Among these communication modules, a corresponding communication module is a first network 398 (eg, a short-range communication network such as Bluetooth, WiFi direct, or infrared data association (IrDA)) or a second network 399 (eg, a cellular network, the Internet, or It may communicate with an external electronic device via a computer network (eg, a telecommunication network such as a LAN or WAN). These various types of communication modules may be integrated into one component (eg, a single chip) or may be implemented as a plurality of components (eg, multiple chips) separate from each other. The wireless communication module 392 uses the subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 396 within a communication network, such as the first network 398 or the second network 399 . The electronic device 301 may be identified and authenticated.

The antenna module 397 may transmit or receive a signal or power to the outside (eg, an external electronic device). According to an embodiment, the antenna module may include one antenna including a conductor formed on a substrate (eg, a PCB) or a radiator formed of a conductive pattern. According to an embodiment, the antenna module 397 may include a plurality of antennas. In this case, at least one antenna suitable for a communication scheme used in a communication network such as the first network 398 or the second network 399 is connected from the plurality of antennas by, for example, the communication module 390 . can be selected. A signal or power may be transmitted or received between the communication module 390 and the external electronic device through the selected at least one antenna. According to some embodiments, other components (eg, RFIC) other than the radiator may be additionally formed as a part of the antenna module 397 .

At least some of the components are connected to each other through a communication method between peripheral devices (eg, a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and a signal ( eg commands or data) can be exchanged with each other.

According to an embodiment, the command or data may be transmitted or received between the electronic device 301 and the external electronic device 304 through the server 308 connected to the second network 399 . Each of the electronic devices 302 and 304 may be the same or a different type of device as the electronic device 301 . According to an embodiment, all or part of the operations performed by the electronic device 301 may be executed by one or more of the external electronic devices 302 , 304 , or 308 . For example, when the electronic device 301 is to perform a function or service automatically or in response to a request from a user or other device, the electronic device 301 may perform the function or service itself instead of executing the function or service itself. Alternatively or additionally, one or more external electronic devices may be requested to perform at least a part of the function or the service. The one or more external electronic devices that have received the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit a result of the execution to the electronic device 301 . The electronic device 301 may process the result as it is or additionally and provide it as at least a part of a response to the request. For this purpose, for example, cloud computing, distributed computing, or client-server computing technology may be used.

4 is a block diagram 400 illustrating a camera module 380, according to various embodiments. Referring to FIG. 4 , the camera module 380 includes a lens assembly 410 , a flash 420 , an image sensor 430 , an image stabilizer 440 , a memory 450 (eg, a buffer memory), or an image signal processor. (460). The lens assembly 410 may collect light emitted from a subject to be imaged. The lens assembly 410 may include one or more lenses. According to an embodiment, the camera module 380 may include a plurality of lens assemblies 410 . In this case, the camera module 380 may form, for example, a dual camera, a 360 degree camera, or a spherical camera. Some of the plurality of lens assemblies 410 may have the same lens properties (eg, angle of view, focal length, auto focus, f number, or optical zoom), or at least one lens assembly may be a different lens assembly. It may have one or more lens properties different from the lens properties of . The lens assembly 410 may include, for example, a wide-angle lens or a telephoto lens.

The flash 420 may emit light used to enhance light emitted or reflected from the subject. According to an embodiment, the flash 420 may include one or more light emitting diodes (eg, a red-green-blue (RGB) LED, a white LED, an infrared LED, or an ultraviolet LED), or a xenon lamp. The image sensor 430 may acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly 410 into an electrical signal. According to an embodiment, the image sensor 430 may include, for example, one image sensor selected from among image sensors having different properties, such as an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor, the same It may include a plurality of image sensors having a property, or a plurality of image sensors having different properties. Each image sensor included in the image sensor 430 may be implemented using, for example, a charged coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.

In response to the movement of the camera module 380 or the electronic device 301 including the same, the image stabilizer 440 moves at least one lens or the image sensor 430 included in the lens assembly 410 in a specific direction or Operation characteristics of the image sensor 430 may be controlled (eg, read-out timing may be adjusted, etc.). This makes it possible to compensate for at least some of the negative effects of the movement on the image being taken. According to an embodiment, the image stabilizer 440, according to an embodiment, the image stabilizer 440 is a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module 380 . can be used to detect such a movement of the camera module 380 or the electronic device 301 . According to an embodiment, the image stabilizer 440 may be implemented as, for example, an optical image stabilizer. The memory 450 may temporarily store at least a portion of the image acquired through the image sensor 430 for a next image processing operation. For example, when image acquisition is delayed according to the shutter or a plurality of images are acquired at high speed, the acquired original image (eg, Bayer-patterned image or high-resolution image) is stored in the memory 450 and , a copy image corresponding thereto (eg, a low-resolution image) may be previewed through the display device 360 . Thereafter, when a specified condition is satisfied (eg, a user input or a system command), at least a portion of the original image stored in the memory 450 may be obtained and processed by, for example, the image signal processor 460 . According to an embodiment, the memory 450 may be configured as at least a part of the memory 330 or as a separate memory operated independently of the memory 330 .

The image signal processor 460 may perform one or more image processing on an image acquired through the image sensor 430 or an image stored in the memory 450 . The one or more image processes may include, for example, depth map generation, three-dimensional modeling, panorama generation, feature point extraction, image synthesis, or image compensation (eg, noise reduction, resolution adjustment, brightness adjustment, blurring ( blurring, sharpening, or softening. Additionally or alternatively, the image signal processor 460 may include at least one of the components included in the camera module 380 (eg, an image sensor). 430), for example, exposure time control, readout timing control, etc. The image processed by the image signal processor 460 is stored back in the memory 450 for further processing. or may be provided as an external component of the camera module 380 (eg, the memory 330 , the display device 360 , the electronic device 302 , the electronic device 304 , or the server 308 ). According to an example, the image signal processor 460 may be configured as at least a part of the processor 320 or as a separate processor operated independently of the processor 320. The image signal processor 460 may include the processor 320 . and a separate processor, the at least one image processed by the image signal processor 460 may be displayed through the display device 360 as it is by the processor 320 or after additional image processing.

According to an embodiment, the electronic device 301 may include a plurality of camera modules 380 each having different properties or functions. In this case, for example, at least one of the plurality of camera modules 380 may be a wide-angle camera, and at least the other may be a telephoto camera. Similarly, at least one of the plurality of camera modules 380 may be a front camera, and at least the other may be a rear camera.

5 is a flowchart illustrating a flow of pre-processing an image of a first object in an electronic device according to an exemplary embodiment. Operations according to various embodiments of FIG. 5 will be described with reference to FIGS. 6A and 6B , which show images of the first object being pre-processed in the electronic device.

According to an embodiment, in operation 510 , the electronic device 110 may photograph the first object.

In an embodiment, the user may photograph the yoga mattress 610 according to the application manual. For example, the electronic device 110 may take a picture or a video of the yoga mattress 610 based on a user's shooting input. Specifically, the electronic device 110 may capture a picture or a video so that the first object (eg, the yoga mattress 610) is located in the center of the photographing screen based on the user's photographing motion.

In an embodiment, the camera module (eg, 380) of the electronic device 110 may include an RGB camera. For example, the electronic device 110 may take a picture or a video of the yoga mattress 610 based on the RGB camera of the camera module (eg, 380 ).

According to an embodiment, in operation 520 , the electronic device 110 may detect color information of the first object from the captured image.

In an embodiment, the electronic device 110 may detect color information of the yoga mattress 610 from an image obtained by photographing the yoga mattress 610 with a camera module (eg, 380 ). For example, when the color of the yoga mattress 610 is white, RGB information for white (eg, (R, G, B)=(255, 253, 244), etc.) may be obtained. For another example, when the color of the yoga mattress 610 is purple, RGB information (eg, (R, G, B)=(255, 0, 244), etc.) for purple may be obtained.

In another embodiment, the electronic device 110 may determine the same object as the same object up to a point where the color information of the first object rapidly changes based on the camera module (eg, 380 ). For example, the electronic device 110 may take a picture so that the yoga mattress 610 is located in the center of the screen in the background of the purple yoga mattress 610 and the white surrounding background. As shown in FIG. 6B , the electronic device 110 may detect RGB color values by moving up/down/left/right for each pixel starting from the center point of the screen. In addition, the electronic device 110 may determine the same object based on a point at which color information (eg, an RGB color value) changes rapidly during the detection as a boundary. For example, while detecting the color information of the yoga mattress 610, the electronic device 110 may identify points at which the color information (eg, RGB color value) changes rapidly at the boundary between the purple yoga mattress 610 and the white background. have. In this case, the electronic device 110 may determine that the yoga mattress 610 is purple up to the points where the color information changes rapidly.

According to an embodiment, in operation 530 , the electronic device 110 may determine a representative color of the first object.

According to an embodiment, there may be a case where the RGB color value of color information in the first object (eg, yoga mattress) is different depending on lighting or a shooting angle. For example, color information (eg, purple) of the yoga mattress 610 may be different depending on a light, sunlight, or shooting angle. In this case, the electronic device 110 may detect color information according to operation 520 described above, and may determine the average value as the representative color of the yoga mattress 610 by obtaining an average value of the detected RGB color values.

In another embodiment, the average value of the RGB color values of the same object may be the same for every photographing. For example, the average value of the RGB color values of the yoga mattress 610 may be a value corresponding to purple even when shooting with a time difference. The electronic device 110 may obtain the average value of the RGB color values of the detected first object (eg, yoga mattress) with a time difference, and the color corresponding to the average value can be obtained with a time difference even when shooting It can be the same as the representative color (eg purple).

[Equation 1]

In various embodiments, the RGB average color value of the image of the first object (eg, the yoga mattress 610) may be obtained based on Equation 1 (hereinafter, 'first condition'). For example, the red (R, red) average color value of the image of the first object (eg, yoga mattress 610) is from the center pixel of the image of the first object (eg, yoga mattress 610) to the outermost pixel It may be a value obtained by dividing the sum of the color values of R by the number of pixels. For another example, the green (G, green) average color value of the image of the first object (eg, yoga mattress 610) is the outermost pixel from the center pixel of the image of the first object (eg, yoga mattress 610) It may be a value obtained by dividing the sum of the color values of G up to and including the number of pixels. In addition, the blue (B, blue) average color value of the image of the first object (eg, yoga mattress 610) is B from the center pixel of the image of the first object (eg, yoga mattress 610) to the outermost pixel It may be a value obtained by dividing the sum of the color values of , by the number of pixels.

According to an embodiment, in operation 540 , the electronic device 110 may store color information and a representative color of the first object. For example, the electronic device 100 may store color information (eg, RGB information, etc.) corresponding to the color of the yoga mattress (eg, purple), the color of the background (eg, white), and data on the representative color in the memory. have.

7 is a flowchart illustrating a flow of determining a first object image in an electronic device according to an exemplary embodiment.

According to an embodiment, in operation 710 , the electronic device 110 may capture a moving picture of the objects. The objects may include at least a first object and a second object moving on the first object. For example, as shown in FIG. 1 , the electronic device 110 may capture a video of the user 140 doing yoga on the yoga mattress 130 . For another example, when the electronic device 110 shoots a video, a video may be captured not only with the user 140 and the yoga mattress 130 , but also with a background including a floor, a vase, a lamp, and the like.

According to an embodiment, in operation 720, the electronic device 110 may compare the color information of the first object stored in the memory with the color information of the photographed objects.

In an embodiment, the electronic device 110 may acquire color information of the objects from a screen being photographed including the user 140 , the yoga mattress 130 , and the like. In this case, the electronic device 110 determines the color of the yoga mattress 130 by the user 140, the yoga mattress 130, etc., based on the color information corresponding to the color (eg, purple) of the yoga mattress 130 stored in operation 540 . An object having the same color information as color information corresponding to a color (eg, purple) can be searched for on the screen. Specifically, for example, the electronic device 110 may determine an object having the same RGB average color value as the calculated RGB average color value (eg, the purple RGB average color value) as an object having the same color information.

For another example, based on the color information of the color (eg, purple) of the yoga mattress 130 stored in operation 540 , the electronic device 110 may display the user 140 , the yoga mattress 130 , a floor, a vase, a lamp, and the like. Among them, an object having color information corresponding to a color (eg, purple) of the yoga mattress 130 may be searched for on the screen. For example, the electronic device 110 may determine that objects having RGB average color values within a specific range from the calculated average RGB color values are the same object. Specifically, for example, the electronic device 110 determines that objects having RGB average color values within 2% or within 5% of the calculated RGB average color value (eg, purple RGB average color value) are the same object. can do. The 2% and 5% described above are an example, and may be various ranges according to the setting of the electronic device 110 .

According to an embodiment, in operation 730, the electronic device 110 may determine an image corresponding to the color information of the first object. Also, the image corresponding to the color information of the first object may be a 'first object image'. For example, the electronic device 110 may perform the yoga mattress 610 having the same color information among the objects being photographed in operation 710 based on color information (eg, purple) of the color (eg, purple) of the yoga mattress 610 stored in operation 540 . It is possible to determine the yoga mattress 610 image by detecting . For another example, the electronic device 110 may store within a specific range (eg, 2% range) among objects being photographed in operation 710 based on color information (eg, purple) of the yoga mattress 610 stored in operation 540 . By detecting the yoga mattress 610 having similar color information to (in), it is possible to determine the yoga mattress 610 image.

8 is a flowchart illustrating a flow of adjusting a distance value d of a first object image and tracking a motion of a second object based on location information in an electronic device, according to an exemplary embodiment. Operations according to various embodiments of FIG. 8 will be described with reference to FIGS. 9A, 9B, and 9C illustrating a concept of tracking a motion of a second object based on location information in an electronic device.

According to an embodiment, in operation 810 , the electronic device 110 may capture a moving picture of the objects. For example, the electronic device 110 may capture a video of the yoga mattress 910 , the person 920 , and the like. In addition, the person 920 may be exercising in a state in close contact with the yoga mattress 910 (eg, a prone motion, a lying motion, etc.).

According to an embodiment, in operation 820, the electronic device 110 may obtain location information of objects. The objects may include a first object (eg, the yoga mattress 910) and a second object (eg, a person) moving on the first object.

In an embodiment, the electronic device 110 may obtain position information of the yoga mattress 910 and the person 920 by using the stereo camera of the camera module 380 . The location information may include distance information (eg, a distance value), coordinate information, coordinate movement information, joint information, coordinate axis information, and the like. Also, as shown in the dot of FIG. 9C , the joint information of the person 920 may include coordinates of a neck joint, a shoulder joint, a wrist joint, a wrist joint, a finger joint, a pelvic joint, a knee joint, an ankle joint, and the like. For another example, when the person 920 raises an arm, the location information may include coordinate information of a shoulder joint, a wrist joint, and the like, and motion information of the coordinates. As another example, location information including various coordinate information of objects, coordinate movement information, etc. may be obtained based on the coordinate axes (x-axis, y-axis, and z-axis) with respect to the electronic device 110 .

According to an embodiment, in operation 830, the distance value d of the image corresponding to the color information of the first object may be adjusted. Referring to FIG. 9B , the electronic device 110 may determine an image of the yoga mattress 910 in operation 730 and acquire location information of the yoga mattress 910 and the person 920 . For example, distance information from the electronic device 110 to the yoga mattress 910 may be acquired. The distance information may include a distance value from the electronic device 110 to the yoga mattress 910, a coordinate value for the distance, and the like. In various embodiments, the distance value from the electronic device 110 to the person 920 may be expressed as d. The electronic device 110 may set coordinates (eg, (3, 4, 0)) corresponding to the distance value d (eg, 5) of the person 920 with respect to the electronic device 110 .

In an embodiment, the electronic device 110 may change the position of the yoga mattress 910 image based on a specific coordinate axis based on a set value α or a specific distance value β. The set value α may be an arbitrary value preset for distance adjustment in the electronic device 110 . The specific distance value β may be a distance value obtained while capturing a video by the electronic device 110 . Detailed descriptions of the set value (α) and the specific distance value (β) are separately described in each operation.

The electronic device 110 may change the distance value d from the original position of the image of the yoga mattress 910 to be further away from the electronic device 110 in the -z-axis direction by a preset value α. In addition, the yoga mattress 910 may be moved further away from the original position of the image by a specific distance value β in the -z-axis direction. In this case, the distance value (d) of the image of the yoga mattress 1010 may be changed (eg, the distance value (d')), and the coordinates (eg, (a, b, c)) are also different coordinates (eg (a)) , b, c')).

According to various embodiments, the electronic device 110 may store a value obtained by changing the distance value d of the image of the first object (eg, yoga mattress 910) based on a specific coordinate axis as specific data in the memory.

According to an embodiment, in operation 840, the electronic device 110 may acquire joint information of the second object based on the location information of the objects.

In an embodiment, the electronic device 110 sets a second object (eg, a person 920) based on the camera module in a state where the first object (eg, yoga mattress 910) is further away from the data based on a specific coordinate axis. For example, the electronic device 110 may obtain the joint information of the person 920, as indicated by dots in FIG. It is possible to obtain joint information including an ankle joint, etc. In addition, according to the data, the electronic device 110 indicates that the yoga mattress 910 is not positioned under the person 920 (eg, the person 920 and the yoga mattress 910 ). The joint information 930 of the person 920 may be obtained in a state in which the 910 is separated).

According to an embodiment, in operation 850, the electronic device 110 may track the motion of the second object based on joint information. For example, when the person 920 is exercising in a prone position, the electronic device 110 may obtain joint information of each joint including coordinate information. Also, when the person 920 performs push-ups in the prone position, the electronic device 110 may acquire coordinate information of each joint in real time while the person 920 changes the posture. The electronic device 110 may track the movement of the person 920 based on the coordinate information and the coordinate movement information.

In various embodiments, the electronic device 110 may transmit data related to the location information of the first object image and data related to the motion tracking of the second object to the display device 120 , and the display device 120 may transmit the first object and a motion image of the second object may be displayed based on data about the second object.

In an embodiment, the electronic device 110 displays original data, not specific data obtained by changing the distance value d of the image of the first object (eg, yoga mattress 910) based on a specific coordinate axis, on the display device 120 ) can be sent to Also, the electronic device 110 may transmit data tracking the motion of the second object to the display device 120 . In this case, the display device 120 receiving the above-described data may display an image of the person 920 exercising on the yoga mattress 910 .

In another embodiment, the electronic device 110 may transmit specific data obtained by changing the distance value d of the image of the first object (eg, yoga mattress 910 ) based on a specific coordinate axis to the display device 120 . Also, the electronic device 110 may transmit data tracking the motion of the second object to the display device 120 . In this case, the display device 120 receiving the above-described data may display an image of the person 920 exercising in a state where the yoga mattress 910 is not positioned below.

10 is a diagram illustrating a concept of tracking a motion of a second object by using a change in a distance value d according to various embodiments of the present disclosure;

For example, the electronic device 110 may capture a video of the yoga mattress 1010 and the person 1020 . For another example, the electronic device 110 may capture a moving picture of objects including the vase 1040 and the lamp 1050 .

In an embodiment, the electronic device 110 may acquire location information of the yoga mattress 1010 and the person 1020 based on the stereo camera of the camera module 380 . Also, the electronic device 110 may acquire location information of the vase 1040 and the lamp 1050 based on the camera module.

In various embodiments, the location information may include distance information (eg, a distance value), coordinate information, coordinate movement information, joint information, coordinate axis information, and the like. For example, as shown in FIG. 10 , the joint information 1030 of the person 1020 may include coordinates of a neck joint, a shoulder joint, a wrist joint, a wrist joint, a finger joint, a pelvic joint, a knee joint, an ankle joint, etc. have. For another example, when the person 1020 raises an arm, the location information may include coordinate information of a shoulder joint, a wrist joint, and the like, and motion information of the coordinates. As another example, location information including various coordinate information of objects, coordinate movement information, etc. may be obtained based on the coordinate axes (x-axis, y-axis, and z-axis) with respect to the electronic device 110 .

The electronic device 110 may determine an image of the yoga mattress 1010 and obtain location information such as the yoga mattress 1010 , the person 1020 , the vase 1040 , and the lamp 1050 .

In various embodiments, the electronic device 110 may change the distance value d of the image corresponding to the color information of the first object (eg, the yoga mattress 1010). Also, the electronic device 110 may change the position of the yoga mattress 1010 image based on various coordinate axes based on the set value α or a specific distance value β obtained from the captured image.

For example, the distance value d may be changed so that the position of the image of the yoga mattress 1010 is further away in the y-axis direction by using the distance value from the electronic device 110 to the vase 1040 from the original position. For another example, the distance value (d) of the yoga mattress 1010 image is set further away in the -z-axis direction by using the distance value from the electronic device 110 to the electric lamp 1050 from the original position. ) can be changed. In the above example, when using the distance value to the vase 1040 and the light 1050, the electronic device 110 adjusts the image of the person 1020 and the yoga mattress 1010 so that the image does not overlap to obtain the distance value d. can be changed

As a specific example, the electronic device 110 may obtain the distance value β of the object (eg, the vase 1040 ) determined to be the furthest from the y-axis on the camera screen. The electronic device 110 converts the coordinates (eg, (a, b, c)) corresponding to the distance value (d) of the original first object image (eg, the image of the yoga mattress 1010) to the distance value (β) can be used to move further away in the y-axis direction. In this case, the distance value (d) of the first object image (eg, the image of the yoga mattress 1010) may be changed (eg, the distance value (d')), and the coordinates (eg (a, b, c)) can also be changed to other coordinates (eg (a, b+ β', c)). In order to separate the first object (eg, the yoga mattress 1010 ) and the second object (eg, the person 1020 ) by a specific distance value, the electronic device 110 may be implemented in various ways other than the above-described method. .

In various embodiments, the electronic device 110 may store a value obtained by changing the distance value d of the image of the first object (eg, yoga mattress 1010 ) based on a specific coordinate axis as specific data in the memory.

According to an embodiment, the electronic device 110 may acquire joint information of the second object based on the location information of the objects.

In an embodiment, the electronic device 110 uses a stereo camera in a state in which the first object (eg, the yoga mattress 1010) is further away from the data based on a specific coordinate axis, and the second object (eg, the person 1020) ) of the joint information can be obtained. For example, the electronic device 110 obtains joint information 1030 including a neck joint, a shoulder joint, a wrist joint, a wrist joint, a pelvic joint, a knee joint, an ankle joint, etc. of the person 1020 as shown in FIG. 10 . can be obtained For another example, the electronic device 110 may obtain the joint information 1030 of the person 1020 in a state where the yoga mattress 1010 is not located under the person 1020 on the data.

According to an embodiment, the electronic device 110 may track the movement of the second object 1020 based on the joint information 1030 . For example, when the person 1020 is standing in an upright posture, the electronic device 110 may obtain joint information 1030 of each joint including coordinate information. In this case, the person 1020 may perform a yoga movement in which the right knee is bent and the right hand is stretched toward the sky. Also, the electronic device 110 may acquire coordinate information of each joint in real time while the person 1020 is changing from the normal posture to the yoga posture. The electronic device 110 may track the movement of the person 1020 based on the coordinate information and the coordinate movement information.

In various embodiments, the electronic device 110 may transmit data related to the location information of the first object image and data related to the motion tracking of the second object to the display device 120 , and the display device 120 may transmit the first object and a motion image of the second object may be displayed based on data about the second object.

In an embodiment, the electronic device 110 displays original data, not specific data obtained by changing the distance value d of the first object (eg, yoga mattress 1010) image based on a specific coordinate axis, to the display device 120 . can be sent to Also, the electronic device 110 may transmit data tracking the movement of the second object 1020 to the display device 120 . In this case, the display device 120 receiving the above-described data may display, for example, an image of a person 1020 doing yoga on the yoga mattress 1010 .

In another embodiment, the electronic device 110 may transmit specific data obtained by changing the distance value d of the image of the first object (eg, yoga mattress 1010 ) based on a specific coordinate axis to the display device 120 . Also, the electronic device 110 may transmit data tracking the movement of the second object 1020 to the display device 120 . In this case, the display device 120 receiving the above-described data may display, for example, an image of a person 1020 doing yoga in a state where the yoga mattress 1010 is not positioned below.

Electronic devices according to various embodiments disclosed in this document may be devices of various types. The electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance device. The electronic device according to the embodiment of this document is not limited to the above-described devices.

The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of the embodiments. In connection with the description of the drawings, like reference numerals may be used for similar or related components. The singular form of the noun corresponding to the item may include one or more of the item, unless the relevant context clearly dictates otherwise. As used herein, "A or B", "at least one of A and B", "at least one of A or B," "A, B or C," "at least one of A, B and C," and "A , B, or C" each may include any one of the items listed together in the corresponding one of the phrases, or all possible combinations thereof. Terms such as “first”, “second”, or “first” or “second” may simply be used to distinguish the component from other components in question, and may refer to components in other aspects (e.g., importance or order) is not limited. It is said that one (eg, first) component is "coupled" or "connected" to another (eg, second) component, with or without the terms "functionally" or "communicatively". When referenced, it means that one component can be connected to the other component directly (eg by wire), wirelessly, or through a third component.

As used herein, the term “module” may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as, for example, logic, logic block, component, or circuit. A module may be an integrally formed component or a minimum unit or a part of the component that performs one or more functions. For example, according to an embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

According to various embodiments of the present disclosure, one or more instructions stored in a storage medium (eg, internal memory 336 or external memory 338) readable by a machine (eg, electronic device 301) may be implemented as software (eg, the program 340) including For example, the processor (eg, the processor 320 ) of the device (eg, the electronic device 301 ) may call at least one of one or more instructions stored from a storage medium and execute it. This makes it possible for the device to be operated to perform at least one function according to the called at least one command. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain a signal (eg, electromagnetic wave), and this term is used in cases where data is semi-permanently stored in the storage medium and It does not distinguish between temporary storage cases.

According to an embodiment, the method according to various embodiments disclosed in this document may be provided by being included in a computer program product. Computer program products may be traded between sellers and buyers as commodities. The computer program product is distributed in the form of a device-readable storage medium (eg compact disc read only memory (CD-ROM)), or through an application store (eg Play Store™) or on two user devices (eg, It can be distributed (eg downloaded or uploaded) directly, online between smartphones (eg: smartphones). In the case of online distribution, at least a part of the computer program product may be temporarily stored or temporarily created in a machine-readable storage medium such as a memory of a server of a manufacturer, a server of an application store, or a relay server.

According to various embodiments, each component (eg, a module or a program) of the above-described components may include a singular or a plurality of entities. According to various embodiments, one or more components or operations among the above-described corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg, a module or a program) may be integrated into one component. In this case, the integrated component may perform one or more functions of each component of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component are executed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations are executed in a different order, omitted, or , or one or more other operations may be added.

Claims (20)

In an electronic device,
at least one camera module;
Memory; and
a processor operatively coupled to the at least one camera module and the memory, the processor comprising:
Taking an image of the first object,
Obtaining color (RGB) information of the photographed first object,
In response to a video recording command, starting video recording of the first object and a second object moving on the first object,
Based on the color information of the first object, determining a first object image corresponding to the first object from the data obtained by shooting the video,
Changing the three-dimensional coordinates of the area determined as the first object image in the data based on a specified distance value,
An electronic device for tracking a movement of the second object based on data in which the three-dimensional coordinates of the first object image are changed. The method according to claim 1,
The at least one camera module further comprises an RGB camera and a stereo camera,
The processor is
An electronic device that processes an image obtained from an RGB camera and an image obtained from a stereo camera to correspond to each other. The method according to claim 1,
The color information of the first object is color information detected from a pixel at a central point of the first object. The method according to claim 1,
The processor is
An electronic device that tracks the movement of the second object in real time. The method according to claim 1,
The processor is
and calculating a representative color of the first object based on color information of the first object and a first condition, and storing the representative color in a memory. The method according to claim 1,
The processor is
and storing, in the memory, data obtained by changing the distance value of the first object image. The method according to claim 1,
The location information of the second object includes joint information of the second object,
The processor, based on the joint information, tracking the movement of the second object (tracking), the electronic device. The method according to claim 1,
The processor is
An electronic device for transmitting motion tracking data obtained by tracking a motion of the second object to a display device. The method according to claim 1,
The at least one camera module includes an RGB camera and a stereo camera,
The processor is configured to obtain color information of the first object based on the RGB camera and obtain location information of the objects based on the stereo camera. The method according to claim 1,
The location information of the objects includes distance information, coordinate information, coordinate movement information, joint information, and coordinate axis information of the objects. A method of operating an electronic device, comprising:
photographing an image of the first object;
obtaining color (RGB) information of the photographed first object;
in response to a video recording command, starting video recording of the first object and a second object moving on the first object;
determining a first object image corresponding to the first object from data obtained by photographing the moving picture based on the color information of the first object;
changing the three-dimensional coordinates of the region determined as the first object image in the data based on a specified distance value;
and tracking the motion of the second object based on data in which the three-dimensional coordinates of the first object image are changed. 12. The method of claim 11,
The method further comprising the operation of processing the image acquired from the RGB camera and the image acquired from the stereo camera to correspond to each other. 12. The method of claim 11,
The color information of the first object is color information detected from a pixel of a central point of the first object. 12. The method of claim 11,
and determining an image having color information corresponding to the color information of the first object from among the photographed images as a first object image. 12. The method of claim 11,
calculating a representative color of the first object based on color information of the first object and a first condition; and
The method further comprising the operation of storing the representative color in a memory. 12. The method of claim 11,
The method further comprising the operation of storing data obtained by changing the distance value of the first object image in the memory. 12. The method of claim 11,
The location information of the second object includes joint information of the second object,
The method further comprising the operation of tracking the motion of the second object based on the joint information. 12. The method of claim 11,
The method further comprising the operation of transmitting the motion tracking data obtained by tracking the motion of the second object to a display device. 12. The method of claim 11,
obtaining color information of the first object based on an RGB camera; and
The method further comprising: obtaining position information of the objects based on a stereo camera. 12. The method of claim 11,
The location information of the objects includes distance information, coordinate information, coordinate motion information, joint information, and coordinate axis information of the objects.

Images