WO2024135789A1

WO2024135789A1 - Camera control device and apparatus equipped with same

Info

Publication number: WO2024135789A1
Application number: PCT/JP2023/046000
Authority: WO
Inventors: 卓也北出; 渉山田; 桂一落合
Original assignee: 株式会社Ｎｔｔドコモ
Priority date: 2022-12-22
Filing date: 2023-12-21
Publication date: 2024-06-27

Abstract

The present invention determines an object to be imaged.　In a camera control device 100 according to the present disclosure, an event camera 101 acquires a captured image on the basis of the light quantity change (flashing) of a marker 200 which is an object to be imaged. A control unit 102 detects the light quantity change (flashing) of the marker 200 from the captured image, and acquires a pattern image (vocode pattern 203) which becomes recognizable in a state where the focus of the event camera 101 is set to a long distance or to an infinite distance with respect to the marker 200 located within a prescribed distance. Then, the control unit 102 determines the positional relationship with the marker 200.

Description

Camera control device and device equipped with the same

The present invention relates to a camera control device that controls an event camera (Event Based Camera) and a device equipped with the same.

Patent Document 1 describes a robot control system that monitors the robot's work area, and includes an event camera that detects the movement of a moving object in the work area, an imaging camera that captures images of the work area, and a control device connected to the event camera and the imaging camera. This control device is configured to determine whether a person has entered or exited the work area based on the image captured by the imaging camera when the movement of a moving object is detected by the event camera.

JP 2021-53741 A

In Patent Document 1, in order to determine the state of the object being photographed, a camera that photographs the object is required in addition to the event camera, which creates problems of increased cost and a complex configuration.

In order to solve the above problems, the present invention aims to provide a camera control device that can determine the state of the subject being photographed, and a device equipped with the same.

The camera control device of the present invention comprises an event camera that acquires a captured image based on changes in the amount of light of the subject being photographed, and a control unit that acquires a pattern image from the captured image by detecting changes in the amount of light caused by the subject being photographed, and the control unit determines the state of the subject being photographed based on the pattern image.

In accordance with the present invention, a camera capable of low power consumption and high speed processing can be used to recognize pattern images from a photographed subject, and the condition of the photographed subject can be determined.

FIG. 1 is a system configuration diagram including a robot-shaped camera control device 100 according to the present disclosure. FIG. 2 is a diagram showing the configuration of a marker 200 having a vocoding function. FIG. 2 is a diagram showing a functional configuration of a robot 100a including a camera control device 100. 4 is a flowchart showing an operation of the camera control device 100. 1 is a diagram illustrating the positional relationship between an event camera 101 and a marker 200 (vocode pattern). FIG. FIG. 2 shows a detail of a marker 200 or a vocode pattern 203 visible therein. 13 is a flowchart showing a process for recognizing a blinking marker 200. 13 is a diagram showing an LED control pattern of the marker 200. FIG. FIG. 11 is a diagram illustrating a detailed explanation of an LED control pattern. 13 is a flowchart showing a modified example of the blinking ID recognition process and the marker attitude calculation process. FIG. 1 is a diagram showing an example in which the camera control device 100 is applied to VR goggles. A block diagram showing the functions of the VR goggles 100b. FIG. 2 is a diagram showing the configuration of a marker 200a. FIG. 2 is a diagram showing a functional configuration of a robot 100a including a camera control device 100c. 10 is a flowchart showing the operation of the camera control device 100c. FIG. 1 is a diagram illustrating an example of a hardware configuration of a camera control device 100 according to an embodiment of the present disclosure.

The embodiments of the present disclosure will be described with reference to the attached drawings. Where possible, identical parts will be designated by the same reference numerals, and duplicate explanations will be omitted.

FIG. 1 is a system configuration diagram including a robot-shaped camera control device 100 according to the present disclosure. As shown in the figure, the camera control device 100 can determine the posture (state) of a work object 300 by detecting a marker 200 attached to the work object 300.

This camera control device 100 is built into the robot 100a. The robot 100a has an arm and the like, and grasps and transports the work object 300 according to the detected posture of the work object 300. The work object 300 is, for example, a cardboard box in which products are packed.

Marker 200 is a tiny light source with embedded ID information, and has a vocode function. Vocode is a function or device for exchanging information that changes depending on the distance or camera focus position by taking pictures from close to long distances using an image input with a focus adjustment mechanism such as a digital camera. When vocoded, only the illumination of the light source is captured by the camera at long distances, and when the camera is focused at infinity or long distances at close range, the pattern composed in the vocode light source can be recognized by the camera.

FIG. 2 shows the configuration of a marker 200 with a vocode function. As shown in FIG. 2(a), the marker 200 is configured by stacking an LED 201, a diffuser 202, a vocode pattern 203, and a lens set 204. FIG. 2(b) shows the marker 200.

Figure 2(c) shows a vocode pattern 203. As shown in the figure, different patterns are written according to the position in a 5x5 matrix. Depending on how this pattern appears, it is possible to determine from which direction the image was taken.

FIG. 3 is a diagram showing the functional configuration of a robot 100a including a camera control device 100. This camera control device 100 includes an event camera 101 and a control unit 102. The robot 100a also includes a camera control device 100, an arm control unit 103, an arm motor 104, an arm 105, a wheel control unit 106, a wheel motor 107, and wheels 108.

The event camera 101 is a camera that detects changes in the amount of light on the subject being photographed and outputs the data.

The control unit 102 controls the event camera 101 based on the data output from the event camera 101, and also determines the state of the marker 200 (work object 300), such as its posture, based on that data.

The arm control unit 103 is a part that controls the arm motor 104 and the arm 105 according to the posture of the marker 200 recognized by the camera control device 100. The arm motor 104 is a power source that moves the arm 105 up, down, left, and right, and also causes the holding part of the arm 105 to grip and release the marker 200 (or the work object 300 to which it is attached).

The wheel control unit 106 is a part that controls the wheel motor 107 and the wheels 108 so that the robot heads toward the marker 200 (work object 300) recognized by the camera control device 100. The wheel motor 107 is a power source for controlling the direction of travel and steering, and rotates the wheels 108 and controls the steering direction.

Next, the operation of the camera control device 100 will be described. Figure 4 is a flowchart showing the operation of the camera control device 100.

Prior to this operation, the camera control device 100 instructs the marker 200 to start a blinking operation. The marker 200 and the camera control device 100 are equipped with a short-range wireless communication unit (not shown), and this is performed through short-range wireless communication. Then, when the event camera 101 detects the blinking of the marker 200 and the control unit 102 determines this, it adjusts the focus of the event camera 101 according to the distance to the marker 200 (S101).

The control unit 102 searches for, recognizes, and tracks images in which the marker 200 blinks (S102). The control unit 102 performs a time series analysis of the light source of the marker 200 to recognize a blinking ID that indicates the type of the marker 200 (S103). The process of recognizing this blinking ID will be described later.

The control unit 102 determines whether the size of the light source of the marker 200 captured by the event camera 101 is equal to or larger than a certain size (S104). If it is equal to or larger than a certain size, the control unit 102 determines that the distance to the marker 200 is short, and adjusts the focus of the event camera 101 to infinity (S105).

Then, the control unit 102 accumulates the event data (image data) output by the event camera 101 in a buffer memory (built into the control unit 102), and then superimposes a certain section of the event data to convert it into time-series frame data and store it (S106). The control unit 102 recognizes a vocode pattern from the stored event data (S107). The control unit 102 calculates the posture of the marker 200 from the recognized vocode pattern (S108).

In parallel with the above process, the control unit 102 searches for, recognizes, and tracks blinking images in the images acquired for the other markers 200 (S109). The control unit 102 performs a time series analysis of the light source of the other markers 200 to recognize the blinking ID indicating the type of the other marker 200 (S110). The control unit 102 performs processes S106 to S108 for the other markers 200 recognized here.

While performing the above-mentioned control, the wheel control unit 106 of the robot 100a advances in the direction in which the marker 200 is flashing, and when it is within a predetermined distance, it controls the arm 105 according to the posture of the marker 200 to transport the work object 300.

Figure 5 explains the positional relationship between the event camera 101 and the marker 200 (vocode pattern). As shown in the figure, the vocode pattern 203 captured varies depending on the position of the event camera 101. For example, pattern P15 in the vocode pattern 203 is visible from the event camera 101 at position P1 through the lens set 204. Also, vocode pattern P11 is visible from the event camera 101 at position P3. The posture of the marker 200 is determined based on the positional relationship between this visible image (position) and the position of the event camera 101.

Figure 6 shows details of the marker 200 or the vocode pattern 203 visible within it. Figure 6(a) shows an image of the marker 200 captured by the event camera 101 when the camera control device 100 (event camera 101) and the marker 200 are located at a long distance (a predetermined distance or greater). As shown in the figure, the light from the LED 201 that constitutes the marker 200 is captured in a round spherical shape, and the vocode pattern 203 is not captured. At this time, the focus of the event camera 101 is adjusted to the marker 200.

Figure 6(b) shows an image captured when the camera control device 100 approaches the marker 200. In this figure, the focus of the event camera 101 is set to an infinity state when the light from the marker 200 reaches a predetermined magnitude. As shown in the figure, a portion of the vocode pattern 203 is captured.

Figure 6(c) shows an image captured when the camera control device 100 approaches the marker 200 even closer. As shown in the figure, each pattern of the vocode pattern 203 is clearly captured.

FIG. 6(d) shows an example of vocode pattern 203. As shown in the figure, vocode pattern 203 is formed in a 5x5 matrix pattern. As shown in FIG. 5, the position of vocode pattern 203 differs from the captured image shown in FIG. 6(c). This is because, as explained in FIG. 5, the vocode pattern captured differs depending on the positional relationship between the camera control device 100 and the marker 200 and the capturing direction. The camera control device 100 determines its relative attitude and position with respect to the marker 200 depending on this captured image. In this disclosure, vocode pattern 203 is a 5x5 matrix pattern, but of course it is not limited to this and may be a small matrix, or conversely, a pattern with many matrices.

Next, the process of recognizing a blinking marker 200 (blinking ID) will be described. FIG. 7 is a flowchart showing the process of recognizing a blinking marker 200. As shown in the figure, the event camera 101 captures images according to the blinking of the marker 200. That is, the amount of light changes due to blinking, and the event camera 101 captures this change in the amount of light. In this disclosure, the marker 200 blinks according to on/off data in which the blinking ID is encoded using Manchester code. This blinking ID indicates the type of marker 200, etc.

Then, the control unit 102 accumulates the on/off data of the time region where the marker 200 is blinking in a fixed frame length (a predetermined time) in a memory unit (not shown) (S301). The control unit 102 searches the accumulated on/off data of the fixed frame length and detects the on/off period of the blinking (S302). The control unit 102 determines whether the length of the off period is equal to or longer than a certain period (S303). The control unit 102 then decodes the on/off data indicating the Manchester-encoded blinking ID, starting from the rise time of the on period (S304).

The event camera 101 has the characteristic that a signal is generated only when the brightness of the subject being photographed changes. Therefore, in order for the marker 200 to read 0/1 accurately in accordance with this characteristic, the marker 200 performs Manchester code processing on the blinking ID indicating the ID, and performs a blinking operation based on the encoded information. Note that Manchester codes are characterized by the fact that there is always a transition in the middle of each bit period, and depending on the information being transmitted, there is also a transition at the beginning of the period. Therefore, an event always occurs at a period of frequency/2, making it easier for the event camera 101 to recognize the blinking ID.

FIG. 8 is a diagram showing an LED control pattern based on the Manchester-encoded on/off data of the marker 200. This diagram shows a Manchester-encoded control pattern. As shown in the diagram, an LED control unit (not shown) provided in the marker 200 repeatedly turns the LED on and off. In the diagram, solid lines indicate on transitions, and dashed lines indicate off transitions. The marker 200 expresses its own ID by repeatedly turning the LED on and off. The blinking ID of the marker 200 indicates the type of work object 300 or indicates the vocode pattern 203. In FIG. 8, the vertical axis indicates the amount of change in luminance, and the horizontal axis indicates time.

Figure 9 is a diagram that provides a detailed explanation of the bit detection method used by an event camera. As shown in the diagram, an event camera can only recognize the occurrence and disappearance (On/Off) of an event, and cannot detect the continuation of an event. In other words, only the On/Off edges of an LED can be detected as an event. Manchester encoding is suitable for bit reading by an event camera because any bit sequence of 00, 01, 10, or 11 always results in an On/Off switch. In Figure 9, bit "0" indicates On → Off, and bit "1" indicates Off → On.

When the LED blinks on and off, a sleep period is provided to indicate the start and end of the blinking that indicates the blinking ID, and the blinking is off for a specified period of time.

The camera control device 100 can recognize the blinking ID by analyzing the blinking on/off status over time.

Next, a modified example of the blinking ID recognition process and the marker attitude calculation process will be described. FIG. 10 is a flowchart showing the operation of the camera control device 100 in this modified example. In this process, the focus of the event camera 101 is adjusted to infinity from the beginning (S201). Then, the event camera 101 and the control unit 102 perform the blinking ID recognition process of the marker 200 and the attitude calculation process of the marker 200 in parallel (S202 to S206). The process S202 to process S203 are performed for multiple markers 200.

In other words, while recognizing the blinking ID (S202, S203), the control unit 102 recognizes the stored vocode pattern and calculates the posture of the marker 200 (S204 to S206). Note that since the recognition of the vocode pattern is performed based on the recognition result of the blinking ID, the initial steps S204 to S206 are performed after the blinking ID is recognized by steps S202 to S203.

In this modified example of the camera control device 100, the focus is adjusted to infinity from the beginning, so focus processing can be omitted, simplifying processing.

Next, another application example of the camera control device 100 will be described. FIG. 11 is a diagram showing an example in which the camera control device 100 is applied to VR goggles 100b. As shown in the figure, a user U wears VR goggles 100b equipped with the camera control device 100. The VR goggles 100b are goggles that provide a virtual space to the user. A user wearing this VR goggles 100b is immersed in the virtual space and cannot see anything in the real space. In this example, a user wearing the VR goggles 100b can move through a virtual space that imitates the real space while being in the real space. A controlled object 300a that exists in the real space is represented in the virtual space, and its positional relationship is represented in the virtual space by recognizing a vocode pattern, etc.

In this disclosure, it is assumed that the real space is an interior room, and the virtual space is an arbitrary space that is different from the real space. It is assumed that home appliances, furniture, etc. that exist in the real space will become obstacles or targets in the virtual space, and their shapes will be represented according to the content of the virtual space.

Markers 200 (vocodes) are attached to lighting and air conditioners (appliance indicators, etc.) installed in the indoor environment in real space, and the VR goggles 100b (camera control device 100) can recognize the type, posture, etc. of these markers 200. The VR goggles 100b can then calculate the positional relationship of the VR goggles user in the room from the vocodes embedded in the markers 200.

FIG. 12 is a block diagram showing the functions of the VR goggles 100b. As shown in the figure, the VR goggles 100b include a camera control device 100, a virtual space providing unit 111, a communication unit 112, and a display 113.

The virtual space providing unit 111 is a part that generates a virtual space by image processing, generates an image for the virtual space based on the control object 300a determined by the camera control device 100, and represents the positional relationship and posture with respect to the user within the virtual space.

The communication unit 112 is the part that acquires information about the virtual space from an external server, etc.

The display 113 is a part that displays the virtual space and messages provided by the virtual space providing unit 111 to the user.

The VR goggles 100b perform the following operations: When in a distant location, the VR goggles 100b (camera control device 100) identifies the blinking ID of the blinking markers 200 (light sources) attached to multiple control objects.

Then, the VR goggles 100b generate a virtual space on the VR app that includes the controlled object 300a based on the blinking ID and represent the user's walking. The control unit 102 determines the type of controlled object according to the blinking ID, and the virtual space providing unit 111 generates an image according to the controlled object 300a (television, air conditioner, lighting). Since the user is in the virtual space, an image is generated with the angle of the controlled object 300a adjusted.

In other words, when the control object 300a approaches the VR goggles 100b, the VR goggles 100b calculate the positional relationship between the control object 300a and the VR goggles 100b, along with the blinking ID of the approaching control object. The VR goggles 100b then generate a virtual space on the VR app based on the calculated positional relationship, and also generate an approach warning message for the control object 300b as necessary.

The VR goggles 100b can also be used to detect the user's movement and obstacles in the real world. In FIG. 11, the controlled objects 300a are lighting, air conditioners, etc., but this is not limited to these. They may be attached to chairs, tables, etc., and other objects such as potted plants may also be detectable as obstacles.

As mentioned above, by embedding markers 200 (vocodes) in objects in real space, it is possible to express interactions with a user wearing VR goggles.

Next, other examples of the marker 200 will be described. The above-mentioned marker 200 was a device equipped with vocode, but this is not limited thereto, and the marker 200 does not need to be equipped with vocode. Fig. 13 is a diagram showing another example of the marker 200a. As shown in Fig. 13(a), one pattern image is drawn on the marker 200a. This marker 200a blinks in the same way as the marker 200. Fig. 13(b) is a schematic diagram showing the configuration of the marker 200a. As shown in the figure, the marker 200a is configured by overlapping an LED 201a and a pattern image 203a (transparent plate). The LED 201a blinks under the control of a control unit (not shown). As mentioned above, this blinking operation is preferably based on Manchester-encoded on/off data, but is not limited thereto.

FIG. 14 shows a robot 100a equipped with a camera control device 100c of the present disclosure. The functional configuration is the same as that of FIG. 3, but the control operation in the control unit 102a is different. That is, the control unit 102a determines the inclination of the shape and pattern of the marker 200a photographed by the event camera 101, and can determine the posture of the marker 200a (work object 300, etc.) accordingly. In addition, the control unit 102a always appropriately adjusts the focus of the event camera 101 according to the distance to the marker 200a.

FIG. 15 is a flowchart showing the operation of the camera control device 100c. The event camera 101 adjusts the focus according to the distance (S401), and the control unit 102a searches for, recognizes, and tracks the blinking light source (S402). The control unit 102a performs a time series analysis of the light source and recognizes the blinking ID (S403). The control unit 102a accumulates the event data acquired by the event camera 101 in time series (S404). The control unit 102a recognizes a pattern from the accumulated event data, and calculates the attitude of the marker 200a from the pattern (S406). In this example, the control unit 102a can calculate the attitude of the work object 300 from the shape of the marker 200a and the inclination of the pattern, etc.

The camera control device 100 of the present disclosure provides the following operational effects. Note that in the following explanation, the work object 300 is used as an example, but it can also be applied to the control object 300a.

In the camera control device 100 of the present disclosure, the event camera 101 acquires a captured image based on the change in the amount of light (blinking) of the marker 200, which is the object of capture. The control unit 102 then acquires a pattern image by detecting the change in the amount of light (blinking) caused by the marker 200 from the captured image. The control unit 102 then determines the positional relationship with the marker 200.

With this configuration, a camera capable of low power consumption and high speed processing, such as the event camera 101, can be used to recognize pattern images, and their positional relationship can be determined from the degree of inclination of the pattern images, etc.

The control unit 102 may also acquire a pattern image (vocode pattern 203) from the marker 200 that becomes recognizable when the focus of the event camera 101 is set to a long distance or infinity for a marker 200 that is within a predetermined distance.

With this configuration, a camera with low power consumption and high speed processing such as the event camera 101 can be used to recognize pattern images such as vocodes and determine their positional relationship. In other words, the event camera 101 cannot capture an object if the object is not moving, but it can capture an image by blinking.

In this disclosure, the marker 200 is a vocode having a light source that blinks in a predetermined pattern. The event camera 101 acquires a vocode pattern 203 (pattern image) contained in the vocode.

With this configuration, the event camera 101 and the vocoded marker 200 can be used to recognize the work object 300 and its state (posture) at high speed and with low power consumption.

In addition, the control unit 102 sets the focus of the event camera 101 to infinity or a long distance when the distance to the marker 200 (photographed object) is equal to or less than a predetermined value. The control unit 102 then determines the state (posture, etc.) of the marker 200 (photographed object) based on the pattern image.

With this configuration, when the distance is long, the marker 200 (i.e., the work object 300) is identified by changes in the amount of light, and when the distance is short, a vocode pattern such as a vocode can be identified by focusing the event camera 101 at infinity, etc. Thus, the posture of the marker 200 can be determined.

In the present disclosure, the control unit 102 determines the state of the marker 200 or marker 200a based on a pattern image. "Based on a pattern image" refers to a recognized vocode pattern for marker 200a, and based on the tilt of the pattern image for marker 200b, and the state (posture) of the marker 200, etc. can be determined from the pattern image.

The control unit 102 also identifies the work object 300 based on the blinking ID indicated by the change in the light intensity (blinking) of the marker 200. Then, the control unit 102 determines the state of the work object 300 according to the identified work object.

The control unit 102 can determine the type of work object 300 to which the marker 200 is attached based on the blinking ID of the marker 200. Therefore, the state of the work object 300, for example, its posture, can be determined based on the type.

In the present disclosure, the camera control device 100 further includes a storage unit that stores the captured images acquired by the event camera 101 in chronological order. The control unit 102 determines the blinking pattern based on the captured images, and determines the work object 300 (blinking ID) based on the blinking pattern.

Generally, an event camera 101 can recognize changes in the amount of light, but cannot tell the time. In this disclosure, the characteristics of the event camera 101 are utilized to recognize the blinking pattern, making it possible to recognize the work object 300.

In this disclosure, the robot 100a includes a camera control device 100. The robot 100a includes an arm 105 for holding a work object 300, and an arm control unit 103 for operating the arm 105 according to the posture of the work object 300 determined by the camera control device 100.

This configuration makes it possible to perform advanced tasks using the camera control device 100.

Furthermore, in this disclosure, the VR goggles 100b include a camera control device 100. In this VR goggles 100b, the virtual space providing unit 111 places a virtual object based on the controlled object 300a and its posture in a virtual space, and provides the virtual space to the user.

With this configuration, the camera control device 100 can be used to represent the posture, etc., of a virtual object in virtual space in accordance with an object in real space.

　Has the camera control device disclosed herein and a configuration under the device equipped with the same.

[1]
an event camera that captures images based on changes in the amount of light of an object to be photographed;
a control unit that acquires a pattern image by detecting a change in light amount caused by the object to be photographed from the photographed image;
Equipped with
The control unit determines a positional relationship with the object to be photographed based on the pattern image.
Camera control device.

[2]
The control unit is
detecting a change in light amount caused by the object from the captured image, and acquiring a pattern image from the object that is within a predetermined distance and that is recognizable when the focus of the event camera is set to a long distance or infinity;
The camera control device according to [1].

[3]
The subject includes a vocode having a light source that blinks in a predetermined pattern;
The control unit acquires the pattern image included in the vocode.
The camera control device according to [2].

[4]
the control unit sets a focus of the event camera to infinity or a long distance when a distance to the object to be photographed is equal to or less than a predetermined value.
The camera control device according to [2] or [3].

[5]
The control unit is
determining a state of the object to be photographed based on the pattern image;
The camera control device according to any one of [1] to [4].

[6]
The control unit is
Identifying the object to be photographed based on ID information indicated by the change in the amount of light;
determining a state of the photographed object according to the identified photographed object;
The camera control device according to [5].

[7]
A storage unit that stores the images captured by the event camera in chronological order,
The control unit determines a blinking pattern based on the captured image, and determines the object to be photographed based on the blinking pattern.
The camera control device according to [6].

[8]
An apparatus including the camera control device according to any one of [1] to [7],
an arm for holding the photographing object;
an arm control unit that operates the arm based on a positional relationship with the object to be photographed determined by the camera control device;
An apparatus comprising:

[9]
A VR goggle-type device comprising the camera control device according to any one of [1] to [7],
a virtual space providing unit that arranges the subject to be photographed and a virtual object based on the positional relationship in a virtual space and provides the virtual space to a user;
An apparatus comprising:

The block diagrams used to explain the above embodiments show functional blocks. These functional blocks (components) are realized by any combination of at least one of hardware and software. Furthermore, there are no particular limitations on the method of realizing each functional block. That is, each functional block may be realized using one device that is physically or logically coupled, or may be realized using two or more devices that are physically or logically separated and connected directly or indirectly (e.g., using wires, wirelessly, etc.) and these multiple devices. The functional blocks may be realized by combining the one device or the multiple devices with software.

Functions include, but are not limited to, judgement, determination, judgment, calculation, computation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, resolution, selection, election, establishment, comparison, assumption, expectation, regard, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating, mapping, and assignment. For example, a functional block (component) that performs the transmission function is called a transmitting unit or transmitter. As mentioned above, there are no particular limitations on the method of realization for either of these.

For example, the camera control device 100 in one embodiment of the present disclosure may function as a computer that performs processing of the camera control method of the present disclosure. FIG. 16 is a diagram showing an example of the hardware configuration of the

camera control devices

100 and 100c in one embodiment of the present disclosure. The camera control device 100 described above may be physically configured as a computer device including a processor 1001, memory 1002, storage 1003, communication device 1004, input device 1005, output device 1006, bus 1007, and the like. In the following description, the camera control device 100 includes the camera control device 100c unless otherwise specified.

In the following description, the word "apparatus" can be interpreted as a circuit, device, unit, etc. The hardware configuration of the camera control device 100 may be configured to include one or more of the devices shown in the figure, or may be configured to exclude some of the devices.

Each function of the camera control device 100 is realized by loading specific software (programs) onto hardware such as the processor 1001 and memory 1002, causing the processor 1001 to perform calculations, control communications via the communication device 1004, and control at least one of the reading and writing of data in the memory 1002 and storage 1003.

The processor 1001, for example, runs an operating system to control the entire computer. The processor 1001 may be configured as a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic unit, registers, etc. For example, the above-mentioned control unit 102 may be realized by the processor 1001.

The processor 1001 also reads out programs (program codes), software modules, data, etc. from at least one of the storage 1003 and the communication device 1004 into the memory 1002, and executes various processes according to these. The programs used are those that cause the computer to execute at least some of the operations described in the above-mentioned embodiments. For example, the control unit 102 may be realized by a control program stored in the memory 1002 and running on the processor 1001, and similarly may be realized for other functional blocks. Although the above-mentioned various processes have been described as being executed by one processor 1001, they may be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. The programs may be transmitted from a network via a telecommunications line.

Memory 1002 is a computer-readable recording medium, and may be composed of at least one of, for example, ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), RAM (Random Access Memory), etc. Memory 1002 may also be called a register, cache, main memory (primary storage device), etc. Memory 1002 can store executable programs (program codes), software modules, etc. for implementing a camera control method according to one embodiment of the present disclosure.

Storage 1003 is a computer-readable recording medium, and may be, for example, at least one of an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, etc. Storage 1003 may also be referred to as an auxiliary storage device. The above-mentioned storage medium may be, for example, a database, a server, or other suitable medium including at least one of memory 1002 and storage 1003.

The communication device 1004 is hardware (transmitting/receiving device) for communicating between computers via at least one of a wired network and a wireless network, and is also referred to as, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 1004 may be configured to include a high-frequency switch, a duplexer, a filter, a frequency synthesizer, etc., to realize, for example, at least one of Frequency Division Duplex (FDD) and Time Division Duplex (TDD).

The input device 1005 is an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that accepts input from the outside. The output device 1006 is an output device (e.g., a display, a speaker, an LED lamp, etc.) that performs output to the outside. Note that the input device 1005 and the output device 1006 may be integrated into one structure (e.g., a touch panel).

Furthermore, each device such as the processor 1001 and memory 1002 is connected by a bus 1007 for communicating information. The bus 1007 may be configured using a single bus, or may be configured using different buses between each device.

Note that the communication device 1004 and the input device 1005 are not essential to the camera control device 100 of this disclosure.

The camera control device 100 may also be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA), and some or all of the functional blocks may be realized by the hardware. For example, the processor 1001 may be implemented using at least one of these pieces of hardware.

The notification of information is not limited to the aspects/embodiments described in this disclosure, and may be performed using other methods. For example, the notification of information may be performed by physical layer signaling (e.g., DCI (Downlink Control Information), UCI (Uplink Control Information)), higher layer signaling (e.g., RRC (Radio Resource Control) signaling, MAC (Medium Access Control) signaling, broadcast information (MIB (Master Information Block), SIB (System Information Block)), other signals, or a combination of these. In addition, RRC signaling may be referred to as an RRC message, and may be, for example, an RRC Connection Setup message, an RRC Connection Reconfiguration message, etc.

The processing steps, sequences, flow charts, etc. of each aspect/embodiment described in this disclosure may be reordered unless inconsistent. For example, the methods described in this disclosure present elements of various steps using an example order and are not limited to the particular order presented.

The input and output information may be stored in a specific location (e.g., memory) or may be managed using a management table. The input and output information may be overwritten, updated, or added to. The output information may be deleted. The input information may be sent to another device.

The determination may be based on a value represented by one bit (0 or 1), a Boolean value (true or false), or a numerical comparison (e.g., with a predetermined value).

Each aspect/embodiment described in this disclosure may be used alone, in combination, or switched depending on the execution. In addition, notification of specific information (e.g., notification that "X is the case") is not limited to being done explicitly, but may be done implicitly (e.g., not notifying the specific information).

Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended as an illustrative example and does not have any limiting meaning with respect to the present disclosure.

Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

Software, instructions, information, etc. may also be transmitted and received via a transmission medium. For example, if the software is transmitted from a website, server, or other remote source using at least one of wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and/or wireless technologies (such as infrared, microwave, etc.), then at least one of these wired and wireless technologies is included within the definition of a transmission medium.

The information, signals, etc. described in this disclosure may be represented using any of a variety of different technologies. For example, the data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

Note that the terms explained in this disclosure and the terms necessary for understanding this disclosure may be replaced with terms having the same or similar meanings. For example, at least one of the channel and the symbol may be a signal (signaling). Also, the signal may be a message. Also, a component carrier (CC) may be called a carrier frequency, a cell, a frequency carrier, etc.

In addition, the information, parameters, etc. described in this disclosure may be represented using absolute values, may be represented using relative values from a predetermined value, or may be represented using other corresponding information. For example, a radio resource may be indicated by an index.

The names used for the parameters described above are not intended to be limiting in any way. Furthermore, the formulas etc. using these parameters may differ from those explicitly disclosed in this disclosure. The various channels (e.g., PUCCH, PDCCH, etc.) and information elements may be identified by any suitable names, and the various names assigned to these various channels and information elements are not intended to be limiting in any way.

In this disclosure, terms such as "Mobile Station (MS)," "user terminal," "User Equipment (UE)," and "terminal" may be used interchangeably.

A mobile station may also be referred to by those skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or some other suitable terminology.

As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, search, inquiry (e.g., searching in a table, database, or other data structure), and considering ascertaining as "judging" or "determining." Also, "determining" and "determining" may include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), and considering ascertaining as "judging" or "determining." Additionally, "judgment" and "decision" can include considering resolving, selecting, choosing, establishing, comparing, etc., to have been "judged" or "decided." In other words, "judgment" and "decision" can include considering some action to have been "judged" or "decided." Additionally, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

The terms "connected," "coupled," or any variation thereof, refer to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to one another. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to one another using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

Any reference to an element using a designation such as "first," "second," etc., used in this disclosure does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed or that the first element must precede the second element in some way.

When the terms "include," "including," and variations thereof are used in this disclosure, these terms are intended to be inclusive, similar to the term "comprising." Additionally, the term "or," as used in this disclosure, is not intended to be an exclusive or.

In this disclosure, where articles have been added through translation, such as a, an, and the in English, this disclosure may include that the nouns following these articles are plural.

In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combined" may also be interpreted in the same way as "different."

100...camera control device, 300...work object, 200...marker, 201...LED, 202...diffuser, 203...vocode pattern, 204...lens set, 100a...robot, 101...event camera, 102...control unit, 103...arm control unit, 104...arm motor, 105...arm, 106...wheel control unit, 107...wheel motor, 108...wheel.

Claims

an event camera that captures images based on changes in the amount of light of an object to be photographed;
a control unit that acquires a pattern image by detecting a change in light amount caused by the object to be photographed from the photographed image;
Equipped with
The control unit determines a positional relationship with the object to be photographed based on the pattern image.
Camera control device.
The control unit is
A change in the amount of light caused by the object to be photographed is detected from the photographed image, and a pattern image that can be recognized when the focus of the event camera is set to a long distance or infinity with respect to the object to be photographed within a predetermined distance is obtained from the object to be photographed.
The camera control device according to claim 1 .
The subject includes a vocode having a light source that blinks in a predetermined pattern;
The control unit acquires the pattern image included in the vocode.
The camera control device according to claim 2 .
the control unit sets a focus of the event camera to infinity or a long distance when a distance to the object to be photographed is equal to or less than a predetermined value.
The camera control device according to claim 2 .
The control unit is
determining a state of the object to be photographed based on the pattern image;
The camera control device according to claim 1 .
The control unit is
Identifying the object to be photographed based on ID information indicated by the change in the amount of light;
determining a state of the photographed object according to the identified photographed object;
The camera control device according to claim 5 .
A storage unit that stores the images captured by the event camera in chronological order,
The control unit determines a blinking pattern based on the captured image, and identifies ID information of the object to be photographed based on the blinking pattern.
The camera control device according to claim 6.
An apparatus comprising the camera control device according to claim 1,
an arm for holding the photographing object;
an arm control unit that operates the arm based on a positional relationship with the object to be photographed determined by the camera control device;
An apparatus comprising:
A VR goggle type device comprising the camera control device according to claim 1,
a virtual space providing unit that arranges the subject to be photographed and a virtual object based on the positional relationship in a virtual space and provides the virtual space to a user;
An apparatus comprising: