HomeCloudSoftwareHardware

API reference

package

rae_sdk

package
robot
package

rae_sdk.robot

package
api
module
audio
module
display
module
led
module
navigation
package
perception
module
robot
module
robot_options
module
state
package

rae_sdk.robot.api

module
openai
module
performance
package
ros
module

rae_sdk.robot.api.openai

class
OpenAIClient
class

rae_sdk.robot.api.openai.OpenAIClient

method
__init__(self, key = '', description = 'Assume you are a mobile robot that has a camera, speakers, microphone and LCD display. Let\'s say your main goal is to learn how to survive and interact with the world without causing harm to others.')
variable
property
method
generate_speech(self, text: str) -> str: str
Generate speech from text using OpenAI's TTS model.  Args: ----     text (str): The text to generate speech from.  Returns -------     str: The path to the generated speech file
method
describe_image(self, image: np.ndarray, query = 'What\'s in this image?') -> str: str
Describe an image using OpenAI's Vision model.  Args: ----     image (np.ndarray): The image to describe     query (str): The query to ask the model. Default is "What's in this image?"  Returns -------         str: The description of the image
method
respond(self, text: str) -> str: str
Generate a response from the robot using OpenAI's Chat model.  Args: ----     text (str): The text to generate a response from.  Returns -------     str: The response from the robot
module

rae_sdk.robot.api.performance

function
function
variable
variable
variable
variable
function
measure_average_performance(func: Callable [ ... , Any ]) -> Callable[..., Any]: Callable[..., Any]
Report once every 5 minutes what the averages function duration is and what the max and min durations are.
variable
variable
variable
function
function
with_sql_exception_handling(func: Callable [ ... , Any ]) -> Callable[..., Any]: Callable[..., Any]
Try sql operation command, catch exception and finally close the cursor and the connection.
function
package

rae_sdk.robot.api.ros

module
ros_interface
module

rae_sdk.robot.api.ros.ros_interface

constant
class
ROSInterface
A class that manages ROS2 functionalities for a robot or a system.  It includes initializing ROS2 context,  creating and managing nodes, publishers, and subscribers.  It also handles the startup and shutdown processes  for ROS2.  Attributes ----------     _name (str): Name of the ROS2 node.     _context (rclpy.context.Context | None): The ROS2 context.     _node (rclpy.node.Node | None): The ROS2 node.     _publishers (dict[str, Publisher]): Dictionary of ROS2 publishers.     _subscribers (dict[str, Subscription]): Dictionary of ROS2 subscribers.     _service_clients (dict[str, Client]): Dictionary of ROS2 service clients.     _action_clients (dict[str, dict[ActionClient]]): Dictionary of ROS2 action clients.     _timers (dict[str, Timer]): Dictionary of ROS2 timers.     _tf_buffer: The TF2 buffer.     _tf_listener: The TF2 listener.     _executor (Executor): The ROS2 executor.     _executor_thread (threading.Thread): The thread for the ROS2 executor.     _launch_service (LaunchService): The ROS2 launch service.     _stop_event (multiprocessing.Event): The event for stopping the ROS2 launch service.     _process (multiprocessing.Process): The process for running the ROS2 launch service.  Methods -------     start_hardware_process(): Starts the hardware process for ROS2.     start(): Initializes and starts the ROS2 node and executor.     stop_ros_process(): Stops the ROS2 hardware process.     stop(): Shuts down the ROS2 node and context.     create_publisher(topic_name, msg_type, qos_profile): Creates a publisher for a given topic.     publish(topic_name, msg): Publishes a message on a given topic.     create_subscriber(topic_name, msg_type, callback, qos_profile): Creates a subscriber for a given topic.     create_timer(timer_name, period, callback): Creates a timer for a given topic.     create_service_client(srv_name, srv_type): Creates a service client for a given service.     call_async_srv(srv_name, req): Calls a service asynchronously.     create_action_client(action_name, action_type): Creates an action client for a given action.     call_async_action_simple(action_name, goal): Calls an action asynchronously.     call_async_action(action_name, goal, goal_response_callback, goal_result_callback, goal_feedback_callback): Calls an action asynchronously.
class

rae_sdk.robot.api.ros.ros_interface.ROSInterface

method
__init__(self, robot_options: RobotOptions = RobotOptions())
Initialize the ROS2Manager instance.  Args: ----     robot_options (RobotOptions): An object containing the robot's options.
property
method
start_hardware_process(self)
Start RAE hardware drivers in a separate process.
method
start(self)
Run RAE hardware drivers process.Initializes and starts the ROS2 node and executor. It sets up the ROS2 context and starts the ROS2 spin.  Args: ----     start_hardware (bool): Whether to start the hardware process or not.
method
stop_ros_process(self)
Stop the ROS2 hardware process by terminating the related subprocess.
method
stop(self)
Shut down RAE drivers, ROS2 node and context.  This includes stopping the executor, destroying publishers subscribers, service clients, action clients, timers and shutting down the ROS2 context.
method
method
method
method
method
method
method
method
method
method
method
get_frame_position(self, source_frame, target_frame) -> TransformStamped: TransformStamped
Get the position of a frame relative to another frame.  Args: ----     source_frame (str): The source frame.     target_frame (str): The target frame.  Returns -------     TransformStamped: The position of the source frame relative to the target frame.
module

rae_sdk.robot.audio

class
AudioController
A class for controlling the robot's audio.  Attributes ----------     ros_interface (ROSInterface): An object for managing ROS2 communications and functionalities.     audio_client (Client): A ROS2 client for playing audio.     assets_path (str): The path to the robot's assets directory.  Methods -------     play_audio_file(audio_file_path): Plays an audio file.     honk(): Plays a horn sound.     play_random_sfx(): Plays a random sound effect.
class

rae_sdk.robot.audio.AudioController

method
__init__(self, ros_interface)
method
method
save_recorded_sound(self, audio_data, output_file = '/app/mic_recording.wav')
Decode the Base64 audio data and save it as a WAV file.  Attributes ----------     audio_data (str): Base64 encoded audio data.     output_file (str, optional): Path to save the WAV file. Defaults to "/app/output.wav".
method
method
module

rae_sdk.robot.display

function
quaternion_to_rotation_matrix(q)
Convert a quaternion into a rotation matrix.  Arguments ----------- q (tuple): A quaternion represented as (q_w, q_x, q_y, q_z).  Returns ------- numpy.ndarray: A 3x3 rotation matrix.
class
DisplayController
A class for controlling the robot's display.  Attributes ----------     _ros_interface (ROSInterface): An object for managing ROS2 communications and functionalities.     _bridge (CvBridge): An object for converting between ROS2 and OpenCV image formats.     _screen_width (int): The width of the robot's display.     _screen_height (int): The height of the robot's display.     _assets_path (str): The path to the robot's assets directory.  Methods -------     stop(): Stops the display.     display_default(): Displays the default image on the robot's display.     display_face(payload): Displays a face on the robot's display.     display_image(image_data): Displays an image on the robot's display.     display_imu_data(imu_data): Displays IMU data on the robot's display.     display_animation(): Displays an animation on the robot's display.     ball_callback(): Callback method for displaying an animation on the robot's display.
module

rae_sdk.robot.led

class
LEDController
A class for controlling the robot's LEDs.  Attributes ----------     ros_interface (ROSInterface): An object for managing ROS2 communications and functionalities.  Methods -------     set_leds(payload): Sets the robot's LEDs to a given color.
class

rae_sdk.robot.led.LEDController

method
__init__(self, ros_interface)
method
hex_to_rgb(self, hex)
Convert a hex color to an RGB tuple.
method
normalize(self, num)
Normalize a number to a float between 0 and 1.
method
set_leds_from_payload(self, payload: dict)
Set the robot's LEDs to a given color.  Args: ----     payload (dict): A dictionary containing the color to set the LEDs to.      Example payload struct: {'brightness': 50, 'color': '#FFFFFF', 'effect': 'pulse', 'interval': 5}
method
set_leds(self, color: str, brightness: int = 100, effect: str = 'solid', interval: int = 5)
Set the robot's LEDs to a given color.  Args: ----     color (str): The color to set the LEDs to.     brightness (int): The brightness of the LEDs. (Default: 100)     effect (str): The effect to apply to the LEDs. (Default: "solid")     interval (int): The interval of the effect. (Default: 5)
method
set_leds_from_msg(self, msg: LEDControl)
Set the robot's LEDs to a given color.  Args: ----     msg (LEDControl): The message containing the color to set the LEDs to.
module

rae_sdk.robot.navigation

class
NavigationController
A class for controlling the robot's movement.  Attributes ----------     ros_interface (ROSInterface): An object for managing ROS2 communications and functionalities.  Methods -------     move(linear, angular): Moves the robot in a given direction.
class

rae_sdk.robot.navigation.NavigationController

method
__init__(self, ros_interface)
method
move(self, linear, angular)
Move the robot in a given direction.  Args: ----     linear (float): The linear velocity.     angular (float): The angular velocity.
method
get_odom_position(self) -> TransformStamped: TransformStamped
Get the robot's current position relative to odom frame. Returns None if the robot's position is not available.  Returns -------     TransformStamped: The robot's current position.
package

rae_sdk.robot.perception

module
detections
module
perception_system
module
pipeline
module

rae_sdk.robot.perception.detections

variable
class
Detections
function
class

rae_sdk.robot.perception.detections.Detections

variable
variable
module

rae_sdk.robot.perception.perception_system

variable
class
PerceptionSystem
A class for managing camera functionalities in a robot, interfacing with both depthai and robothub libraries.  It includes initialization and management of camera streams, publishing capabilities, and camera device control.  Attributes ----------     ros_context_manager (dai_ros.ROSContextManager): Manager for ROS context.     dai_node (dai_ros.ROSNode): ROS node for depthai operations.     device_mxid (str): The serial number of the depthai device.     device_info (dai.DeviceInfo): Information about the depthai device.     device (dai.Device): The depthai device instance.     cal_handler: Calibration handler for the depthai device.     pipeline: The pipeline for camera data processing.     rh_stream_handles (dict): Handles for RobotHub video streams.     ros_stream_handles (dict): Handles for ROS video streams.  Methods -------     stop(): Closes the depthai device connection.     add_rh_stream(stream_name): Adds a RobotHub stream with the given name.     add_ros_img_stream(stream_name): Adds a ROS Image stream with the given name.     add_ros_imu_stream(stream_name): Adds a ROS IMU stream with the given name.     add_queue(name, callback): Adds a queue to the device for handling callbacks.     add_composable_node(package_name, plugin_name, options): Adds a composable node to the ROS context manager.     start_pipeline(pipeline): Starts the camera pipeline and initializes ROS node and context.     publish_rh(name, color_frame, timestamp, metadata): Publishes video data to RobotHub.     publish_ros(name, msg): Publishes a message to a ROS topic.     get_image(stream_name): Retrieves an image from the specified stream.
class

rae_sdk.robot.perception.perception_system.PerceptionSystem

method
__init__(self, namespace = '')
Initialize the Camera instance.  Connect to the depthai device and initializes the ROS context manager.  Args: ----     namespace (str, optional): The namespace for the ROS nodes. Defaults to ''.
method
__del__(self)
method
method
connect_to_device(self) -> bool: bool
Connect to the depthai device and initialize the calibration handler.
method
start(self)
Start the ROS context manager and spins the ROS node.
method
stop(self)
Close the connection to the depthai device, ensuring a clean shutdown.
method
add_rh_stream(self, stream_name)
Add a video stream to RobotHub with the specified name.  Args: ----     stream_name (str): The name of the stream to be added.
method
add_ros_img_stream(self, stream_name, topic_name, frame_name, socket, width = -1, height = -1, convertFromBitStream = False, frame_type = dai.RawImgFrame.Type.BGR888i)
Add a ROS video stream with the specified name and sets up the necessary configurations.  Args: ----     stream_name (str): The name of the ROS stream to be added.     topic_name (str): The name of the ROS topic to be published to.     frame_name (str): The name of the ROS frame.     socket (dai.CameraBoardSocket): The socket to be used for the stream.     width (int, optional): The width of the stream. Defaults to -1.     height (int, optional): The height of the stream. Defaults to -1.     convertFromBitStream (bool, optional): Whether to convert the stream from a bitstream. Defaults to False.     frame_type (dai.RawImgFrame.Type, optional): The type of the frame. Defaults to dai.RawImgFrame.Type.BGR888i.
method
add_ros_imu_stream(self, stream_name, topic_name, frame_name)
Add a ROS IMU stream with the specified name and sets up the necessary configurations.  Args: ----     stream_name (str): The name of the ROS stream to be added.     topic_name (str): The name of the ROS topic to be published to.     frame_name (str): The name of the ROS frame.
method
add_ros_feature_stream(self, stream_name, topic_name, frame_name)
Add a ROS feature stream with the specified name and sets up the necessary configurations.  Args: ----     stream_name (str): The name of the ROS stream to be added.     topic_name (str): The name of the ROS topic to be published to.     frame_name (str): The name of the ROS frame.
method
add_queue(self, name, callback = None)
Add a queue to the depthai device for processing callbacks.  Args: ----     name (str): The name of the queue.     callback (callable): The callback function to be added to the queue.
method
add_composable_node(self, package_name, plugin_name, options = dai_ros.ROSNodeOptions())
Add a composable node to the ROS context manager.  Args: ----     package_name (str): The name of the ROS package.     plugin_name (str): The name of the ROS plugin.     options (dai_ros.ROSNodeOptions, optional): The options for the ROS node. Defaults to dai_ros.ROSNodeOptions().
method
start_pipeline(self, pipeline)
Start the camera pipeline, initializes the ROS node, and spins the ROS context manager.  Args: ----     pipeline: The pipeline configuration for the camera.
variable
method
publish_rh(self, name, color_frame, timestamp, metadata)
Publish video data to a RobotHub stream.  Args: ----     name (str): The name of the RobotHub stream.     color_frame: The color frame data to be published.     timestamp: The timestamp associated with the frame.     metadata: Additional metadata for the frame.
method
publish_ros(self, name, msg)
Publish a message to a ROS topic.  Args: ----     name (str): The name of the ROS topic.     msg: The message to be published.
method
get_image(self, stream_name) -> np.ndarray: np.ndarray
Retrieve an image from the specified stream.  Args: ----     stream_name (str): The name of the stream to retrieve the image from.  Returns -------     Image: The image retrieved from the stream in OpenCV format.
method
method
method
variable
variable
variable
module

rae_sdk.robot.perception.pipeline

constant
function
function
function
module

rae_sdk.robot.robot

class
Robot
A class representing a robot, integrating various controllers for movement, display, and LED management and interfacing with ROS2 for communication and control.  Attributes ----------     ros_interface (ROSInterface): An object for managing ROS2 communications and functionalities.     led (LEDController): Controls the robot's LEDs.     display (DisplayController): Manages the robot's display.     navigation (NavigationController): Handles the robot's movement.     audio (AudioController): Controls the robot's audio.     state (StateController): Manages the robot's state information.     perception (PerceptionSystem): Handles the robot's perception system.  Methods -------     start(): Initializes the robot's components and starts ROS2 communications.     stop(): Stops the ROS2 communications and shuts down the robot's components.
class

rae_sdk.robot.robot.Robot

method
__init__(self, robot_options: RobotOptions = RobotOptions())
Initialize the Robot instance.  Args: ----     robot_options (RobotOptions): An object containing the robot's options.
method
__del__(self)
method
stop(self)
Stop the ROS2 communications and deactivates the robot's controllers.  Ensures a clean shutdown of all components.
property
property
perception
Create perception system if it doesn't exist and return it.
property
property
property
property
property
module

rae_sdk.robot.robot_options

class
RobotOptions
A class for storing the robot's options.  Attributes ----------     name (str): The robot's name.     namespace (str): The robot's namespace.     launch_controllers (bool): Whether to launch the robot's controllers.     start_hardware (bool): Whether to start the robot's hardware.     launch_mock (bool): Whether to launch the robot's mock interfaces if start_hardware=True.     publish_state_info (bool): Whether to publish state information.
module

rae_sdk.robot.state

class
StateInfo
A class for representing the robot's state.  Attributes ----------     battery_state (BatteryState): The current state of the robot's battery.     cpu_usage (float): The current CPU usage of the robot.     mem_usage (float): The current memory usage of the robot.     temp (float): The current temperature of the robot.     disk (float): The current disk usage of the robot.     net_up (float): The current upload speed of the robot.     net_down (float): The current download speed of the robot.
class
StateController
A class for managing the robot's state.  Attributes ----------     ros_interface (ROSInterface): An object for managing ROS2 communications and functionalities.     state_info (StateInfo): Stores the current state of the robot's battery.