Visual AI for space robotics obstacle detection

Problem statement

Collision-free autonomous path execution in a dynamic and uncertain environment is an important issue which CSA seeks to resolve in order to increase robotic system autonomy for future space missions. Robotic systems operate on space infrastructure or in other environments where collisions can potentially be catastrophic. Future space systems will need to have greater autonomy in order to be less reliant on communications with the ground and to alleviate crew workload; consequently they will need to rely on multiple layers of intelligent software for supervision and safety assurance.

Cameras both built into the robot and mounted to the infrastructure provide a relatively low mass, power, and volume solution for sensing the robotic workspace. Canadarm2 for example is equipped with on-board cameras in its end-effectors and along its booms in order to allow human operators to guide and monitor its operations. Future missions such as the cislunar Gateway call for autonomous robotic operations during periods when video cannot be downlinked to the ground for oversight.

This challenge seeks to explore the feasibility of an AI-based vision system to assist or offload the need for human ground controllers to monitor clearances. For CSA, an efficient robot supervisory system translates into improved safety and operational efficiency of the mission. CSA believes that a computer vision system based on machine learning will be able to detect obstacles and monitor proximity to structure and obstacles while handling payloads. CSA can provide Mobile Servicing System and International Space Station video and telemetry data during phase 1 and 2.

Desired outcomes and considerations

Essential (mandatory) outcomes

The solution must:

• Operate in real-time, in a space environment, under all solar lighting conditions.
• Use 2D camera data (still images and video streams) from existing cameras built into a space manipulator or mounted to its infrastructure and not assume any additional sensor functionality.
• Have sufficient resolution and accuracy to detect the presence of slender objects (~2 cm wide) such as cable harnesses and EVA tethers.
• Have a false positive rate of 2 % or lower and a false negative rate of 1% or lower.
• Have to assume that the cameras could be mounted on the manipulator's booms or end-effector (e.g. may not be stationary during operation).
• Work with a data bandwidth (video stream from the camera to processor) limited to 512 kbps and an update rate of at least 2 Hz.
• Provide a 3D virtual world display of the manipulator and any obstacles detected.
• Provide obstacle detection along the entire length of a space robotic arm and about any attached or manipulated payloads.
• Be able to operate on a typical space platform such as a 2-4 core, ARM-based FPGA System on a chip, running VxWorks or Linux, a few GB of RAM, a solid-state drive, a power budget of approximately 10 Watts, Ethernet connectivity capable of transfer rates of at least 100Mbps.

Additional outcomes

The solution should:

• Have robust performance with respect to distorted and blurred images streamed from the onboard cameras due to the oscillations caused by movements of the robotic arm.
• Incur minimum budget increase to power, mass, volume, and computation resources (CPU and memory) allocated to a space robot's avionics concept.
• Investigate the feasibility of adding path-to-flight computing hardware to the on-board avionics if the computation power of the space robot's avionics is found to be insufficient.
• Be able to detect features inside of a given clearance volume or a keep out zone, for example a cylinder with depth of 2.4 m and diameter 2 m.
• Investigate multiple cameras to cover the entire volume of the keep out zone.
• Use a vision algorithm requiring a minimum number of independent camera views (expected to be part of study).
• Be applicable to a 7 degree of freedom, offset joint robot similar to the Special Purpose Dexterous Manipulator on the International Space Station.

Background and context

It is a critical safety consideration for a space servicing robotic manipulator to avoid unintended contact during operations. For example, collision prevention in Canadarm2 begins at the planning stage. Robotic trajectories are designed using a kinematic simulator that uses high-fidelity graphical models of the external International Space Station (ISS) structure, the Space Station Remote Manipulator System (SSRMS), and the Special Purpose Dexterous Manipulator (SPDM). Robotics mission designers use this tool to define a series of poses and trajectories for the manipulator. A graphical planning tool can be used to verify that the trajectory meets the needs of the mission while avoiding singularities, self-collisions, and joint limits, and respecting clearances to structure and any defined keep-out zones (e.g., a volume around a radiating antenna).

While these planned paths are a good starting point, they cannot be executed blindly for the following reasons:

• Disturbances in the station geometry due to pressure effects at docking ports and thermal expansion/contraction cannot be accounted for in the Computer-Aided Design (CAD) model.
• Differences may exist between the as-designed model and the as-built modules.
• Space vehicles have appendages such as solar arrays and antennas which can deploy or articulate to maintain their required pointing. If their articulation states are unknown, the robot must avoid their entire possible swept volume.
• The exterior configuration of the vehicle may be physically altered by Extra-Vehicular Activity (EVA) crew members, left-behind debris, or installation of logistics and science payloads which were not represented in the model at planning time.
• Faults in the attachment of external components may occur, such as cable harnesses coming loose or floating vehicle tether, and these may interfere with the operational workspace of the robotic system.
• EVA crew members may not be aware of or misjudge upcoming manipulator motion, so may accidentally be in the planned path of the end-effector or booms.

Again in the example of Canadarm2, at execution time, when the manipulator's end effector approaches proximity to the station, a human operator must observe all motions to ensure that clearance is maintained between the station structure, arm, and payload. The SSRMS elbow pan-tilt-zoom cameras, the station infrastructure cameras, or both can provide the necessary video views for this task. For SPDM operations, ideal camera views are often not available. It is important to note that the clearance monitoring hinges on the human's perceptual system to filter out noise, infer depth, and accommodate the harsh and variable on-orbit lighting conditions.

For future space missions, such as the cislunar Gateway station, the majority of robotic operations are expected to be performed autonomously with no crew onboard and no communication link to the ground. As such, the hazard control function that was previously provided by the operator watching the downlinked video must be performed by some aspect of the onboard system. The AI-based vision system is one possible solution for providing this hazard control.

It may be possible for the robot to build or correct its world model based on some high-density exteroceptive sensors (e.g., stereo camera or lidar). This approach would require accurate and reliable 3D reconstruction, registration, and segmentation, and these problems have not been solved to the point where they can be fully trusted with the safety of a crewed space vehicle without human supervision. As such, a reactive behavior-based system that uses real-time sensing to gracefully override commands which would result in collisions seems to be a more appropriate solution.

The solution being sought takes the converse approach by using existing or easily accommodated sensors (i.e., cameras), and a more advanced decision-making algorithm. AI-based approaches such as machine learning, pattern recognition, or spatiotemporal reasoning can be used to detect obstacles and monitor proximity to structure. The operation is then halted (or potentially adjusted autonomously) if any unexpected object is found to be in proximity to the arm.

An example use case of the vision system is as follows:

• A dexterous manipulator is autonomously manoeuvering to insert a payload into a berthing location.
• Mission control, and in particular the robotics personnel who planned the operation, are unaware that a slender antenna from an adjacent site has come loose, and is now floating and partially blocking the approach corridor.
• As the payload is being moved into position, the vision system estimates that there is insufficient clearance between the payload and the antenna.
• The autonomous agent halts the motion of the arm, and signals to the ground that the operation cannot continue as planned.

Links (for information only):

• "Where are the cameras on Canadarm2, Dextre and the mobile base?", Canadian Space Agency

Multiple contracts could result from this challenge.

Phase 1:

• The maximum funding available for any Phase 1 contract resulting from this challenge is : $150,000.00 CAD excluding applicable taxes, shipping, travel and living expenses, as required.
• The maximum duration for any Phase 1 contract resulting from this challenge is up to 6 months (excluding submission of the final report).
• Estimated number of Phase 1 contracts: 2

Phase 2:

Note: Only eligible businesses that have successfully completed Phase 1 will be invited to submit a proposal for Phase 2.

• The maximum funding available for any Phase 2 contract resulting from this challenge is : $1,000,000 CAD excluding applicable taxes, shipping, travel and living expenses, as required.
• The maximum duration for any Phase 2 contract resulting from this challenge is up to 24 months (excluding submission of the final report).
• Estimated number of Phase 2 contracts: 1

This disclosure is made in good faith and does not commit Canada to award any contract for the total approximate funding. Final decisions on the number of Phase 1 and Phase 2 awards will be made by Canada on the basis of factors such as evaluation results, departmental priorities and availability of funds. Canada reserves the right to make partial awards and to negotiate project scope changes.

Note: Selected companies are eligible to receive one contract per phase per challenge.

Travel

For Phase 1, a successful bidder may need to travel to the Canadian Space Agency (Longueuil, QC) for the Final Review Meeting.

Kick-off meeting

Teleconference/videoconference

Progress review meeting(s)

Teleconference/videoconference

Final review meeting

Longueuil, Quebec (anticipated), or teleconference/videoconference. All other communication can take place by telephone, or videoconference.

Solution proposals can only be submitted by a small business that meets all of the following criteria:

• for profit
• incorporated in Canada (federally or provincially)
• 499 or fewer full-time equivalent (FTE) employees^{Footnote **}
• research and development activities that take place in Canada
• 50% or more of its annual wages, salaries and fees are currently paid to employees and contractors who spend the majority of their time working in Canada^{Footnote **}
• 50% or more of its FTE employees have Canada as their ordinary place of work^{Footnote **}
• 50% or more of its senior executives (Vice President and above) have Canada as their principal residence^{Footnote **}

The official source of the Evaluation Criteria for this challenge is the Government Electronic Tendering System (CanadaBuys) (https://canadabuys.canada.ca/en/tender-opportunities/tender-notice/pw-22-01003385)

In the event of a discrepancy between the information below and the information published on CanadaBuys, CanadaBuys will take precedence.

The Bidder must complete the Challenge Stream Electronic Submission Form with a degree of information sufficient to enable Canada's assessment of the proposal against the criteria and the Evaluation Schema. The information must demonstrate how the proposal meets the criterion.

Part 1: Mandatory Criteria

Proposals must meet all mandatory criteria identified by achieving a "Pass" in order to proceed to Part 2. Proposals that do not meet all mandatory criteria will be deemed non-responsive and given no further consideration.

Mandatory Criteria

(Bidder's proposal must address)

Part 2: Point-Rated Criteria

Proposals must meet the overall minimum pass mark of 50% to be deemed responsive. Proposals that do not achieve the minimum pass mark will be declared non-responsive and given no further consideration.

Point-Rated Criteria

(Bidder's proposal to address)

Please refer to the tender notice for this challenge on CanadaBuys.

All incoming questions regarding this specific challenge should be addressed to SIC-ISC@pwgsc.gc.ca

You can also consult the Frequently asked questions about the Innovative Solutions Canada Program.

A glossary is also available.