Accurate, real-time segmentation of different anatomical structures is essential for assisting surgeons during complex neurosurgical procedures. While pre-operative study often involves the use of MRI and other imaging techniques, neurosurgical navigation provides real-time assistance relying solely on video data. This is needed for identifying critical areas with high precision, as the relative positions of soft tissues can change during the procedure, and errors in anatomical identification can directly impact patient safety and procedural success. These scenes often involve subtle structures, such as arterial branches or aneurysms, that may only become visible partway through a procedure. Training models for this task requires dense, per-pixel annotations, which are labor-intensive and time-consuming, limiting the availability of fully labeled data in medical contexts.