The ability to accurately identify anatomical landmarks in real time is crucial to ensuring safety and navigating through laparoscopic surgery. Additionally, the reduced field of view, along with the significant similarity among tissues, creates a significant barrier to using multi-label classification techniques to detect the primary target organ and nearby organs during computer-assisted interventions. Although current state-of-the-art vision-language models, such as BiomedCLIP and its prompt learning variations (BiomedCoOp) have proven successful at treating classification as a visual pattern matching problem, they have largely ignored the strong spatial and semantic relationships inherent in surgical environments. We propose a new multi-modal framework called SpatialContext that utilizes a combination of frozen visual embeddings and spatial text descriptions derived from segmentation masks to integrate scene context into the classification process explicitly.