The increasing adoption of Large Language Models (LLMs) has changed how individuals engage in knowledge exploration activities, providing new ways to explore and consume knowledge through conversational interfaces. However, these interfaces often fail to meet the accessibility needs of Blind and Visually Impaired (BVI) users. In fact, as current chat-based interfaces for LLMs, such as ChatGPT, fail to prioritize accessibility, assistive technologies like screen readers encounter significant difficulties in enabling a more accessible interaction with LLMs. While voice-based interfaces are available, current implementations are often insufficient for BVI users, offering a limited control over navigation and orientation within these systems. To address the accessibility challenges faced by BVI users with conversational interfaces for LLMs, this thesis introduces HALLM (Highly Accessible LLM), a prompt-based prototype system for enhanced and controllable voice-based interaction.