It’s likely you and your organization have accumulated a wealth of knowledge as you have grown, but if you’re struggling to make that knowledge accessible to your teams or customers, you’re not alone. 

Traditional knowledge bases, wikis, websites, and other document repositories are the foundation of your knowledge ecosystem, but keeping them organized can be difficult, and the user experience isn’t always easy to navigate. Finding answers amongst hundreds or thousands of resources is time-consuming and burdensome, and it can take sifting through dozens of pages to confidently answer a simple question. What if you had a staff of people charged with pre-reading, filtering, and condensing the answer for you – like a personalized guide to the entire ecosystem? AI may be the answer.

Recently, our team had the privilege of helping one of our clients solve this very problem. The challenge? Enable a diverse user community to retrieve information instantly and effortlessly across a vast and ever-changing knowledge ecosystem. Our solution? A cutting-edge generative AI chatbot, powered by Azure’s OpenAI services.

In this blog post, we’ll take you through the contours of the project and offer some insights about how we used our proven product management techniques to design and deliver the new chatbot.

Complementing Website Navigation with a GenAI Chatbot

Our client operates a public-facing website brimming with a vast array of resources in varying formats. Based on the results of user research, they learned that their website navigation needed to be optimized, but an improved navigation and content rewrite was only one part of their ambitious plan. They also wanted to offer a more inclusive, accessible, relatable experience that would dramatically improve how users find the answers they need.

Enter the state-of-the-art generative AI chatbot. Since ChatGPT’s consumer launch in late 2022 sparked a surge of interest in this exciting technology, usage of AI has skyrocketed in nearly every industry. As this type of experience has become more familiar to a wider audience, it’s emerging as an appealing alternative for users who prefer a conversational experience to browsing or searching a website in a more traditional way. For that reason, our client selected Microsoft’s Azure OpenAI services to power their GPT chatbot. Then, they turned to us to implement their website redesign and build out the chatbot application. Together, we launched the chatbot in just six weeks. Here are just a few of our learnings along the way. 

Design a Great User Experience

At Nuvalence, we’re strong believers in the principles of human-centered design, and our approach to the chatbot was no exception. 

Intuitive User Interface

Recognizing the diverse backgrounds and varying levels of technical experience within the organization’s audience, we prioritized creating a user-friendly interface for first-time chatbot users. We also drew inspiration from ChatGPT’s existing user experience. 

We found that several aspects of the user experience were especially consequential in the context of a chatbot.

1. Welcome page – Since chatbots are still a relatively new concept for many users, a welcome page can help orient them and set the tone for their interactions. In our case, we highlighted the chatbot’s key capabilities (and limitations) as well as basic guidance on how to interact with it. Since our client was eager to collect real-time user feedback to improve the experience over time, our welcome page also included a link to a feedback form.
   
2. Legal disclaimers – Although the technology is constantly improving, AI can introduce risk when users interact directly with it. Depending on the industry, there may be legal or regulatory requirements to communicate that risk to the user up front. Our client was hyper-aware of this, so we worked closely with their legal team to include appropriate disclaimers regarding the chatbot’s abilities and restrictions. Both the capabilities and limitations of the chatbot are described comprehensively within the user interface.
   
3. Sample questions – Understanding your target audience’s most common questions, and using them as clickable shortcuts, is an effective and relatable way to introduce users to the experience. In collaboration with subject matter experts, we identified sample questions and integrated them as clickable buttons on the interface, allowing users to initiate conversations and experience the chatbot seamlessly.
   
4. Clean conversation interface – Today’s tech-savvy users have high expectations for an elegantly-designed experience that’s easy to interact with on any device. We prioritized a clean and content-focused user interface, featuring a minimalistic input area for user requests and maximum screen space for chatbot responses.
   
5. Supporting reference links – Establish trust and credibility with your users by offering direct links to the supporting content in your knowledge base. Our collection of supporting reference links connects users to relevant website pages, allowing them to verify the answers provided by the chatbot and explore related topics in depth, as required. 
   
6. Accessibility – Human-centered design is focused on delivering an equitable and accessible experience to every user, every time. To ensure an inclusive user experience, we partnered with an accessibility vendor to verify compliance with WCAG standards, including keyboard navigability for interactive elements, proper screen reader compatibility, and adherence to color contrast requirements for all visual elements.

Implement Guardrails and Safety Measures

Ensuring a great user experience is key to AI adoption, but you can only truly delight users if you earn their trust. They may be wary of AI-powered experiences due to their novelty and negative stories in the media. Making sure you build in strong guardrails is crucial to the success of any GenAI chatbot.

Grounding

Ensuring that a chatbot provides accurate and relevant responses is paramount. We carefully grounded our chatbot in the data residing on our client’s website, limiting its responses to questions within the scope of the available resources. We accomplished this by developing a web crawler that systematically reviewed the entire website domain, effectively establishing it as the chatbot’s knowledge base. Given our client’s frequent updates to their website content, we implemented a schedule that would run the crawler multiple times a day, ensuring that the chatbot’s knowledge remained current with the latest website publications. 

In addition to grounding the chatbot’s knowledge, we provided specific instructions for it to politely decline responses to questions falling outside its designated domain. Users making such queries received a courteous message along the lines of ‘I’m sorry, I don’t have information on that topic. Please try another query or topic.’ This approach ensured that users received accurate and contextually relevant answers, enhancing their overall experience.

Content Filtering

A successful chatbot must be able to detect and block inappropriate requests, thereby upholding content standards and maintaining a positive user experience. In order to provide a safe and respectful environment for users, our generative AI chatbot leverages the Content Filtering Service provided by Azure. 

When the content filter identifies a prompt that falls outside of acceptable content parameters, it takes immediate action. In such instances, the user will receive a system-generated message indicating that the specific question or prompt cannot be addressed. Any offending prompts are swiftly removed from the ongoing conversation.

Personally Identifiable Information (PII) Filtering

The importance of safeguarding personally identifiable information (PII) against potential breaches and unauthorized access is widely acknowledged. The risks of exposing PII, from sensitive personal details to contact information, are well known and can have severe consequences. In our specific use case, we were able to ensure the chatbot had no access to any PII data. However, we recognized that users may be inclined to share PII information with the chatbot during the course of their conversation, just like they might during an interaction with a real human agent. We wanted to ensure that no user-provided PII was stored in any system logs, or sent to any of our dependent services, so we opted to filter all user prompts for PII. To meet this requirement, we integrated OpenAI’s PII filtering service into our system, enabling us to detect and promptly remove any PII found in user requests.

If the chatbot detects PII during an interaction, it immediately removes the data and alerts the user. For instance, if a user provided their email address alongside a relevant question, the chatbot would respond with: ‘I have removed any personally identifiable information from your message to ensure your security and processed your request.’ Following the alert, the chatbot will finish responding to the query without referencing the removed PII.

Furthermore, this same data scrubbing was applied to our logging practices, ensuring that no PII data was captured in our data logs.

Rate Limiting and System Protection

Cost should be considered with any developed service to manage budget expectations. Especially given that our chatbot is available to the public, we wanted to make sure the client was able to budget predictably. To prevent unforeseen expenses, we collaborated with the organization to establish a rate limiting threshold. This threshold would be activated during periods of high concurrent user activity. In such instances, the chatbot politely informs users of the situation, offering a message such as: ‘I apologize for the inconvenience. I am currently experiencing a high volume of traffic. Please try again in a few minutes.’ This proactive approach ensures a balanced user experience while keeping costs in check.

Additional abuse protection measures included integrating ReCAPTCHA v3 to deter inappropriate usage and safeguard user interactions, challenging only those who exhibit suspicious behavior. This not only maintains the integrity of the chatbot’s interactions, but also enhances user security and prevents high-usage attacks that could add significant load to the system.

Test the Response Quality

Chatbots are inherently non-deterministic; the generative AI models they rely on produce varied outputs for the same input, influenced by multiple factors, which makes conventional testing methods less effective. In addition to the usual UX/UI testing that would apply to any product, evaluating the quality of the chatbot’s responses requires new tactics.

To prepare our test plan, we first engaged a group of business stakeholders to curate a comprehensive suite of expected user requests. Each of these requests was accompanied by a summary detailing the characteristics of a high-quality response. 

Next, we conducted a series of manual tests. Using the expected user requests as a guide, multiple testers posed similar questions to the chatbot and captured its responses. These testers assessed whether the chatbot’s responses met the predetermined expectations or fell short. We triaged the results, and prioritized the types of improvement that were most important to the client.

Going forward, we’re working with the organization’s QA team to develop an automated testing framework for response quality evaluations. Our first step is to use a testing framework to automatically retrieve responses in a single file, which can be reviewed and graded by a subject matter expert. Over time, further automation will enable us to rapidly evaluate the impact of prompt and configuration changes on the overall quality of responses.

Empower Your Users with a GenAI Chatbot 

Imagine being able to give your users the freedom to decide how they get information from your website. How much better would that experience be? And how much freedom would you have to continually add knowledge while making strategic choices about how and when to redesign the website? With a generative AI chatbot, all this is within reach. 

When implementing an OpenAI-powered chatbot for our client, we learned many valuable lessons about how to design a great customer experience, protected with common-sense guardrails, that could be effectively tested before it launched just six short weeks after project start. 

Looking for some more technical tips on how we made this solution a reality? Stay tuned for the next post in this series, where we’ll dive deeper into the technical requirements and implementation details for our chatbot. 

What’s the difference between a generative AI chatbot and a virtual agent? Both simulate human conversations, and eliminate the friction that comes with navigating a complex website or IVR. Virtual agents, though less focused on engaging in open-ended questions, are designed to complete specific tasks for their human counterparts. Check out our series on virtual agents to learn more.

Let’s talk about your future.

LET’S TALK