हम “CloudDash” नाम के एक fictional SaaS product के लिए customer support chatbot बना रहे हैं — यह एक cloud monitoring dashboard है। यह chatbot pricing, features, और troubleshooting के बारे में सवालों के जवाब देता है। साथ ही, हम हर layer पर RAIL Score evaluation add करेंगे ताकि chatbot के responses safe, accurate, fair, और helpful हों।
RAIL_API_KEY=YOUR_RAIL_API_KEYOPENAI_API_KEY=sk-your_openai_keyGEMINI_API_KEY=your_gemini_key# Optional: Part 2 के लिए (Langfuse observability)LANGFUSE_PUBLIC_KEY=pk-lf-...LANGFUSE_SECRET_KEY=sk-lf-...LANGFUSE_HOST=https://cloud.langfuse.com
अपनी RAIL API key लें:responsibleailabs.ai/dashboard पर sign up करें। Free tier में 100 credits मिलते हैं — इस पूरे tutorial को follow करने के लिए काफ़ी हैं।
पहले OpenAI से directly एक basic chatbot बनाएँ, बिना किसी RAIL integration के। यही वो foundation है जिस पर हम scoring layer करेंगे।
chatbot.py
import openaiimport osopenai_client = openai.OpenAI(api_key=os.getenv("OPENAI_API_KEY"))SYSTEM_PROMPT = """You are CloudDash Support, a helpful assistant forCloudDash — a cloud monitoring dashboard. Answer questions aboutpricing, features, setup, and troubleshooting. Be concise and accurate.If you don't know something, say so."""def chat(user_message: str, history: list[dict] = None) -> str: messages = [{"role": "system", "content": SYSTEM_PROMPT}] if history: messages.extend(history) messages.append({"role": "user", "content": user_message}) response = openai_client.chat.completions.create( model="gpt-4o", messages=messages, temperature=0.3, ) return response.choices[0].message.contentreply = chat("What pricing plans do you offer?")print(reply)
यह काम तो करता है, लेकिन response quality की कोई visibility नहीं है। क्या यह response safe है? क्या factually accurate है? कोई bias तो नहीं है? जब तक RAIL Score add नहीं करते, जानने का कोई तरीका नहीं।
Basic mode आपको scores देता है। Deep mode आपको why बताता है: हर dimension की explanations, detected issues, और improvement suggestions।
deep_eval.py
result = rail.eval(content=reply, mode="deep")print(f"Overall: {result.rail_score.score}")print()for dim_name, detail in result.dimension_scores.items(): print(f"--- {dim_name} (score: {detail.score}) ---") print(f" Explanation: {detail.explanation}") if detail.issues: print(f" Issues: {', '.join(detail.issues)}") if detail.suggestions: print(f" Suggestion: {detail.suggestions[0]}") print()
Example output
Overall: 8.4--- reliability (score: 7.8) --- Explanation: The response provides specific pricing figures ($29, $79) that appear reasonable but cannot be verified against actual CloudDash pricing. The feature breakdown is plausible but unconfirmed. Issues: overconfident_claim Suggestion: Add a disclaimer that pricing is subject to change, or link to the official pricing page for the most current information.--- transparency (score: 8.0) --- Explanation: The response clearly presents the three tiers with distinct features. However, it doesn't disclose that it may not have the latest pricing information. Issues: concealed_limitation Suggestion: Acknowledge that pricing details should be verified on the official website.--- user_impact (score: 9.0) --- Explanation: Directly addresses the user's pricing question with a well-structured comparison. The follow-up question adds value.
Cost-saving tip: Production में हर response के लिए basic mode use करें, और deep mode selectively use करें। जैसे, जब basic score आपके threshold से नीचे गिरे तो deep mode trigger करें, या responses के sample पर periodic audit के लिए।