December 18: AI Troubleshooting Masterclass 🛠️

A Guide to Getting Better Results

Inspired by and adapted from Allie K. Miller's "Complete ChatGPT Troubleshooting Guide" (December 2024)

🎯 Objective

Learn how to troubleshoot and refine AI outputs for better results, enhancing your ability to work with AI as a reliable smart assistant.

🛠️ Tools

• ChatGPT
• Claude
• Gemini
• Copilot

---

Why Troubleshooting Matters

Imagine you have a brilliant new colleague who graduated at the top of their class. They're incredibly knowledgeable, eager to help, and can process information at lightning speed. But there's a catch – they think probabilistically, sometimes make creative leaps, and effectively have the memory of a goldfish. That's essentially what working with AI is like.

While we've spent time mastering the art of clear prompts and effective AI communication, equally crucial is knowing what to do when things go sideways. AI isn't like traditional software where the same input reliably produces the same output. Its probabilistic nature – the very thing that makes it creative and adaptable – also means it can sometimes:

• Veer off in unexpected directions
• Miss crucial context
• Provide overly generic responses
• Get stuck in repetitive patterns
• Sound unnaturally robotic

These aren't bugs; they're features of how AI works. Just as you might coach a talented but inexperienced colleague, you can learn to guide AI back on track when it strays. This is where troubleshooting comes in.

The Art of AI Troubleshooting

Effective AI troubleshooting isn't about "fixing errors" – it's about understanding how to:

• Recognize when responses aren't meeting your needs
• Apply specific techniques to improve outputs
• Know when to try a different approach entirely
• Build a toolkit of reliable fallback strategies

Recently, AI expert Allie K. Miller shared a comprehensive guide that brilliantly captures these troubleshooting principles. Today, we'll explore and practice these techniques, adding them to your AI collaboration skill set. Whether you're writing legal briefs, coding, or crafting creative content, these troubleshooting skills will help you get more reliable, higher-quality results from your AI interactions.

---

📝 Challenge

Today, you'll practice troubleshooting AI responses using proven methods to refine, debug, and improve AI outputs.

Choose one of these approaches:

Option A: Personal Reflection Revisit a prompt that didn't quite work for you in the past. This could be:

• A previous advent calendar challenge you want to improve
• A work-related prompt that gave underwhelming results
• A personal project that needs refinement
• Any AI interaction where you thought "there must be a better way"

Option B: Practice Scenarios Or work with one of these example scenarios:

1. Professional Context: "Transform these rough presentation notes into a polished slide deck outline about recent developments in privacy law. Include speaker notes and suggestions for visual elements."

2. Creative Context: "Create a holiday-themed story to share with family (children, grandchildren, nieces, nephews, etc.). You might try:

   • A winter tale about Baldwin the Eagle helping lost students find their way home
   • A modern remix of a classic holiday story set at Boston College
   • Your own heartwarming holiday memory reimagined as a children's story"

3. Complex Project Context: "Create an original legal case study for teaching first-year law students about property rights. Include:

   • A realistic scenario involving neighbors disputing land use
   • Detailed character backgrounds and motivations
   • Key legal principles in conflict
   • Procedural history
   • Evidence and documentation
   • Discussion questions for class"

💡 Tip: The third scenario is intentionally complex. You might find it overwhelming if approached as a single prompt - this makes it perfect for practicing how to break down and troubleshoot larger tasks.

---

Troubleshooting Step 1: Breaking Down Complex Tasks

If you chose the case study scenario or have a similarly complex task, here's how you might break it down:

1. Start with the Core Scenario:

   • First, focus just on developing the basic property dispute
   • Test and refine the scenario before adding details
   • Make sure the legal principles are clear and teachable

2. Build Character Depth:

   • Create backstories for key participants
   • Develop realistic motivations
   • Add relevant historical context

3. Add Legal Framework:

   • Layer in specific legal principles
   • Develop the procedural history
   • Create supporting documentation

4. Enhance Educational Value:

   • Craft discussion questions
   • Identify teaching points
   • Design class activities

💡 Managing Complex Projects:

Remember how we learned about context windows - that limited "viewing space" AI has for conversation history? These strategies help you work effectively within those limits:

1. Fresh Start Approach:

   • Begin new conversations for each major component
   • Helps avoid context confusion
   • Keeps responses focused and clear
   • Perfect for rapid prototyping of ideas

2. Curated Context Method:

   • Copy and paste relevant material from previous successful conversations
   • Strategically choose what context to include
   • Build on what worked while leaving behind what didn't
   • Useful for iterative improvement

3. Custom Assistant Strategy:

   • Create a custom assistant with your project's core knowledge and context
   • Include key background information up front
   • Reduce repetitive context-setting
   • Allows you to focus on refining individual components
   • Perfect for long-term or recurring projects
   • Maintains consistent context across multiple conversations

🌟 Pro Tip: Think of these approaches like different tools in your workflow toolkit. The Fresh Start gives you a clean slate, Curated Context lets you choose what to bring forward, and Custom Assistants help maintain consistent knowledge without repeatedly using up your context window.

---

Additional Troubleshooting Ideas:

Using Allie K. Miller's troubleshooting framework

When you get an initial response, identify an issue and apply one of these troubleshooting methods:

If it's too short:

• Ask to "expand on that"
• Request more specific examples
• Say "continue" or "tell me more"
• Ask for real-world applications

If it seems wrong:

• Request step-by-step reasoning
• Ask for sources and citations
• Challenge specific points
• Have it verify calculations
• Compare with known sources
• Ask about limitations and assumptions

If it's too generic:

• Start with a clear role: "You are an expert in..."
• Set the format: "Respond in the style of..."
• Specify audience: "Explain this to a..."
• Define scope: "Focus only on..."
• Add constraints: "Stay within these parameters..."

If it sounds robotic:

• Request natural, conversational language
• Provide example tones/styles you prefer
• Ask it to "write naturally"
• Request a rewrite with no jargon
• Give examples of good/bad phrasing

If the output is messy:

• Request specific formats (tables, lists)
• Ask for hierarchical organization
• Request executive summaries
• Ask for bulleted action items
• Request flowcharts for complex processes

If it seems low quality:

• Request multiple drafts/options
• Ask it to critique its own response
• Generate "radically different alternatives"
• Request pros and cons analysis
• Ask for confidence levels
• Request error checking

Emergency Fixes:

• Start a fresh conversation
• Break the task into smaller chunks
• Use a different approach
• Try a different time of day
• Consider using a different AI tool

---

🔍 The Meta-Collaboration Technique

Here's another powerful troubleshooting approach that might surprise you: when stuck, try asking the AI itself for help with your prompting. Yes, really! While it might feel strange to ask an AI how to better interact with itself, this meta-level collaboration can be remarkably effective.

Just as you might ask a colleague, "How would you prefer I phrase this request?" or "What information would help you help me better?", you can engage in this type of reflective dialogue with AI.

Here's how to apply this surprisingly effective technique:

1. Pause and Meta-Analyze:

   • Directly ask: "I'm not getting quite what I need. How could I rephrase my request to get better results?"
   • Try: "What additional context would help you understand what I'm looking for?"
   • Consider: "Can you help me break down this task into more manageable prompts?"

2. Reflect and Redirect:

   • Have the AI analyze its own previous responses: "What aspects of your previous answer could be improved?"
   • Request prompting suggestions: "How would you recommend structuring this request?"
   • Ask for clarity about misunderstandings: "Where do you think we're misaligning in this conversation?"

3. Reset and Retry:

   • Start a fresh conversation with your improved prompt
   • Avoid carrying over potentially confusing context
   • Apply the insights gained from your meta-discussion

💡 Pro Tip: While it might feel odd at first to ask an AI how to better interact with AI, remember that these systems are trained to understand their own capabilities and limitations. They can often provide surprisingly insightful suggestions for more effective interaction.

⚠️ Important: If you're ever working on a sensitive or confidential task, be careful not to include specific details when asking for prompting help. Instead, use general descriptions or analogous examples.

---

✨ Bonus Challenge

Document your troubleshooting process:

1. Take screenshots or notes of your original prompt
2. Record the steps you took to improve it
3. Share your before/after results
4. Reflect on what you learned about AI's capabilities

📝 Reflection Questions

1. Which troubleshooting method proved most effective for your needs?
2. How did understanding AI's probabilistic nature affect your approach?
3. What strategies will you incorporate into your regular AI interactions?
4. How might these skills improve your use of AI in work or study?

Remember: Working with AI is more like conducting an experiment than running a program. Sometimes you'll get exactly what you want on the first try. Other times, you'll need to guide, refine, and redirect. That's not just normal – it's part of the process!