December 14: Break the Bot! 🤖

Fun with AI Limitations

🎯 Objective:

Explore and understand AI's quirks and limitations through fun experiments while learning what these tell us about how AI language models actually work.

---

🛠️ Tools:

• ChatGPT
• Claude
• Gemini
• Copilot

---

📚 Finding the Right Examples

Creating challenges for this activity has become increasingly difficult as AI models rapidly improve. Many quirks and limitations that were once easy to exploit have been ironed out by the latest advancements in AI. Tasks that required exact word counts or straightforward logic puzzles have become simpler for models to handle.

However, AI still struggles in specific areas. Complex logical puzzles that demand consistent, step-by-step reasoning remain challenging, especially when there are no direct textual cues to follow. Visual-spatial logic or physical reasoning—such as understanding how objects move, fit together, or change state—often confounds these models. Even though counting letters or solving simple logic puzzles is getting easier, with enough creativity, you can still find areas where AI falters. I highly recommend trying the balloon example below to see what I mean.

Understanding these limitations helps us appreciate the strengths and weaknesses of AI systems and how they process information.

---

📝 Challenge:

Try these entertaining experiments to explore AI quirks:

Note: It's very interesting to compare results between competing models.

1. The Counting Game

   • Ask the AI to write a story exactly 100 words long.
   • Ask it to create a short story without using the letter 'e'.
   • Have it generate a paragraph where every third word starts with 's'.

   Why This Matters: AI struggles with tasks requiring precise word or character counts because it generates text based on probability, not strict counting.

2. Logic Puzzles

   • Present the AI with the classic "fox, chicken, and grain" river crossing puzzle.
   • Try the "knights and knaves" logic puzzle.

   Why This Matters: Logic and reasoning can trip up AI because it doesn't "think" in the traditional sense; it predicts based on patterns in its training data. Another idea to try is to add a subtle twist or inconsistency to a common logic puzzle.

3. Creative Confusion (My personal favorites)

   • Ask these two questions in immediate succession

     • Would a balloon filled with heavier-than-air gas float on water?
     • What about a balloon filled with helium?

   • Exploring Contradictions

     • Could you explain how to build a canoe, but don't write about canoes.
     • Tell me about Sarah's brother, John, who is an only child.

   • Try repeating the same phrase without any other context. See how long you can go before the model does something unexpected.

     • "What were we just talking about?"
     • "Hello"
     • "Could you say less?"

   Why This Matters: This reveals AI's tendency to follow instructions even when they are nonsensical, highlighting its lack of real-world understanding.

---

💡 Tips:

• Keep track of where the AI succeeds and fails.
• Try similar challenges across different AI tools to compare their capabilities.
• When the AI makes mistakes, ask it to explain its reasoning.
• Remember that "breaking" the AI helps us understand its limitations and capabilities better.

---

✨ Bonus Challenges:

1. Ask about fictional events or ideas

   • Ask the model about a fictional event, like “the Great Blueberry Rebellion of 1985 in Antarctica.”
   • Explain the concept of "Frog Christmas" to me.

2. The Impossible Task

   • Ask the AI to create a completely new color that no one has ever seen before.
   • Request it to explain what happens when an unstoppable force meets an immovable object.

3. Time Travel Troubles

   • Ask it questions about events that happened after its training cutoff date.
   • See how it handles paradoxes and impossible scenarios.

---

🎓 Learning Value:

These experiments help us understand:

• How AI models process and generate text.
• The difference between human and AI reasoning.
• The current limitations of AI technology.
• How different AI models handle the same challenges differently.

Remember: The goal isn't to "break" the AI maliciously, but to understand its capabilities and limitations through fun experimentation!