How to Use an Interrogatory LLM to Elicit Knowledge: A Step-by-Step Guide

By • min read

Introduction

When you need an LLM to handle a complex task—like designing a new feature or analyzing a detailed specification—you often have to feed it a mountain of context. That context may include user requirements, implementation guidelines, external system data, or a rambling specification that’s hard to parse. Traditionally, a human writes all that context, which can be slow and painful. But there’s an alternative: let the LLM interview you. By prompting the LLM to ask you targeted questions, you can quickly produce a rich, accurate context document. This technique, sometimes called an “interrogatory LLM,” works for building context from scratch or for reviewing existing documents. In this guide, you’ll learn step-by-step how to set up and run an interrogatory LLM session, whether you’re creating a new knowledge base or checking the accuracy of an existing one.

What You Need

• An LLM interface (e.g., ChatGPT, Claude, Gemini) that supports custom prompts and multi-turn conversations.
• A clear goal: What information do you need to capture? For example, a feature design doc, a system architecture description, or a review of an existing specification.
• A human expert (you or someone else) who knows the domain and can answer questions.
• (Optional) An existing document if you’re reviewing it for accuracy—this could be a software spec, policy manual, or any knowledge artifact.
• Patience and a willingness to correct the LLM if it forgets to ask only one question at a time.

Step-by-Step Instructions

Step 1: Define Your Objective

Before talking to the LLM, decide exactly what you want to achieve. Are you building a context report for a future LLM session? Or are you verifying that an existing document matches a human expert’s knowledge? Write down your goal in plain language. For example:

• “I need a detailed context document describing our new user onboarding flow, including UI mockups, business rules, and integration points.”
• “I need to check whether the attached API specification accurately reflects how our system actually works.”

Having a clear objective helps you craft the initial prompt.

Step 2: Prepare the Initial Prompt

Now write a prompt that instructs the LLM to interview you. The key elements:

• Tell the LLM its role: it will be an interviewer, not a writer.
• Explain the goal: it needs to ask you questions to gather all necessary information.
• Insist on one question at a time. This is critical—otherwise the LLM might dump five questions at once, overwhelming you.
• Optionally, list types of information you want covered (e.g., user roles, error handling, performance requirements).

Example prompt: “You are an expert interviewer. Your task is to interview me to gather all context needed for writing a detailed feature design document for the new user onboarding flow. Ask me one question at a time. Start by asking what the primary goal of this feature is.”

Step 3: Conduct the Interview

Start the conversation. The LLM will ask a question; you answer it. Keep your answers concise but complete. If the LLM asks multiple questions in one message, gently remind it: “Please ask only one question per message.” You may need to repeat this reminder several times—LLMs can be forgetful. As you answer, you can also provide references to external documents or systems: “For the integration with the payment gateway, please consult the Payment API spec that I’ll upload later.”

Continue until the LLM indicates it has enough information, or until you feel the coverage is sufficient. You can also ask the LLM to summarize what it has so far and ask if any areas are missing.

Step 4: Generate the Context Report

Once the interview is complete, ask the LLM to produce a structured context report based on your answers. Specify the format you need—plain text, markdown, JSON, etc. This report will be used in a separate session (perhaps with a different LLM) to perform the actual task. Example request: “Now compile all the information you gathered into a context document in markdown format. Include sections for feature overview, user stories, technical constraints, and references.”

Review the output for accuracy. You may need to ask a few clarifying questions to correct misunderstandings.

Step 5: Alternative Mode – Document Review

If you’re using the interrogatory LLM to review an existing document, start by giving the document to the LLM (upload it or paste it). Then use a prompt like: “You have received a software specification. Your job is to interview a human expert to determine if this document is accurate. Ask one question at a time, focusing on the key claims. Begin by asking whether the overall architecture described is still current.”

The human expert (e.g., a senior developer) answers the LLM’s questions. This conversational method often uncovers errors or omissions that a simple read‑through might miss, especially if the document is poorly written.

Step 6: Combine Both Modes

You can chain interrogatory LLMs: first use one to build a context document, then use another (or the same) to review it with a different expert. For instance, a product manager could use an interrogatory LLM to draft a feature spec, and then a QA engineer could use another interrogatory LLM session to validate the spec against real‑world behavior. This ensures both creation and verification are thorough.

Step 7: Apply Broader Uses

The technique isn’t limited to LLM tasks. Many people find writing difficult and prefer talking. If you need to extract knowledge from a colleague or a subject matter expert who struggles with writing, ask them to be interviewed by an LLM. The resulting text may have an “AI‑writing tang,” but it’s far better than having no documentation at all. To do this, set up the same process: the expert answers the LLM’s questions, and the LLM produces a coherent document.

Tips for Success

• Enforce the one‑question rule. LLMs naturally try to ask multiple questions at once. Remind them—firmly and repeatedly—to ask only one. You can add a system message like: “You must never ask more than one question in a single message.”
• Be specific in your answers. Vague answers lead to vague context. If you’re unsure, say so, and ask the LLM to rephrase the question.
• Use the LLM’s summarization ability. After several questions, ask for a summary to check alignment. For example: “Please list the main points you’ve gathered so far so I can confirm they’re correct.”
• Accept the AI style. The output will sound like it was generated by an LLM. That’s okay—the goal is to capture information, not to win a literary prize. You can always polish later.
• Split complex topics into multiple sessions. If the domain is huge, break it into logical parts. Each session focuses on one part, and you merge the context reports later.
• Document the prompts you use. If you repeat the process, having a library of effective prompts saves time.

By following these steps, you can turn an LLM into a powerful interviewing tool that extracts knowledge efficiently, creating reliable context for complex tasks or verifying the accuracy of existing documents. It’s a practical way to bridge the gap between human expertise and machine processing.