Dealing with ambiguity

Written by Zac Garby, Engineering Intern

Hello everyone! My name is Zac and I worked as a summer Software Engineering intern here at Cogna. I worked on Cogna's synthesis platform, our tool which produces bespoke, precision, applications from our customers' unique requirements. My work improved the reliability and correctness of our platform by detecting, analysing, and resolving ambiguous requirements as early as possible.

My background is in programming languages, and so in my biased point of view I have come to see our synthesis pipeline as a modern, AI-guided, production-ready compiler. Calling our platform, an "autonomous software engineer", a compiler might sound odd. How is this comparable to the C compiler, translating C code to executable binaries? But over the last sixteen weeks, I've come to view these as instances of a more general paradigm. Both Cogna's state-of-the-art AI tool, and a tiny compiler written for a University programming languages course, are translators.

The biggest difference is our source language: English.

A compiler that converts a discussion about a customer's pains, problems, and needs, into executable code is still a compiler. The root of the challenges I faced is that while traditional source code is well-defined, English is most certainly not.

Human language owes a large part of its expressive power to this ability to be ambiguous; to omit specifics, and rely on human inference. As far back as the 50's, pioneering computer scientists such as Grace Hopper saw potential here, designing new programming languages to exploit programmers' deep intuitive understanding of English. Today, the advent of large language models (LLMs) allows us to deal with natural language like never before.

In this blog post, I'll look back on my internship. I'll explore the role of ambiguity in the context of Cogna's "compiler", and give some insight into how exploiting ambiguity helps produce a working, correct app from ill-defined specifications.

Natural language to code

I want to begin by giving you a slightly more concrete idea of what this compiler looks like.

The journey begins with the customer. They tell us their problems, and they want us to fix them. It’s this—a long, wandering transcript—that’s my “source code”. It’s a brand new but exciting concept to me: a compiler where the source language isn’t C, or Haskell, or Python, but English! Cogna’s source code is a description of a customer’s problems.

So we have this really bizarre problem at our startup, and I’m honestly not sure how to solve it. When we moved into our new office, we bought this beautiful brass gong. The idea was simple - someone would bang it to officially kick off our weekly all-hands meetings every Wednesday.

At first, it was great. When we were just 15 people, it was easy - someone would volunteer, or we’d have a quick rock-paper-scissors match. People loved it. But now we’re nearly 80 employees, and this gong has become the source of genuine workplace drama.

There is constant conflict around who gets to bang it next! Some people block in "gong time" on their calendars, some ask for written permission from their manager.

…

We begin with this transcript, straight from the customer in their own language.

This is a playful example, of course, but it represents the ambiguous nature of real customer stories well. The key takeaway is that this is 100% natural language. There is no structure. A lot is left unsaid, and at the same time, much of it is not relevant to the problem at hand. No solution is suggested, and no desired app functionality is explicitly asked for.

Just like a traditional compiler, the one I worked on at Cogna winds its way through several intermediate representations. An AI kicks things off, with an attempt to extract the pain points that the customer is facing, and to propose a potential solution for each.

Pain point no fair way to pick a gong striker caused by lacking gong striking records, leads to consequence meeting starts late; internal tension and conflict. Proposed resolution is create a tool to select employees fairly.

Right away, I can see more structure here. The AI is filling in gaps in a template, based on understanding of the transcript, and inferring what's missing.

Already, this semi-structured representation activates the "compiler engineering" neurons somewhere in my brain. I could imagine writing a parser for this pain-point language. Sure, there's still some English in there, but I can at least work with the syntax around it. I can start to think about traditional programming language techniques.

My "source code" was 100% natural language, and the compiler has reduced this somewhat by adding explicit structure. This is good: the goal is a real, executable app, therefore 0% natural language. We're going in the right direction, then, but we need to repeat this reification process a few times to get there.

Next, the AI attempts to design the app, given the pain points identified. Here we're looking for a rough scaffold. A description of the app's pages, how they should appear, and how they should work together.

The AI designs the app's user experience and appearance, edging even closer to a concrete implementation. At this point, the AI still has not written any programming language code. The design is still dotted with free-form English, allowing the compiler to hold off on implementation details like the database schema, or what exactly should happen when a given button is pressed.

But the user experience is now concrete and well-defined. The final step is to turn this into a usable app.

The final code synthesis requires our AI's expertise once more. All of the natural language in the specification is translated into code. Real code, in a common programming language, as if by a real engineer. In our case, Python and TypeScript. We can package this up and ship it to the customer, who can put their gong-striking problems behind them.

Time flies like an arrow, fruit flies like a banana

Translating between any of the intermediate representations presents a problem.

To see this, I'll focus on one part of the final translation step above. To display the employees table, the AI needs to write some code to fetch the name of each employee and their eligibility to strike the gong, from the database.

The value of these fields is not described explicitly. Instead, we're constrained in part by natural language. The description of an employee as "an individual in the company". Perhaps the button "pick the next gong striker" is relevant, too? Even a table's column header names are potentially relevant!

What I see in practice is that there may be multiple possibilities for such a query or, for instance, for a button’s action. These may all be consistent with the original specification—the “source code”—but they may have entirely different behaviour. My favourite example is the value of the strike eligibility column: how do we determine whether a particular employee is allowed to strike the gong?

Only if they have not struck the gong yet?

Or, only if they have already struck the gong?

Are employees only eligible to strike if they haven't had their turn in so long? How long?

Are all employees eligible, unless somehow banned? In this case, how is the next striker selected?

I suspect I know which one was intended, and likewise the AI can certainly make a guess, but in reality any of these would be consistent with the original specification. Recall that the specification was derived with an aim to "create a tool to select employees fairly". The "correct" choice, therefore, depends on what the customer considers fair.

I consider this a necessary consequence of translating from a partially ambiguous language to a fully unambiguous one: code. Ambiguity is sticky. If the compiler aims to produce a single app, then by definition it must discharge any ambiguity somewhere along the way. I can make a guess. The AI can make a guess. But no amount of guesswork can read the customer's mind.

The customer knows best

If you have ever used an LLM, you will know that they like to guess. To make assumptions when information is missing, or even to invent answers when there are none. To make things worse, they will rarely admit that they've done this. LLMs are notoriously fickle, unpredictable, and very sensitive to small changes. An important lesson I have learned during my summer at Cogna is that it can be very difficult to persuade an AI to do what you ask.

These tendencies manifest throughout Cogna's compilation pipeline. Even in a small example like my gong striking app, there are potentially dozens of ambiguities. Dozens of locations with multiple possible interpretations, each one lacking a ground truth.

As a result, the generated apps are unlikely to exactly reflect the intention of our customers. Also, it's likely that if I ran the compiler three times on the same input I'd get three completely different apps. This unpredictability is far from ideal.

Having a conversation with your compiler

I typically find that I can get a lot of mileage by approaching problems from a programming languages viewpoint, even in apparently unrelated areas. This is why I have landed so heavily on the compiler analogy. Allow me to run with this analogy a little further.

In a traditional compiler, what happens when there's a problem in your source code? Maybe you've made a typo in a variable name, or perhaps you've enabled 27 different Haskell language extensions and the compiler has lost the ability to infer the type of one of your functions. Instead of pushing through these warning lights and handing you a potentially faulty program, it stops. It complains. It flags up anything it's not sure about, and demands that you, a responsible human, fix the problem.

I like to think of this process as a conversation that I'm having with my compiler. After sorting out anything that the compiler wasn't happy with, I can give it my updated "draft". From there it can either successfully compile it for me or, more likely, come back with further problems to fix.

This idea becomes even more powerful, and even more conversational, when integrated into my text editor. In-line linting, type hints, and real-time error messages, make programming feel like a real back-and-forth iterative process.

This concept is nothing new, but I wanted to see what would happen if I incorporated the same idea in Cogna's compiler. Given source code in natural language like English, looking for "syntax errors" or "failed type inferences" makes somewhat less sense.

Instead, I begun to look at ambiguity as a Cogna-equivalent of the compiler errors and warnings we see in other languages. This raised two immediate questions:

How can the compiler detect ambiguity?

How can a human fix it?

Detecting ambiguity

To flag them up as compiler warnings, I need to have some way to automatically detect ambiguities.

I’m defining an ambiguity as a part of the pipeline—say the layout of a page, or a particular SQL query in the codebase—which could be implemented in several ways. These several interpretations must all be consistent with the English-language user requirements, but must also be functionally different from each other. I’m not concerned with syntactic variation in the generated code, or how exactly a query was expressed, if they behave the same from the user’s point of view.

To me, this definition quite naturally suggests an outline of an algorithm to identify these ambiguities autonomously and systematically.

I look for potential ambiguity throughout the pipeline. Whenever the compiler translates from one stage to the next, it converts a portion of the natural language into something more structured. My idea was to hook into this process, wherever it happens. At these transition points, my tool asks the LLM to generate not just one possibility, but multiple.

The more the better. My tool prompts the AI to employ "blue sky thinking". If the compiler can think of a particular implementation when pushed to be as creative as possible, then I am okay considering it a reasonable interpretation, consistent with the specification.

Once the AI has exhausted its creative juices, I can move over to more precise analysis. Again, I only want to flag something as ambiguous if the various interpretations are genuinely distinct. For a piece of code, this means a semantic difference, something I check by actually executing snippets of the code in sandboxed mock environments. Or, for a page in the app design, this means I can ignore superficial differences like the background colour.

To make this concrete, let's take a look at an example. In the final stage, the compiler translates from an app design to an implementation in real, production-ready code. If my app contains a button, this stage of compilation is tasked with writing the code to handle the button being clicked.

The button, in my app design, is tagged with a note, in natural language, saying that it should "fairly select the next employee to strike the gong". But what is "fair"? How should the selection be shown? Should the employee be sent a congratulations email, or a calendar invite?

Previously, the compiler would have silently made a guess. My ambiguity reporting tool allows it to speak up, flagging the ambiguity as a warning, and if desired, halting the compilation process until it's clarified.

Resolving ambiguity

In a traditional compiler, it's not always easy to fix compiler errors, but I generally know what I've got to do to get rid of that red underline in my text editor. Equally as important, I know that when I have done, it should stay fixed. If Python tells me I have a syntax error on line 53, then I can fix that error, and forget about it.

The compiler errors I have now introduced do not have these important properties. It's not clear precisely how I can change the wording in my source code, written in English, to sufficiently clarify an ambiguity that the LLM is complaining about. Equally, if I am able to satisfy it, there's no guarantee that it won't change its mind later on. I might change some wording elsewhere with unintended effects. The LLM may even just change its mind; maybe the model is updated, or maybe the wind has just changed direction. As I say, LLMs are fickle beasts.

These properties make the whole process incredibly tedious, which brings me onto the second major part of my project. How can I build a tool which not only identifies ambiguous parts of the app, but also provides some way to resolve these ambiguities in a reliable and repeatable way?

At the risk of stretching the compiler analogy too thin, you might want to think of this like the "auto-fixes" we get for a range of minor errors in our text editors nowadays. Instead of manually changing our code to resolve a syntax error, or a missing import, we can often click a little lightbulb icon and see it automatically fixed for us. An alternative, more blunt analogy is the ability that most compilers and code linters give us to ignore certain warnings. In Python, I'm shown an error message if I make a type error, but I can easily override this if I know that I'm in the right.

I took inspiration from these two concepts, applying them to my brand new ambiguity warning system. I implemented a feedback mechanism.

My tool collects together all of the ambiguities that it can find, and then comes back to the customer. It presents them as questions, using the domain-specific terminology that they used in their original specification, and avoiding asking about the nitty-gritty details. Questions like "what counts as 'fair' when choosing the next gong striker?", or "can somebody be a proxy gong-striker if they are not eligible themselves?".

As well as the questions, my tool presents some options to choose between. I take these directly from the ambiguity report, which if you recall, constructs such a set of possibilities as an intrinsic part of the detection process! In other words: the ambiguity detector builds a constructive proof that an ambiguity exists, and the tool can exploit this to suggest some explicit choices.

So the customer tells my tool that “fair” means “selecting a random employee who has not struck the gong in four weeks”. What can I do with that to resolve the ambiguity? To help the compiler consistently choose this interpretation when generating the app? As I mentioned, the chaotic nature of LLMs means that just bubbling the decision back up into the source code—patching the natural language input—would be too unreliable.

Instead, I opted to keep these decisions separate. When the customer clarifies the ambiguity at a particular location, my resolution tool simply makes a note of this. From then on, the Cogna compiler can refer to these notes throughout the compilation process, neatly and reliably cutting down the degree of ambiguity in the app.

I like to think of these resolutions, these notes to the compiler, as a "cheatsheet".

Ambiguity: a helpful compiler diagnostic?

I hope I have convinced you that we can derive value from a programming languages outlook on life. In this case, viewing an AI-powered autonomous software engineer as a compiler, and viewing natural language's inherent ambiguity as "errors"; compiler diagnostics, in the same vein as syntax errors or type mismatches.

Both provide an early warning. Traditional compiler errors tell us as programmers that something doesn't smell right, or that it perhaps lacks information. My ambiguity reporting tool does the same: it informs the customer that something's unclear, and that it's unsure how to proceed. In both cases, this allows for a "conversational" approach to writing code. We can fix errors as they come up, until we have none left, in this efficient feedback loop.

In Cogna's context, this early warning demonstrably improves the reliability of the compiler. Recall that one of the issues I set out to solve was the compiler's unreliability: that it was likely to generate three different apps if run three times on the same input. With my tool identifying the possibility for this early on and asking for a decision before continuing, we find that the compiler is far more reliable and repeatable.

Both also provide peace of mind. Haskell's compiler, with its rich type system, lets me specify so much statically that often I don't even feel the need to run my code to be happy that it's correct. Well typed programs do not go wrong, after all.

We experience the same with ambiguity reporting. I like to think of a customer's source code, their natural-language problem description, as code written in a language with a very weak type system. JavaScript, say. It's difficult for that customer to feel confident that the generated app precisely solves the issues they had in mind, in the same way that it's difficult for me to not feel uneasy about what my JavaScript will do when I run it.

My ambiguity resolution tool, then, is analogous to a stronger type system. Annotating my code with its data types gives me a much higher confidence that it will do what I intended. A customer annotating their source code with their intentions, via my tool's generated "cheatsheet", gives them the same confidence boost.

This isn't just a hypothetical. I've looked at past instances where ambiguity arose in practice due to unclear requirements, and where user feedback has indicated undesirable outcomes, and confusion or frustration with the compiler. Retrospectively applying my tool, I've found that the problematic ambiguity can now be detected in many cases. This confusion, frustration, and wasted time, would therefore have been significantly reduced had my tool existed at the time.

I was extremely pleased to see these results. Newly encountered ambiguities and misunderstandings are no different, suggesting that my ambiguity diagnostic tool will be a useful one to have in the future toolbox. It'll save time, avoid frustration, and lead to a better, more reliable compiler.

A reflection on my summer

During my internship at Cogna, I wrote some code. I designed and built a tool to solve a problem. This tool is now integrated into our daily workflow, and is helping real people do their job better!

The direction of my project from my first day back in June to today, my last, was in no way predictable. From identifying the problem, experimenting with countless possible solutions, to building and iterating on the tool itself, my project was an exploration, as much as anything else. A research project.

I personally got a lot out of it. Working at Cogna gave me the freedom to apply my own background and my love of programming languages to a real-life problem. At the same time, I was able to draw on the expertise of my colleagues. I think that it's this combination of near-endless flexibility and creative freedom, whilst still being part of a tightly integrated engineering team, that pushed my project to be what it is. It's in this environment that I got to spend sixteen weeks this summer learning about, and contributing to, an exciting new area.

A big takeaway for me has been the extent to which compilers and autonomous AI systems are related. Designing systems such as ours as translations between a series of layers of abstraction is powerful. The closer the source code is to English, the more accessible it becomes; exactly what Grace Hopper predicted decades ago.

It's been said that modern AI will replace the need for engineers. I disagree. From my point of view, designing AI systems like this one does not propose to replace programmers. Instead, it promises to expand the definition of what a programming language, and therefore what a programmer, can be.

Want to work on interesting problems like this?

This is the end of my internship, but I'm looking forward to coming back late next year when I finish my studies.

Does this kind of work appeal to you, too? Are you somebody who would enjoy working in a place which provides interesting and impactful problems, whilst allowing you freedom and creativity in how you solve them? If so, you'd probably fit in well at Cogna!

Check out our careers page for more details, and to apply.

PostScript: Ambiguity detection in production

Here's a screenshot showing my system running in production after I left.

Zac Garby is a third-year PhD student in the Functional Programming Lab at the University of Nottingham. His research explores the derivation, by calculation, of compilers, type systems, and other programs.

He's fascinated by the areas of computer science that call upon mathematical formalities to not just explain, but to lead us naturally towards, powerful techniques for programming and software development. Program synthesis is exactly this, which brought him to intern at Cogna in the summer of 2025.