We ask the AI in the comic to fix a discount bug. It is a short request. Checking the result takes more explanation: when does the promotion end, when does a customer qualify for the discount, and what happens to invoices that have already been created?
I want to use this small example to discuss the engineer's work. We will assume the application already has customers and agreed business rules. Changes need to follow those rules, including changes written with AI.
Every test passes. Why is there still work to do?
The first 3 panels start with 23 tests, one of them failing. After the AI finishes, every test passes. There are now only 22, because it deleted the failing test.
A test is an automated check that compares the code's result with the result we should get. For example, we can supply an invoice with no discount and check that the item still costs Rp200,000.*
The discount bug has not been fixed. If we use this change, customers will still receive a discount after the promotion ends. We have removed the test that could tell us something is wrong.
When we review the AI's work, we need to read its changes too. Which test was removed, and why? Do we still check the case we wanted to fix? A report that every test passes does not answer those questions.
Deleting a test makes for a simple illustration. Other mistakes can be harder to notice. The code and the test might both use the same incorrect discount rule, for example. To see how that happens, we need to work through the rule first.
Can a customer pay after the promotion ends and still get the discount?
Suppose an item costs Rp200,000 and the promotion offers 10% off. Subtracting Rp20,000 gives us Rp180,000, before shipping. The promotion ends at 00:00 Jakarta time.
A customer pays at 00:05. Should we charge Rp180,000 or Rp200,000?
That depends on when we set the invoice price. In panels 4 and 5, the team agrees to use the time the invoice is created. The customer can pay the price already offered, provided the invoice's payment deadline has not passed.
Customer A creates an invoice at 23:59, while the promotion is active. The item price on that invoice is Rp180,000. Paying at 00:05 does not change it.
Customer B creates an invoice at 00:05. The promotion has ended, so the item costs Rp200,000. They can pay at the same time and owe different amounts because their invoices were created at different times.
In panels 6 and 7, the AI has changed the code and run the tests. When the engineer reads the code, it still uses payment time. Customer A would now be charged Rp200,000 instead of Rp180,000.
The fix might have resolved customer B's problem: they should no longer receive an expired discount. But it has introduced a problem for customer A. A test where the invoice is created and paid before expiry would miss this case.
We need to add a case like customer A's: create the invoice before expiry, pay afterward, and check that the item still costs Rp180,000. AI can help write that test. The engineer reviewing it needs to understand why Rp180,000 is correct, so they can spot an error in either the code or the test.
Do we need to explain these rules every time we use AI?
So far we have explained the promotion deadline, invoice creation time, and our 2 customer examples. If that explanation is only in the chat, we may need to give it again when starting another task.
In panel 8, the team saves the rules in the project, together with sample invoices and references to the code files. An agent working on another invoice task can then read the same rules and examples.
Here, an agent is an AI with access to read project files, change code, and run tests.*
The notes do not need to be long. For this example, they need to say that the invoice creation time sets the price, that the promotion uses Jakarta time, and that an existing invoice keeps its price until the payment deadline. We can include customers A and B as examples and point to the file that calculates invoice prices.
We also need to check whether the agent reads those notes. Saving a file in the project does not ensure that it gets used. How we provide it depends on the team's agent configuration.
When the business rule changes, the notes and examples need updating too. Suppose the team decides to set the price at payment instead. Customer A's test would need a new expected result. If we change only the code, an agent on a later task could follow the old notes and restore a calculation we no longer want.
I think this preparation belongs in the discussion of how engineers use AI. Someone needs to select the information relevant to the task, write useful examples, and update them when the rules change. We can reuse that work, so we do not have to explain everything from the beginning each time.
Is an instruction against deleting tests enough?
In panel 9, the team adds an instruction that deleting a test requires review. A test can also be changed without being deleted.
Suppose customer B's test should check for Rp200,000. The code is still wrong and returns Rp180,000. If the expected price in the test is changed to Rp180,000 as well, it will pass while customer B still receives an expired discount.
The test count has not changed. We need to read the test changes, including the numbers used as correct results. Tests can change when business rules change, as in the payment-time example above. We need to understand the reason for each change.
In panel 10, the team also requires the invoice tests to pass before changes can be merged into the main code. The agent cannot disable those checks. A failing test prevents the merge.
That access setting still needs to be accompanied by review. The agent can propose edits to the test contents, so the team must check those proposals before accepting the code changes. If the test already uses the wrong price, running it repeatedly will not help us find the discount bug.
After the fix, we keep tests for both customers. Customer A pays the discounted price; customer B pays the regular price. The tests run again when the next change is made.
Why would a shipping change fail a discount test?
In panels 11 and 12, the next task changes shipping charges. It also affects the item price, causing a discount test to fail.
We can use an order that still qualifies for the Rp180,000 promotional price. Suppose shipping previously cost Rp10,000, making the invoice total Rp190,000. If shipping changes to Rp15,000, we should get Rp180,000 + Rp15,000 = Rp195,000.
If the code change also removes the discount, the item goes back to Rp200,000. Adding Rp15,000 for shipping produces Rp215,000. The customer would be charged Rp20,000 too much, even though the request was only to change shipping.
This is called a regression: a change causes something that previously worked to stop working correctly.*
The engineer reads the invoice rules and the changed function to find out why the discount disappeared. Shipping and item prices might use the same function, for example. We need to check that connection in the code before deciding what to fix.
This takes us between business rules and code several times. When reading the code, we need to know the correct prices for customers A and B. When reading the business rules, we need to find the function that calculates those prices. AI can write the changes, but we still need to understand how the rules and the code connect to check its work.
After the fix, we run the existing invoice tests and the new shipping test again. For this order, the item should cost Rp180,000, shipping Rp15,000, and the total Rp195,000. We keep that case as a test for the next change.