AI and delivery
AI can make work faster. Did it make delivery better?
Measure the delivery system, find where faster work creates new constraints, and use AI to investigate without letting a model decide what the data means.
Coding and delivery
Faster coding isn’t the same as faster delivery.
A faster activity improves delivery only to the extent that it changes the performance of the whole system.
Before AI32 days total
After AI29 days total
| Process step | Before AI | After AI |
|---|---|---|
| Waiting to start | 8 days | 8 days |
| Development | 6 days | 3 days |
| Code review | 4 days | 4 days |
| Testing and approval | 9 days | 9 days |
| Waiting for release | 5 days | 5 days |
- 50%Development faster
- about 9%Whole request faster
If you only measure coding activity, AI can look transformative while the customer barely notices the difference.
Moving constraints
AI can move the constraint.
When one part gets faster, the constraint can move somewhere else.
If developers produce work faster but review, testing, security approval or release capacity doesn’t change:
- Development gets faster.
- More work reaches code review.
- The queue grows.
- WIP and age increase.
- Cycle time may increase.
AI could therefore make one activity more efficient while making the overall system more congested.
Measuring impact
Measure the system, not the AI activity.
Activity measures show how a tool is used. Delivery outcomes show whether the system became better at delivering software.
AI activity measures
- Lines of code
- Pull requests created
- AI suggestions accepted
- Prompts
- Coding time
These measures may tell you something about tool use or a specific activity.
Delivery outcomes
- Cycle Time
Did work actually reach customers faster?
- Throughput
Did the system actually deliver more?
- Work in Progress
Did faster production create more unfinished work?
- Work Item Age
Is current work getting stuck?
- Flow Efficiency
Did waiting decrease?
- Cycle Time per Process Step
Where did time move after AI was introduced?
- Defect Work
Did defect throughput or cycle time change?
Jira
Azure DevOps
Trello
Asana
CSV and JSON
Don’t use an activity metric to answer an outcome question.
Before and after
Establish a baseline and measure what happened afterward.
Capture how the delivery system behaves before introducing AI, then compare what happens after the change.
Before
Establish cycle time, throughput, WIP, flow efficiency and relevant process-step performance.
Change
Introduce the AI capability to the relevant part of the workflow.
After
Observe what happens to the system.
Investigate
If something meaningfully changes, determine where and why.
A metric moving does not prove AI caused it.
Before-and-after data can show that delivery changed after an intervention. It does not, by itself, prove causation.
If AI is already in use
Earlier history depends on the source. In Azure DevOps, moving the dashboard date range back starts a one-time full sync. In Jira, older finished work loads only when a start date is set. Uploads cover only the uploaded period.
Make the comparison more credible
- Mark the date the change was introduced.
- Compare with teams that did not make the change.
- Note other changes made during the same period.
- Allow work started after the change to finish before judging cycle time.
Process performance
Find where the constraint moved.
Look at process-step cycle time, WIP accumulation, aging and flow efficiency to see where time and congestion moved.
A second illustrative example: Development fell from 8 to 4 days; testing rose from 4 to 9.
| Process step | Before AI | After AI |
|---|---|---|
| Development | 8 days | 4 days |
| Testing | 4 days | 9 days |
- Did AI make developers faster? In this illustrative example, clearly yes.
- Did the system become better at delivering software?
Cycle Time Breakdown
- In Development
- Code Review
- Ready for SIT
- SIT
- UAT
- Awaiting Release
- Blocked time
AI for analysis
AI can also help you understand delivery.
Delivery systems generate extensive evidence. AI can help investigate it, provided the underlying evidence is computed first.
AI should investigate the evidence. It shouldn’t invent the evidence.
Analysis sequence
Compute first. Let AI read second.
Separate computation, investigation and judgment so each part of the system does the work it is suited to do.
- Delivery dataThe board’s delivery history
- Deterministic computationCycle timeThroughputWIPFlow efficiency
- AI investigationWhat changed?Which stage is involved?Which items contribute?What evidence supports the finding?Where should I look?
- Human decisionDoes the context support this conclusion?Should we intervene?What should we change?
The numbers are computed, not generated. AI interprets the evidence, it doesn’t calculate the underlying metrics.
Deterministic detection
Some questions don’t need AI.
Statistical signals should be detected with math and fixed rules, not a language model’s opinion.
Has cycle time crossed a natural process limit?
This is a mathematical question.Has a run pattern appeared?
This is a mathematical question.Nave uses natural process limits and fixed detection rules to distinguish signals from normal variation.
Detect
MathInvestigate
AIIntervene
HumanMeasure
Data and math
A real investigation
Follow the evidence to a decision.
This investigation shows how a finding moves from evidence to human judgment.
Initial finding
The Investigator identifies a repeated timing pattern in the computed stage data.
AI investigation
This identical timing pattern suggests batched work may be waiting on a shared dependency.
“Three SIT items share identical stage timing (4.96 days)”
Evidence
Three SIT items and two Ready for SIT items each show exactly 4.96 days.
“Items “1127”, “1128”, and “1129” all show exactly 4.96 days in SIT, and items “1130” and “1131” both show exactly 4.96 days in Ready for SIT.”
Suggested action
Identify what is blocking SIT test execution for “1127”, “1128”, and “1129”
“establish a policy of continuous pull into SIT.”
Human decides
The team reviews the evidence and decides what actually changes.
Human judgment
Nave investigates. You decide.
AI can investigate the delivery evidence. You bring the organizational context.
AI may identify
- Where work accumulated
- Which items contributed
- What changed
- What the delivery evidence shows
AI may not know
- A regulatory deadline changed.
- A customer escalation altered priorities.
- Half the testing team was temporarily reassigned.
- A deliberate release freeze was in place.
- A dependency was accepted for strategic reasons.
AI gives the decision-maker evidence faster. It doesn’t remove the decision-maker.
No forced insight
“Nothing meaningful found” is a valid AI result.
An analysis system should not manufacture an insight simply because the user asked for one.
An AI analysis system should be able to say that the available evidence does not support a meaningful finding. The built-in instructions tell the Investigator not to investigate to fill a quota. It can return no findings, shown as an empty panel.
Delivery forecasting
You don’t need generative AI to forecast delivery.
Monte Carlo simulation answers delivery questions probabilistically from actual throughput history. A language model isn’t needed to calculate the forecast.
Probabilistic forecasting
Uses historical delivery data, simulations and a probability distribution.
- How much can we finish by this date?
- When will this amount of work likely be done?
Generative AI
Useful for explaining, investigating, summarizing or interacting with evidence.
Use the right technique for the question.
Not everything becomes better because we put AI in front of it.
Leadership questions
AI changes what leaders should ask.
Move the conversation from tool adoption and code production to delivery performance, system behavior and reliable commitments.
Instead of asking
- How many developers are using Copilot?
- How much code did AI generate?
Ask
- Did total delivery time change?Cycle Time
- Where did waiting time move?Cycle Time Breakdown
- Did throughput increase?Throughput
- Did WIP increase with it?WIP
- Did variability change?Cycle Time Scatterplot
- Did quality-related work increase or decrease?Throughput Run Chart
- Did the constraint move downstream?Cycle Time Breakdown
- Are commitments becoming more reliable?Due Date Performance
Enterprise AI trust
Questions to ask before letting AI analyze delivery data.
Understand what the model receives, how findings are produced, what is retained and which decisions remain outside AI.
| Questions to ask | How Nave answers |
|---|---|
| What data is sent to the model? | The AI Investigator sends computed metrics, stage percentiles and statistics, current items with IDs pseudonymized by one-way hashing, stage blockers, the dashboard name and ID, workflow stage names, and date range. It does not send item names, URLs, or descriptions. |
| Are metrics calculated by the model or outside it? | Metrics are computed deterministically outside the model. The AI only reads them. |
| Can you inspect the evidence behind a finding? | Each finding includes its evidence, next steps and a “See why” view with analysis and numbered evidence. |
| Can the AI return no finding? | Yes. Its built-in instructions say not to investigate to fill a quota. Zero findings is a valid outcome and appears as an empty panel. |
| Is customer data used for model training? | Nave does not train or fine-tune AI models on customer data. AI providers are used through their APIs for inference only. |
| Can AI be disabled? | The AI Investigator can be switched off for an account. That switch does not disable a separate OpenAI call made when a dashboard is created to suggest column mapping from the workflow’s status names. |
| What is retained and for how long? | Completed Investigator requests and responses are stored for traceability. The AI cache is kept for 30 days, and stored AI results are deleted with the account. Failed runs and column-mapping calls are not stored. |
| What decisions remain deterministic? | Metric computation and signal detection remain deterministic. |
| What decisions remain human? | The output is advisory. People review the evidence and decide whether to act. |
Evaluation framework
How to evaluate an AI change in your delivery system.
Start with the intended delivery outcome, observe the whole system and keep measuring after the change.
Before the change
State the expected outcome
not “adopt AI”; for example: reduce development time without increasing downstream queues
Establish the baseline
measure the relevant delivery metrics before the change
Make the change
Introduce the change
know which part of the workflow you’re affecting
Watch the whole system
don’t measure only the activity AI directly touches
After the change
Look for meaningful signals
separate normal variation from genuine system change
Investigate where the effect appeared
did time disappear, or move elsewhere?
Decide whether the outcome improved
use the business goal, not AI adoption itself, as the success criterion
Keep measuring
see whether the change persists
The principle
AI changes parts of the delivery system. Measure the whole system to understand the effect. And when you use AI to analyze that system, keep facts, interpretation and human decisions separate.
Further reading
Explore AI & Software Delivery
Go deeper into measuring AI’s impact, investigating delivery evidence, forecasting work and governing AI use.
Measuring AI impact
Measure whether faster activities improved the performance of the whole delivery system.
AI for delivery analysis
Use AI to investigate computed evidence while keeping decisions with people.
AI and forecasting
Understand when simulation answers delivery questions better than generative AI.
AI governance
Review data handling, security information and the boundaries around AI decisions.