GitHub Copilot Custom Agent Usage Observability
Client Use Case & Proof-of-Concept Findings
A privacy-conscious approach to understanding AI consumption by custom development agents.
In a small controlled test on one environment, the proof of concept demonstrated that usage generated by different GitHub Copilot custom agents can be distinguished and attributed to their underlying AI model activity. This creates a practical foundation for agent-level usage visibility, optimization, and future cost governance across development teams. The results have not yet been validated at scale or across multiple environments.
1. Business Challenge
Organizations are increasingly introducing specialized AI agents for backend development, code review, testing, architecture, documentation, and other engineering workflows. However, standard usage reporting does not provide enough operational visibility into how those custom agents consume AI resources.
- Which custom agents are consuming the most AI resources?
- Which models are those agents using?
- How much token usage is associated with each agent?
- Which agents are comparatively efficient or expensive?
- How can AI consumption later be allocated across teams, repositories, or projects?
Can usage reported by the development environment be attributed to the specific custom agent that generated it, accurately enough to support operational analytics and future cost attribution?
2. What the Proof of Concept Demonstrated
The proof of concept was a small controlled test on a single environment, using two custom development agents, one focused on backend tasks and another focused on code review. The proof of concept was able to distinguish the agents, identify their model activity, and attribute source-reported token consumption independently.
| Custom Agent | Model | Model Calls | Input Tokens | Output Tokens |
|---|---|---|---|---|
| Reviewer | GPT-5 Mini | 3 | 85,553 | 3,116 |
| Backend | GPT-5 Mini | 2 | 55,798 | 860 |
| Unattributed activity* | GPT-4o Mini | 4 | 3,775 | 167 |
* Model calls with no traceable custom-agent ancestor. The POC did not determine their origin.
Different custom agents operating within the same development conversation could still be measured independently. This is the central technical requirement for a future company-wide observability service.
3. What the Solution Provides
- Agent-level AI usage visibility
- Model usage by custom agent
- Source-reported input and output token consumption
- Separation of unattributed activity instead of guessing
- A basis for comparing agent efficiency
- A foundation for future team-, developer-, and repository-level allocation
4. Example Business Use Cases
Agent Efficiency
Organizations can compare how much context and model activity different agents consume. This can highlight agents that require unusually large context windows or multiple model calls for similar tasks. The figures below only illustrate the kind of comparison the solution enables.
| Agent | Average Input Tokens / Call | Average Output Tokens / Call |
|---|---|---|
| Reviewer | 28,518 | 1,039 |
| Backend | 27,899 | 430 |
Illustrative, small sample (2–3 calls per agent), not a benchmark. The two agents performed different tasks, so these figures should not be read as a measure of relative efficiency.
Project and Team Allocation
In a production deployment, the same agent-level measurements can be enriched with approved company metadata such as developer identity, team, repository, and project. This enables a hierarchy of reporting from company-level consumption down to individual agents. This enrichment was not part of the POC and remains to be validated.
Potential future reporting structure
5. Privacy-Oriented Design (Intended)
The intended production approach focuses on operational metadata rather than development content. By design, the business objective should be achievable without collecting source code or developer conversations. The full privacy and security review has not yet been carried out.
| Usage Metadata (Intended Design) | Not Required by the Intended Design |
|---|---|
| Custom agent identity | Prompt text |
| Model | AI response text |
| Input/output token counts | Source code |
| Session/activity metadata | File contents |
| Timing and usage metrics | Secrets or credentials |
| Repository/project identifier (future enrichment, to be validated) | Developer terminal output |
Legend: a check mark means included in the intended design; a cross mark means not required by the intended design; a hollow circle marks future enrichment that remains to be validated.
This table describes the intended design, not a verified guarantee. What is collected and excluded will be confirmed by the full privacy and security review. Repository metadata can be sensitive and was not captured or validated in the POC.
6. What Has Been Proven vs. What Comes Next
| Proven in the POC (small controlled test) | Next Validation / Productization |
|---|---|
| Custom-agent identification | Cache and reasoning token coverage |
| Multiple-agent tracking in one workflow | Authoritative cost/billing reconciliation |
| Model-call attribution | Subagent validation and compatibility across additional Copilot versions |
| Source-reported input/output token usage | Full privacy/security review |
| Agent-level aggregation | Centralized collection, multi-device reporting and validation at scale across environments |
| Unattributed activity separated rather than guessed | Repository/project metadata enrichment |
Legend: a check mark means proven in the proof of concept; an arrow marks work planned for the next validation and productization stage, not yet completed.
7. Target Production Capability
The next stage is to transform the local proof of concept into a centralized observability capability that can aggregate usage across developers and devices under company governance. Security and access controls will be defined during productization and the privacy/security review.
Target production capability. This is the intended end state, not a system demonstrated by the POC.
8. Business Value
- Understand where AI development consumption is going.
- Identify high-consumption or inefficient custom agents.
- Optimize agent instructions, context size, model selection, and workflows.
- Create a basis for AI usage allocation by team, project, or repository.
- Support future AI FinOps, budgeting, and governance.
- Measure adoption and operational efficiency without requiring source-code inspection.
Outcome
The proof of concept shows how the organization could move from simply knowing that developers use AI tools to understanding which custom agents consume AI resources, how heavily they use them, and where optimization opportunities exist. It provides early evidence for a broader AI observability and cost-governance platform.