Facing a data-related challenge? Explore some of the most common issues and discover how we can help you tackle them, gain deeper insights from your data, and find practical solutions.
AI agent observability through connected enterprise data
Q-Sensei connects enterprise data from business systems, operational processes, and technical knowledge sources, helping you provide better context for AI agents, investigate their behavior, identify performance gaps, and improve results.Learn more
Why did an AI agent make this decision?
The challenge
When an AI agent produces an unexpected result or takes an incorrect action, I need to understand how it reached that decision. I want to investigate the prompts, responses, tool calls, retrieved information, and business data involved in the execution.
How Q-Sensei helps
Q-Sensei helps you connect agent execution information with relevant enterprise data. You can explore request-level and end-to-end traces, examine prompts and responses, and inspect interactions with tools, APIs, and external services. You can also investigate the data and context available at each stage of an execution.
You get a more complete view of how an agent arrived at a particular result. By comparing executions and exploring relationships, you can identify patterns that may explain unexpected behavior.
Trace an observed action through the recorded steps
01Request R41Order support request
02Order lookupTool call: order A17
03Policy lookupTool call: policy P02
04Send for reviewRecorded action
Available evidenceOrder record A17 · Policy revision P02 · Returned tool responses
The agent looked up the order and delivery policy before sending the request for review.
Data typically used in this investigation
Agent execution traces
Record the steps and events involved in an agent execution. They help you reconstruct how an interaction progressed.
Prompts and responses
Show the instructions provided to the agent and the outputs it generated. They help you understand how inputs and intermediate responses relate to the final result.
Model and prompt configurations
Document the models and prompt versions used during execution. They help you compare behavior across different configurations.
Tool and API calls
Record interactions with external tools, services, and APIs. They help you investigate which operations the agent initiated and what information it received.
Retrieved documents and source data
Identify the information made available to the agent. They help you assess whether the retrieved context was relevant to the request.
Business rules and process definitions
Describe the requirements and processes relevant to an agent's task. They help you assess whether its actions and outputs align with business expectations.
Why does an AI agent produce inconsistent results?
The challenge
When an AI agent produces different results for similar requests, I need to understand what is driving the differences. I want to compare interactions, identify unusual behavior, and investigate whether changes in prompts, models, tools, input data, or configurations may be contributing factors.
How Q-Sensei helps
Q-Sensei helps you analyze historical agent activity and compare results across requests, models, prompt versions, and tools. You can filter interactions by request type, execution characteristics, configuration, and outcome. Statistical and rule-based analysis can help highlight anomalies, while AI-assisted analysis, where configured, can suggest factors worth investigating.
This makes it easier to spot patterns in inconsistent results and narrow down potential explanations. You can focus your investigation on the differences that matter.
Compare similar requests under different conditions
Same request type · Two recorded interactions
Request S21
Prompt version
Revision A
Retrieved context
Earlier record
Recorded outcome
Completed
Request S22
Prompt version
Revision B
Retrieved context
Updated record
Recorded outcome
Sent for review
Compare the prompt revision and retrieved record for the request that went to review.
Data typically used in this investigation
Agent requests and responses
Capture the inputs received and outputs generated across interactions. They help you compare results for similar requests.
Execution traces
Record the sequence of operations during agent execution. They help you identify differences in execution paths and tool usage.
Model and prompt versions
Identify the configurations used for individual interactions. They help you investigate whether configuration changes coincide with different results.
Tool and API responses
Show the information returned by external services. They help you assess whether differences in external data or responses may affect outcomes.
Evaluation datasets and scores
Provide reference examples and available quality measurements. They help you compare agent behavior against defined expectations.
Error logs, configuration history, and user feedback
Document failures, configuration changes, and user assessments. They help you identify recurring issues and potential factors behind inconsistent results.
How can we ensure AI agents have the right enterprise data and context?
The challenge
Our enterprise information is spread across different systems and formats. I need to connect relevant data, understand relationships between records, and make the right information available to AI agents so they can work with a more complete and relevant business context.
How Q-Sensei helps
Q-Sensei helps you prepare and connect structured and unstructured enterprise data from different sources. You can explore information across databases, business applications, documents, and operational systems, and analyze the content, attributes, entities, and relationships within that information. Depending on your implementation and available interfaces, connected information can also be made accessible to analytical applications and AI workflows through interfaces and APIs.
The result is a more integrated view of relevant enterprise knowledge across domains. You can make information more accessible to AI workflows and provide agents with business context that extends beyond individual systems or data sources.
Connect enterprise records and knowledge around a task
Which delivery rule applies to order A17?
Business applicationOrder A17Customer reference C04
Master dataCustomer C04Delivery region North
Knowledge repositoryDelivery policy P02Applies to region North
Metadata and lineageSource historyCheck versions and freshness
Context to inspectSource records · Entity relationships · Relevant policy · Data quality
The customer’s delivery region links the order to the policy. Check the policy version when reviewing the agent’s context.
Data typically used in this investigation
Enterprise databases and data warehouses
Store structured business and operational records. They help make relevant information available across different data domains.
Business applications and APIs
Provide information from systems that support business activities and processes. They help extend the available context beyond individual databases.
Data lakes and document repositories
Store structured and unstructured information in different formats. They help you access a broader range of enterprise knowledge.
Knowledge bases and technical documentation
Contain organizational, product, and technical knowledge. They provide relevant domain context for AI agents.
Master, reference, and entity data
Provide standardized identifiers and describe relationships between records. They help you connect information about the same entities across systems.
Metadata, data lineage, and quality records
Describe data structures, origins, transformations, and quality checks. They help you assess where information comes from and whether it is suitable for analytical and AI use.
How can we measure AI agent performance and identify regressions?
The challenge
I need to know whether our AI agents are performing as expected. Response time alone is not enough: I also need to understand reliability, execution behavior, usage, and output quality over time, and identify regressions after changes to models, prompts, tools, or deployments.
How Q-Sensei helps
Q-Sensei helps you monitor and analyze agent performance through dashboards, historical comparisons, and configurable alerts. You can examine metrics such as latency, errors, reliability, and usage, and filter results by agent, model, or request type. Request-level and end-to-end traces let you investigate individual executions, while historical comparisons help you examine changes after configuration updates, model changes, or deployments. AI-assisted analysis, where configured, can help identify patterns and areas for further investigation.
With a clearer picture of agent performance over time, you can spot changes, investigate potential regressions, and understand how updates affect reliability and output quality. This supports more informed decisions about agent improvements.
Spot a latency regression, then inspect the affected traces
Latency for comparable requests Same latency scale; lower is better
Inspect the marked interaction and compare its trace with the earlier release. Sample data.
Data typically used in this investigation
Performance, latency, and reliability metrics
Measure response times, throughput, errors, and successful executions. They help you assess operational performance and identify changes over time.
Agent requests, responses, and execution traces
Record interactions and the steps involved in agent execution. They help you investigate performance at both request and workflow levels.
Evaluation results and benchmarks
Provide measurements against defined expectations. They help you assess output quality and identify changes in task performance.
Model and prompt versions
Identify the configurations used during execution. They help you investigate performance differences after updates.
Tool and API performance records
Capture the activity and performance of external services. They help you identify potential delays or reliability issues outside the agent itself.
User feedback and release records
Document user assessments and changes to agent applications or environments. They help you relate performance changes to user experience and specific releases.
How can we govern AI agent access, security, and business rules?
The challenge
We need to make sure AI agents operate within our security requirements and business rules. I need to control access to data and tools, protect sensitive information, and maintain oversight of actions that could affect business processes.
How Q-Sensei helps
Q-Sensei helps you connect agent activity with relevant identity, access, configuration, policy, and audit information. You can investigate which users, agents, and services accessed information or initiated actions, and examine those activities in the context of your security requirements and business rules. Where the relevant controls are implemented or integrated, you can also support investigations involving sensitive data detection and redaction, policy checks, scoped permissions, and approvals or escalations. Depending on your implementation, configured remediation capabilities may include rollback, kill switches, quarantine, or incident response.
You gain better visibility into how agents interact with enterprise information and systems. This supports investigations against your security requirements and gives you greater oversight of actions that may affect business processes.
Inspect agent activity against recorded access and policy controls
Access recordRead order statusRecorded within the assigned scope
Approval recordChange delivery address
Human approval recorded
Policy checkExport customer recordsRecorded as blocked by an integrated control
The integrated control blocked the export. Further steps, such as rollback or quarantine, depend on the configured response.
Data typically used in this investigation
Identity and access records
Document users, roles, permissions, and access activity. They help establish which identities can access specific data and services.
Security policies and business rules
Define requirements and restrictions for data access and business operations. They provide criteria for investigating agent activity.
Agent, tool, and API configurations
Describe how agents and their connected services are configured. They help you assess whether settings align with security and operational requirements.
Approval and escalation records
Document decisions requiring human review. They help establish how sensitive or exceptional actions are handled.
Audit logs and agent action histories
Record system activity, access events, and actions initiated by agents. They help you trace activities and investigate whether they followed defined requirements.
Sensitive data classifications and incident records
Identify information requiring special handling and document security events. They help you investigate potential exposure and recurring exceptions.
How can we reduce AI agent costs and resource consumption?
The challenge
Our AI agent costs are difficult to understand. I need to see which agents, models, tasks, and workflows consume the most resources, and determine where optimization could reduce spending without compromising task quality.
How Q-Sensei helps
Q-Sensei helps you analyze usage and cost information across agent interactions, models, and connected services. You can compare token consumption, model usage, API activity, and infrastructure metrics, and relate these measurements to execution traces, latency, and available quality indicators. Dashboards and historical comparisons help you examine cost patterns. Where supported by your implementation, you can also monitor quotas, budgets, and configurable thresholds.
You can see where resources are being consumed and which agents, models, or workflows account for the greatest costs. This makes it easier to identify optimization opportunities and assess them alongside performance and task quality.
Connect resource usage to a repeated execution step
01
Request R41Start the task
02
Model call AInitial call
03
Tool lookupOrder information
04
Model call BRecorded retry
05
ResponseAssess the outcome
Activity behind resource use
Model usage
Call ACall B · Retry
Tool activity
Order lookup
Check alongside usageRuntime and task quality
Compare the repeated call’s usage with its outcome before deciding whether the workflow should change.
Data typically used in this investigation
Model and token usage
Record model activity and token consumption. They help identify which models and interactions account for the greatest usage.
Agent requests, sessions, and execution traces
Document agent activity and the operations involved in execution. They help relate resource consumption to specific tasks and workflows.
Model and API pricing information
Define the applicable costs for model and service usage. They help translate usage measurements into estimated costs.
Tool and API usage records
Capture interactions with external services. They help identify usage patterns and potential sources of additional cost.
Latency, runtime, and infrastructure metrics
Measure execution duration and resource consumption. They help identify performance bottlenecks and infrastructure-intensive workloads.
Configuration history and task quality results
Document changes to models and agent settings alongside available outcome measurements. They help assess optimization opportunities in relation to performance and task quality.
How can we evaluate AI agent task success and output quality?
The challenge
I need to know whether our AI agents are actually completing their tasks successfully. I want to evaluate their outputs against expected results, compare quality across different use cases, and understand what may be influencing poor outcomes.
How Q-Sensei helps
Q-Sensei helps you analyze agent outputs alongside execution information, evaluation results, and relevant enterprise data. You can compare interactions, tasks, and use cases, examine available evaluation scores, and investigate how output quality relates to execution traces, retrieved information, and configuration history. The evaluation itself depends on the datasets, criteria, and evaluation systems available in your implementation.
You can recognize where agents meet expectations, where they fall short, and which factors may be influencing their results. These insights give you a clearer basis for improving task outcomes.
Compare task completion with the quality of the output
Same task requirement · Answer with supporting source evidence
Task T21
Execution record
Completed
Observed output
Answer and supporting source supplied
Available evaluation
Meets the criteria
Task T22
Execution record
Completed
Observed output
Answer supplied without supporting source
Available evaluation
Evidence requirement not met
Investigate what may influence qualityExecution traces · Retrieved sources · Model and prompt versions
Both tasks returned an answer. Only T21 included the source required by the evaluation criteria.
Data typically used in this investigation
Agent requests and responses
Show the inputs received and outputs generated. They provide the basis for reviewing individual task results.
Evaluation datasets, expected outcomes, and quality scores
Define reference examples and record available measurements. They help you compare agent outputs against expectations.
Execution traces
Document the steps involved in agent execution. They help you investigate how execution behavior relates to task outcomes.
Retrieved documents and source records
Identify the information available to the agent. They help you assess whether relevant context was used.
User feedback and historical outcomes
Record user assessments and results from previous executions. They help you identify recurring patterns and outcomes that may not meet expectations.
Business rules, task requirements, and model versions
Define expected behavior and identify the configurations used to generate outputs. They help you investigate differences in task success and output quality.
Where do AI agent workflows fail?
The challenge
Our agent workflows sometimes fail, time out, or get stuck during execution. I need to understand which steps are affected, whether failures are recurring, and how they impact the overall reliability of our processes.
How Q-Sensei helps
Q-Sensei helps you analyze request-level and end-to-end execution traces alongside relevant operational records. You can inspect model calls, tool invocations, API interactions, response times, and error information to investigate where workflows encounter problems. Where this information is captured, you can examine repeated calls, retries, and timeouts, and compare affected executions with successful ones. You can also relate execution behavior to configuration history, tools, models, services, and deployments.
You gain a clearer picture of recurring failure patterns, the workflow steps affected, and the potential causes that warrant further engineering investigation. You can also understand how individual failures affect the reliability of your overall processes.
Locate the recorded interruption in a workflow
Comparable requests · Captured execution records
W31 · Completed
01
RequestTask received
02
Model callTool requested
03
Tool / API callResponse returned
04
Agent responseCompletion recorded
W32 · Interrupted
01
RequestTask received
02
Model callTool requested
03
Tool / API callTimeout recorded
04
RetryFurther timeout recorded
No completion recorded for W32Compare tool responses, service records and configuration history to investigate the interruption.
W32 timed out on the tool call and again on retry. Check the service logs for those calls.
Data typically used in this investigation
Execution traces and agent sessions
Record the sequence of steps and events during agent execution. They help identify where workflows deviate from their expected paths.
Tool, Model Context Protocol (MCP), and API calls
Record interactions with models and external services. They help identify which operations were initiated during execution.
Tool and API responses
Show the results returned by external services. They help investigate errors, unexpected responses, and communication issues.
Error, exception, and latency logs
Record failures, unexpected events, and response times. They help identify recurring errors, delays, and timeouts.
Reliability and historical execution metrics
Provide records of successful and failed executions. They help assess workflow stability and compare failed and successful runs.
Model, prompt, configuration, and deployment history
Document changes to agents, tools, and their environments. They help investigate whether workflow failures coincide with specific changes or releases.
Give your AI agents better context. Improve their performance.
Q-Sensei connects your enterprise data and AI agent activity, giving you the context to understand how agents behave, investigate unexpected results, and identify opportunities to improve performance.
