- Overview → Quality (app.raily.ai/dashboard/quality) gives you the big picture: success rates, a trend line, and a table of every grader.
- Evaluations (app.raily.ai/dashboard/evals) lists each graded search, with every grader’s score and the reasoning behind it.
Searches that return no results are not graded. They don’t count toward any quality number. You’ll find them on the Usage tab instead.
The graders
Four graders drive the Quality board. Each one labels the evidence item by item (each result, each part of the question, each claim in an answer), and Raily computes the score from those labels. That keeps scores consistent from one search to the next.
Relevance and Coverage grade retrieval. Groundedness and Answer quality grade answers, so they only run on chat turns that produced an answer. Your Account may also run other graders, such as Freshness (are the results recent enough for the question). These show in the Graders table but not on the board.
Hover the info icon next to any grader name to see its description.
Read the Quality board
Go to Overview in the sidebar, then open the Quality tab. The board has up to three cards. The number in parentheses next to each card title is how many evaluations in the period got a final result. Evaluations where a grader returned no verdict are left out of the success rates. Click the info icon on a card, or click a grader name inside it, to open a popover with the formula and a worked example using your own numbers for the period.Retrieval success rate
How well your endpoints find the right content.- Formula:
Avg(Relevance + Coverage), the average of the Relevance average and the Coverage average. - Example: Relevance averages 80.0% and Coverage averages 78.9%, so the popover shows
(80.0% + 78.9%) / 2 = 79.4%. - If only one of the two has scores in the period, the rate is that one’s average.
Sum of Relevance scores / Number of Relevance scores × 100 and how many scores went into it.
Answer success rate
How good the chat answers are. This card appears only when the period has chat turns with an answer.- Formula:
Avg(Groundedness + Answer quality). - If only one of the two has scores, the rate is that one’s average.
Output policies
Accounts that generate images get a third card. Its headline, Compliant images, is the average score of every image check on the graded images. The two highest-scoring checks are listed beside it.Quality over time
The Quality over time chart plots one line per card that has data: Retrieval, Answers and Policies. Each line uses the same formula as its card.- Each point covers one hour, day or week, depending on the date range.
- Hover a point to see the rate and how many evaluations it’s based on.
- A hollow point means fewer than five evaluations, so read it with care.
- A period with no scores is left off the line rather than plotted as 0%.
Dig into each grader
Under the board, the Graders section is collapsed. Click Graders to open it. Graders are grouped into Retrieval quality, Answer quality (measured on chat traffic) and Output policies. Each row shows:
Use the sort menu to reorder the table, for example by Status: needs attention or Average score: low to high. Turn off Active only to include graders that ran earlier in this Account but no longer run in the selected Organization and period.
Click the arrow on a row to expand it. You’ll see:
- Score & negative feedback over time
- Score distribution, the spread of scores in 10-point groups
- Lowest-scoring queries, the five searches this grader scored lowest. Click Details to read the grader’s reasoning, or open the query in Evaluations.
- Feedback, every comment left for this grader in the period, which you can filter by source and rating
Review a single search
Click Evaluations in the sidebar. The Assessments list shows every graded search in the period, newest first, with its average score. Narrow the list with the filters at the top:- All questions, Search or Deep research
- All evals, or one grader
- Any score, Below 50%, 50–80% or 80% and above
- A cluster, if your search sorts queries into clusters
- Retrieved sources (or Sources given to the model for a chat turn), the results the graders looked at.
- Retrieval, with each retrieval grader’s score.
- Answer (from chat) and the Answer graders, when the search produced a chat answer.
- Output policies, for image generations.
Read a grader’s reasoning
Each grader writes a short explanation of its score in plain language. Click the scroll icon next to a score to show or hide it.- When a grader fails, errors or is skipped, its reasoning opens on its own.
- When a grader passes, the reasoning is still there. Click the icon to see why it passed.
- Long reasoning shows a summary first. Click Show more for the full text.
Leave feedback
Use the thumbs next to each score to tell Raily whether you agree with that grader. Use Overall feedback at the bottom of the row to rate the search as a whole. Your feedback appears in the Specific feedback and Overall feedback columns on the Quality tab. Owners, Admins and Editors of the Organization can leave feedback. Auditors and Viewers see existing feedback but can’t change it.Turn a check on or off
Raily sets up which graders run when your Source connects. To turn a grader off or back on, contact Raily.- A grader that’s off doesn’t run, so it adds no cost.
- It disappears from both pages, including its past scores.
- If another endpoint in the same Organization still runs it, it stays visible.
- Turning it back on brings its history back. Nothing is deleted.
Run graders on your own Gemini key
The graders can run on your own Google Gemini API key. Go to Settings → AI models provider, add your key and assign it to Quality evaluators. With your own key, Raily can also send grader requests in batches, which halves the cost per search. Results can then take a few minutes longer to appear. Contact Raily to turn batching on.AI models provider is available on request. Contact Raily to turn it on for your Account.