Skip to main content
Raily grades the searches and chat answers your endpoints serve. Each search is checked by a set of graders, and the results show up in two places in the dashboard: Both pages follow the date range at the top of the dashboard and show the Organization you have selected. Anyone with view access to the Organization can open them.
Searches that return no results are not graded. They don’t count toward any quality number. You’ll find them on the Usage tab instead.

The graders

Four graders drive the Quality board. Each one labels the evidence item by item (each result, each part of the question, each claim in an answer), and Raily computes the score from those labels. That keeps scores consistent from one search to the next. Relevance and Coverage grade retrieval. Groundedness and Answer quality grade answers, so they only run on chat turns that produced an answer. Your Account may also run other graders, such as Freshness (are the results recent enough for the question). These show in the Graders table but not on the board. Hover the info icon next to any grader name to see its description.

Read the Quality board

Go to Overview in the sidebar, then open the Quality tab. The board has up to three cards. The number in parentheses next to each card title is how many evaluations in the period got a final result. Evaluations where a grader returned no verdict are left out of the success rates. Click the info icon on a card, or click a grader name inside it, to open a popover with the formula and a worked example using your own numbers for the period.

Retrieval success rate

How well your endpoints find the right content.
  • Formula: Avg(Relevance + Coverage), the average of the Relevance average and the Coverage average.
  • Example: Relevance averages 80.0% and Coverage averages 78.9%, so the popover shows (80.0% + 78.9%) / 2 = 79.4%.
  • If only one of the two has scores in the period, the rate is that one’s average.
Below the rate, Relevance and Coverage each show their average score with a bar. The popover on each one shows Sum of Relevance scores / Number of Relevance scores × 100 and how many scores went into it.

Answer success rate

How good the chat answers are. This card appears only when the period has chat turns with an answer.
  • Formula: Avg(Groundedness + Answer quality).
  • If only one of the two has scores, the rate is that one’s average.
Below the rate, Groundedness and Answer quality show their averages, calculated the same way as above.

Output policies

Accounts that generate images get a third card. Its headline, Compliant images, is the average score of every image check on the graded images. The two highest-scoring checks are listed beside it.

Quality over time

The Quality over time chart plots one line per card that has data: Retrieval, Answers and Policies. Each line uses the same formula as its card.
  • Each point covers one hour, day or week, depending on the date range.
  • Hover a point to see the rate and how many evaluations it’s based on.
  • A hollow point means fewer than five evaluations, so read it with care.
  • A period with no scores is left off the line rather than plotted as 0%.

Dig into each grader

Under the board, the Graders section is collapsed. Click Graders to open it. Graders are grouped into Retrieval quality, Answer quality (measured on chat traffic) and Output policies. Each row shows: Use the sort menu to reorder the table, for example by Status: needs attention or Average score: low to high. Turn off Active only to include graders that ran earlier in this Account but no longer run in the selected Organization and period. Click the arrow on a row to expand it. You’ll see:
  • Score & negative feedback over time
  • Score distribution, the spread of scores in 10-point groups
  • Lowest-scoring queries, the five searches this grader scored lowest. Click Details to read the grader’s reasoning, or open the query in Evaluations.
  • Feedback, every comment left for this grader in the period, which you can filter by source and rating
Click Evaluations in the sidebar. The Assessments list shows every graded search in the period, newest first, with its average score. Narrow the list with the filters at the top:
  • All questions, Search or Deep research
  • All evals, or one grader
  • Any score, Below 50%, 50–80% or 80% and above
  • A cluster, if your search sorts queries into clusters
A row may carry a chip such as 1 evaluation failed (a grader returned no verdict) or 1 evaluation skipped. Click a row to expand it. You’ll see:
  1. Retrieved sources (or Sources given to the model for a chat turn), the results the graders looked at.
  2. Retrieval, with each retrieval grader’s score.
  3. Answer (from chat) and the Answer graders, when the search produced a chat answer.
  4. Output policies, for image generations.

Read a grader’s reasoning

Each grader writes a short explanation of its score in plain language. Click the scroll icon next to a score to show or hide it.
  • When a grader fails, errors or is skipped, its reasoning opens on its own.
  • When a grader passes, the reasoning is still there. Click the icon to see why it passed.
  • Long reasoning shows a summary first. Click Show more for the full text.
A grader that couldn’t score the search shows an Evaluation error or Skipped badge instead of a bar, and the search is not scored by that grader.

Leave feedback

Use the thumbs next to each score to tell Raily whether you agree with that grader. Use Overall feedback at the bottom of the row to rate the search as a whole. Your feedback appears in the Specific feedback and Overall feedback columns on the Quality tab. Owners, Admins and Editors of the Organization can leave feedback. Auditors and Viewers see existing feedback but can’t change it.

Turn a check on or off

Raily sets up which graders run when your Source connects. To turn a grader off or back on, contact Raily.
  • A grader that’s off doesn’t run, so it adds no cost.
  • It disappears from both pages, including its past scores.
  • If another endpoint in the same Organization still runs it, it stays visible.
  • Turning it back on brings its history back. Nothing is deleted.
You can see which graders run on an endpoint in the Evaluations section of the endpoint’s Search Behavior settings. That section also controls how graders appear: Customer-facing graders show on the dashboard in the order you set, and graders you move to Internal are hidden from these pages. Drag a grader to reorder it, or use the button on its row to move it between the two groups.

Run graders on your own Gemini key

The graders can run on your own Google Gemini API key. Go to Settings → AI models provider, add your key and assign it to Quality evaluators. With your own key, Raily can also send grader requests in batches, which halves the cost per search. Results can then take a few minutes longer to appear. Contact Raily to turn batching on.
AI models provider is available on request. Contact Raily to turn it on for your Account.