@goodside - Generated scores need to be a single token for this. 0-100
Riley Goodside✓@goodside
Numeric scores generated in GPT-3.5/ChatGPT snap to discrete increments, causing ties in comparisons. To fix this, try a probability-weighted average using the top-5 token logprobs in GPT-3.5.
E.g. here: (50 * .7396 + 60 * .1027 + ...) / (.7396 + ...)
image not captured
Riley Goodside✓@goodside
Sampling discards information. Avoid it when you can.
Riley Goodside✓@goodside
2023-03-11Generated scores need to be a single token for this. 0-100 is fine, or 0-10, A-F, ordered bin labels (good/neutral/bad) — whatever you can correspond to a number. Also, no logprobs in ChatGPT API, so this is GPT-3/3.5 only.