Min P style sampling - an alternative to Top P/TopK

[kalomaze]

Feature request

This is a sampler method already present in other LLM inference backends that aims to simplify the truncation process & help accomodate for the flaws/failings of Top P & Top K.
Min P.

What Min P is doing is simple: we are setting a minimum percentage value that a token must reach to be considered during sampling. However, this is not a hard limit. The minimum will 'scale' based on the top token's probability. So, if you have a Min P value of 0.1 (for example), that would mean your base Min P requirement is 10%. So if your top token is 25%, that means it will only consider tokens that have at least 2.5% probability.

This method subjectively seems to improve results across the board with no noticeable downside, and has been merged into the following FOSS LLM backends:

• llama.cpp
• vllm
• text-generation-webui (through both the HF loaders and llama-cpp-python)

I would suggest a default of 0.05.

Motivation

I noticed certain 'flaws' in the popular Top P sampling method:

• When the model does not have sufficient confidence/concentration on the next token candidate(s), it's possible for the sampler to consider many tokens that are highly unlikely compared to the few choices it has confidence in.
• Top K helps limit the amount of 'low confidence' tokens period as a supplement to Top P, but this often comes at a cost of token choice diversity (often arbitrarily).
• In addition to this, Top P can sometimes cut reasonable tokens. What if there's a 90.1% probability token, followed by a 9% probability token? A Top P value of 0.90 would completely gloss over the 9% token in this instance.

For this reason I made Min P which seems to have positive reception across the board.

Your contribution

I may consider making a PR for this.

[ArthurZucker]

fyi @gante 🤗

[gante]

Hi @kalomaze 👋 Thank you for opening this issue!

In addition to Temperature, Top p, and Top k, which apply distribution-agnostic transformations, we have three other distribution-aware transformations:

1. Typical P Decoding
2. Epsilon Sampling
3. Eta Sampling

These techniques do a similar thing to what you mention: they apply a "Top p"-like transformation, adjusted by the probability distribution.

Since we already have similar techniques, backed up by papers with benchmarks, I'm reluctant to add this technique without further benchmarks. Maintenance is a heavy long-term burden in transformers that we want to contain 🤗

[kalomaze]

Hi @kalomaze 👋 Thank you for opening this issue!

In addition to Temperature, Top p, and Top k, which apply distribution-agnostic transformations, we have three other distribution-aware transformations:

1. Typical P Decoding
2. Epsilon Sampling
3. Eta Sampling

These techniques do a similar thing to what you mention: they apply a "Top p"-like transformation, adjusted by the probability distribution.

Since we already have similar techniques, backed up by papers with benchmarks, I'm reluctant to add this technique without further benchmarks. Maintenance is a heavy long-term burden in transformers that we want to contain 🤗

The scaleability of Min P in comparison to Top P seems to objectively be more consistent beyond just theorycrafting.

Min P also highly interpretable in comparison to Locally Typical sampling which gets into denser, more subjective interpretations of information theory, which begs to question whether or not it's overdesigned. This makes Typical sampling less intuitive to use for the end user.

In addition to this, Typical sampling, Epsilon sampling, and Eta sampling as techniques have seen extremely limited real world adoption in terms of open source LLM interfaces, which, at large, have continued to use Top K and Top P in their wake. If not those two, Mirostat has seen mild popularity, but I would argue the latter two samplers (Epsilon sampling, Eta sampling) are perhaps less proven in terms of subjective quality.

In conclusion, Min P:

• Is more interpretable to end users and developers compared to the methods you listed. This has less risk of unintended behavior in terms of achieving the same goal as Top P / Top K when compared to typical sampling, which is less proven in the 'real world'.
• It has been proven to scale more consistently in comparison to Nucleus sampling in practice, as mentioned earlier
• It has consistently seen positive reception and adoption from the open source language model community at large to the point where most inference backends (vllm, llama.cpp, exllamav2, text-generation-webui's HF loaders, etc) have adopted it:

I will also note that a common issue for open source language models is the lack of truly objective metrics for testing beyond manual human analysis; so any apparently 'standard' testing metrics should be given serious scrutiny before they are considered absolute and final measures in which to compare sampler methods.

If there are any specific metrics you would like to see on any specific models, I can try to provide them to support my case beyond the subjective results and widespread adoption of the technique (which I figured would stand out on their own, but having numbers would be beneficial... assuming we can trust the numbers, which is an assumption I'm hesitant to make without sufficient fundamental evidence for their use beyond "arxiv papers used it")

[gante]

@kalomaze precisely because in the past we've added techniques that had some results but ended up not having much use (like Eta sampling) I'm asking for additional validation :) For instance, Eta sampling had a blind human preference test, where it was shown as preferred over top p, with a relatively low sample size (N=294). However, the upside (and the marketing) was not large enough, so the community decided to stick with simpler, established techniques like top p.

Just because other repos have merged your technique, it does not make it inherently good. ML is a data-driven science, so let's collect data -- I have yet to see any data beyond a few examples. Note that this is nothing against your creation, I actually agree with it in principle. transformers is a large library with a few maintainers, we have to be conscious of what we add here.

A good test would be to compare your technique against others with blind human preference 🤗 There is nothing better than human preference -- I'd be happy to participate in the evaluation.

[kalomaze]

A good test would be to compare your technique against others with blind human preference 🤗 There is nothing better than human preference -- I'd be happy to participate in the evaluation.

Do we have enough people who are willing to test / evaluate this to rule out the margin of error, though? The main thing we are looking for is to minimize the included outliers when improving the truncation schemes (and those are usually low probability to begin with), and outliers are going to be hard to test for without sufficient data if you sample normally, unless we change the sampler to only pick the least likely token (as a way to measure the truncation consistency directly).

I've done exactly that before for Top P and Min P and I saw that Min P was an obvious improvement. Would you like me to reproduce that experiment but with Typical sampling? (Llama.cpp, my inference engine of choice, has a broken implementation of Typical sampling at the moment but there is a PR to fix that I can use, and Eta/Epsilon just aren't adopted anywhere else in the LLM world so I'd have to learn how to use Transformers to test those, which seems like it will be necessary for my future LLM tests)

I'm also aware that an appeal to popularity isn't hard evidence, but I think it's a stronger marker in this case than it would otherwise be given the context of LLM benchmarks and especially certain metrics (e.g perplexity) being dubiously unreliable markers of quality in the ML space.

[gante]

Do we have enough people who are willing to test / evaluate this to rule out the margin of error, though?

Between your reddit and my twitter/LI reaches, we will definitely have more than enough people to run a proper study. If you agree to build the interface for the study (e.g. through a HF spaces), I'd be more than happy to promote it! I also have the power to allocate GPUs to a space in order to run the study 💪

The main thing we are looking for is to minimize the included outliers when improving the truncation schemes (and those are usually low probability to begin with), and outliers are going to be hard to test for without sufficient data if you sample normally, unless we change the sampler to only pick the least likely token (as a way to measure the truncation consistency directly).

I agree that the biggest difference is in the outliers. However, each output may have tens or hundreds of tokens, so the effect of bad "1% probability tokens" is not that hard to observe :) If there is noticeable human preference after >1000 samples, then we can be sure that it makes a difference.

Also, if the test turns out to be a success, you'd gain much more power over the distribution of your technique :D There are no questions over human preference.

especially certain metrics (e.g perplexity) being dubiously unreliable markers of quality in the ML space.

100% agreed

[kalomaze]

Do we have enough people who are willing to test / evaluate this to rule out the margin of error, though?

Between your reddit and my twitter/LI reaches, we will definitely have more than enough people to run a proper study. If you agree to build the interface for the study (e.g. through a HF spaces), I'd be more than happy to promote it! I also have the power to allocate GPUs to a space in order to run the study 💪

The main thing we are looking for is to minimize the included outliers when improving the truncation schemes (and those are usually low probability to begin with), and outliers are going to be hard to test for without sufficient data if you sample normally, unless we change the sampler to only pick the least likely token (as a way to measure the truncation consistency directly).

I agree that the biggest difference is in the outliers. However, each output may have tens or hundreds of tokens, so the effect of bad "1% probability tokens" is not that hard to observe :) If there is noticeable human preference after >1000 samples, then we can be sure that it makes a difference.

Also, if the test turns out to be a success, you'd gain much more power over the distribution of your technique :D There are no questions over human preference.

especially certain metrics (e.g perplexity) being dubiously unreliable markers of quality in the ML space.

100% agreed

Understood; I've never made a HF space, so that'd be new territory for me, though I'll look into it for sure (since having empirical data would be helpful.)

What would be a fair comparison value to Top P? Or would you prefer something where all methods all evaluated (that might be too aggressive, though?) The next problem, I think, is finding an 'equivalent scale' for all methods. The scale of Min P is obvious and understood; but for Epsilon & etc it's difficult for me to determine...

[gante]

@kalomaze I'd suggest to start simple, going against top p alone. Less work and straight to the point. If we realize we're gathering enough participants, then we can expand it to multiple models and multiple strategies, for a better overview.

I can help you with any roadblock or questions you have along the way: the results are very much of my interest! 💛

(and I'm crossing my fingers for Min P to be successful!)

[kalomaze]

@kalomaze I'd suggest to start simple, going against top p alone. Less work and straight to the point. If we realize we're gathering enough participants, then we can expand it to multiple models and multiple strategies, for a better overview.

I can help you with any roadblock or questions you have along the way: the results are very much of my interest! 💛

(and I'm crossing my fingers for Min P to be successful!)

I see, that's very doable.

How about:

• Top P 0.98 vs Min P 0.02
• Top P 0.95 vs Min P 0.05
• Top P 0.90 vs Min P 0.1
• Top P 0.80 vs Min P 0.2

At temperature 1.0?

[gante]

@kalomaze sounds good (I'm assuming you're more sensible than me to what a good pairing looks like :) )

I'd perhaps suggest lowering the temperature a bit, to 0.7-0.8 (which is what most LLMs use by default nowadays)

[kalomaze]

I'd perhaps suggest lowering the temperature a bit, to 0.7-0.8 (which is what most LLMs use by default nowadays)

The API docs for OpenAI suggest either lowering temperature or using Top P, but not both, which seems to imply truncation sampling was intended for use with a standard temperature (which makes sense to me); and the default provided is also 1.0 for GPT in the first place.
Temperature 1.0 is also representative of the original logit scores transformed into probabilities, and isn't an arbitrary transformation, so it makes the most sense to me at least, to compare at this value (unless you have other reasons for it).

[gante]

@kalomaze temperature can be seen as a post-hoc calibration of the model logits -- an underconfident model should use a temperature below 1.0 and vice-versa. You can also see it as sharpening (<1.0) or flattening (>1.0) the probability distribution. It does have some overlap with top p, with the difference that top p acts on the probabilities and temperature on log probabilities -- after top p, you can end with the same possible tokens, but the temperature will have an impact on their relative distribution.

The optimal temperature changes across models and tasks, with llama models excelling around ~0.7 for most tasks. For instance, the starcoder model is recommended to be used with temperatures around ~0.3 :) My suggestion for 0.7-0.8 assumed the use of models like llama or mistral