Module llmsearch.patches.transformers_monkey_patch

Monkey Patch transformers to add in generation techniques that are currently not supported by the transformers library. Adds tfs, top_a & generation_seed to transformers.GenerationConfig

Implementation Reference of tfs & top_a from oobabooga/text-generation-webui

Functions

def generation_config_init_patch(self, **kwargs)

Patch to add in new generation parameters to transformers.GenerationConfig

def get_logits_warper_patch(self, generation_config)

Patch to add in new generation techniques to transformers.GenerationMixin._get_logits_warper Adds in tfs, top_a and mirostat (not supported currently by llmsearch) to the generation techniques

def hijack_samplers()

Patches generation methods to add in new generation techniques

  • transformers.GenerationMixin._get_logits_warper to add in new generation techniques, Older version of the method is stored in _get_logits_warper_old
  • transformers.GenerationConfig.__init__ to add in new generation parameters, Older version of the method is stored in __init___old

Classes

class MirostatLogitsWarper (mirostat_mode, mirostat_tau, mirostat_eta, filter_value=-inf, min_tokens_to_keep=1, generation_seed=None)

Mirostat sampling - Currently not supported, needs work to work with a batch of sequences.

raises NotImplementedError if it gets called

Ancestors

  • transformers.generation.logits_process.LogitsWarper
class TailFreeLogitsWarper (tfs, filter_value=-inf, min_tokens_to_keep=1)

TFS sampling

Ancestors

  • transformers.generation.logits_process.LogitsWarper
class TopALogitsWarper (top_a, filter_value=-inf, min_tokens_to_keep=1)

Top-A sampling Introduced in RWKV-LM repo.

Ancestors

  • transformers.generation.logits_process.LogitsWarper