Flexible Inference
Flexible Inference is a Gemini-specific technique that allows developers to specify a range of acceptable model outputs, enabling the model to choose the most suitable response within those bounds.
When to use
Use when you need the model to have some creative freedom or variability in its output, but within predefined constraints. This is particularly useful for tasks like generating text with varying lengths, styles, or levels of detail, without needing to hardcode multiple prompts.
How to apply
When making an API call to Gemini, specify the inference parameters to include a range for certain attributes.
For example, you can set a temperature range (e.g., 0.5 to 0.9) to allow the model to vary its creativity.
You can also specify a token length range (e.g., 50 to 150 tokens) for the output.
The model will then select a value within your specified range for each parameter, aiming to produce the most appropriate response.
Glossary
- inference parameters
- Settings or configurations passed to a language model during the inference (generation) process that influence its output, such as temperature or maximum token length.
- temperature range
- A specified minimum and maximum value for the 'temperature' parameter, which controls the randomness or creativity of the model's output. A higher temperature leads to more diverse and creative text, while a lower temperature results in more deterministic and focused text.
- token length range
- A specified minimum and maximum number of tokens (words or sub-word units) that the model's generated output should contain.
Sources