Callback to handle rate limiting based on the number of requests or the number of tokens in the input.

It uses Upstash Ratelimit to track the rate limit which utilizes Upstash Redis to track the state.

Should not be passed to the chain when initializing the chain. This is because the handler has a state which should be fresh every time invoke is called. Instead, initialize and pass a handler every time you invoke.

Hierarchy (view full)

Constructors

Properties

identifier: string
includeOutputTokens: boolean
llmOutputPromptTokenField: string
llmOutputTokenUsageField: string
llmOutputTotalTokenField: string
name: string = "UpstashRatelimit"
raiseError: boolean = true
requestRatelimit?: RegionRatelimit
tokenRatelimit?: RegionRatelimit

Methods

  • Run when the chain starts running.

    This method is called multiple times during a chain execution. To ensure it only runs once, it checks and updates a _checked state.

    Parameters

    • _chain: Serialized

      Serialized chain

    • _inputs: ChainValues

      Chain input values

    Returns Promise<void>

    UpstashRatelimitError - If the request rate limit is reached

  • Run when the LLM ends running.

    If the includeOutputTokens is set to true, the number of tokens in the LLM completion are counted for rate limiting.

    Parameters

    • output: LLMResult

      LLM result output

    • _runId: string
    • Optional_parentRunId: string
    • Optional_tags: string[]

    Returns Promise<void>

    Error - If the LLM response does not include required token usage information

  • Run when the LLM starts running.

    Parameters

    • _llm: Serialized

      Serialized LLM

    • _prompts: string[]

      Prompts passed to the LLM

    • _runId: string
    • Optional_parentRunId: string
    • Optional_extraParams: Record<string, unknown>
    • Optional_tags: string[]
    • Optional_metadata: Record<string, unknown>
    • Optional_name: string

    Returns Promise<void>

    UpstashRatelimitError - If the token rate limit is reached