Interface TokenTextSplitterParams

Hierarchy

Implemented by

Properties

allowedSpecial chunkOverlap chunkSize disallowedSpecial encodingName keepSeparator lengthFunction?

Properties

allowedSpecial

allowedSpecial: string[] | "all"

chunkOverlap

chunkOverlap: number

chunkSize

chunkSize: number

disallowedSpecial

disallowedSpecial: string[] | "all"

encodingName

encodingName: TiktokenEncoding

keepSeparator

keepSeparator: boolean

Optional lengthFunction

lengthFunction?: ((text) => number) | ((text) => Promise<number>)

Type declaration

    • (text): number

    • Parameters

      • text: string

      Returns number

Type declaration

    • (text): Promise<number>

    • Parameters

      • text: string

      Returns Promise<number>

Settings

Member Visibility

Theme

On This Page

Generated using TypeDoc