tokenizer

package

v0.0.0-...-e803270 Latest Latest Go to latest Published: Dec 19, 2025 License: MIT Imports: 4 Imported by: 0

Details

This section is empty.

This section is empty.

func CalculateSimilarity(s1, s2 string) int

CalculateSimilarity calculates Levenshtein distance between two strings

type Token struct {
	Text     string
	Position int
}

Token represents a processed token with its position

type Tokenizer struct {
	// contains filtered or unexported fields
}

Tokenizer handles text tokenization and normalization

func NewTokenizer() *Tokenizer

NewTokenizer creates a new tokenizer instance

func (t *Tokenizer) Tokenize(text string) []Token

Tokenize splits text into tokens with positions

func (t *Tokenizer) TokenizeToStrings(text string) []string

TokenizeToStrings returns just the token strings without positions