Prefix Aware Scorer

[oglok]

This PR implements:

• prefix store based in a radix tree storing a map of prefixes (key) and prefixEntry (value) that contains the target pod, last time used and model name.
• prefix aware scorer
• addition of new scorer to the scorer manager
• unit tests for new store and scorer

NOTE: there is a runMaintenance function in the store that is not being used. Need to figure out where to call it from without blocking all requests.

[Copilot]

Pull Request Overview

This PR implements a new prefix store based on a radix tree and a prefix-aware scorer to route requests more efficiently. Key changes include:

• Adding a PrefixStore and its configuration to scheduler configuration with environmental overrides.
• Implementing a new PrefixAwareScorer and integrating it into the scheduler.
• Providing unit tests for the new store and scorer components.

Reviewed Changes

Copilot reviewed 7 out of 8 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
pkg/epp/scheduling/types/types.go	Added a Prompt field to the LLMRequest structure.
pkg/epp/scheduling/scheduler.go	Introduced prefix store configuration and integrated the new scorer.
pkg/epp/scheduling/prefix_store.go	Implements the prefix store with entry addition, lookup, eviction, and maintenance.
pkg/epp/scheduling/prefix_store_test.go	Unit tests for basic prefix operations, constraints, TTL expiration, max entries, and maintenance routine.
pkg/epp/scheduling/prefix_aware_scorer.go	Defines the scorer which uses the prefix store to score pods.
pkg/epp/scheduling/prefix_aware_scorer_test.go	Tests behavior of the new scorer with various input scenarios.

Files not reviewed (1)

• go.mod: Language not supported

[Copilot]

Modifying 'entry.LastUsed' under a read lock (RLock) in FindPodForPrefix may lead to a race condition. Consider acquiring the write lock (Lock) when updating mutable state or restructuring the update to ensure thread safety.

Suggested change

	// Update LastUsed time for the matched entry
	entry.LastUsed = time.Now()
	ps.tree.Insert(matchedPrefix, entry)
	// Update LastUsed time for the matched entry
	ps.mu.RUnlock() // Release the read lock before acquiring the write lock
	ps.mu.Lock()
	entry.LastUsed = time.Now()
	ps.tree.Insert(matchedPrefix, entry)
	ps.mu.Unlock()

[rootfs]

this is probably too small :D

[oglok]

oops! testing defaults xD

[rootfs]

minute instead or hour?

[oglok]

I'm not really sure what should be a reasonable TTL for LLM inferencing.

[vMaroon]

The dev branch was rebased on upstream main, which includes the new and different Scorer interface. The Scorer pulled from there defines a Score function that works per-pod and not group of pods. We need to discuss whether to propose changing that but it makes sense to first adapt to it.

I'll open a KVCacheAwareScorer PR soon and we can sync.