Add scorers support in scheduler #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

mayabar merged 10 commits into neuralmagic:dev from mayabar:main

Apr 17, 2025

Collaborator

mayabar commented Apr 14, 2025

No description provided.

mayabar added 2 commits

April 10, 2025 14:50


          Add initial support for scorers, used as part of decision which pod w…

0c95c2a

…ill be the target for a request. Session affinity scorer added


          Fixes in scores infrastructure & session aware scorer

bde57da

vMaroon reviewed

View reviewed changes

pkg/epp/scheduling/scorer.go Outdated Show resolved Hide resolved

elevran requested changes

View reviewed changes

pkg/epp/datastore/datastore.go Outdated Show resolved Hide resolved

pkg/epp/datastore/datastore.go Show resolved Hide resolved

pkg/epp/handlers/server.go Outdated Show resolved Hide resolved

pkg/epp/scheduling/scheduler.go Outdated Show resolved Hide resolved

pkg/epp/scheduling/scheduler.go Outdated Show resolved Hide resolved

pkg/epp/scheduling/scheduler.go Outdated Show resolved Hide resolved

pkg/epp/scheduling/scorer.go Outdated Show resolved Hide resolved

pkg/epp/scheduling/scorer.go Outdated Show resolved Hide resolved

pkg/epp/scheduling/scorer.go Show resolved Hide resolved

pkg/epp/scheduling/types/types.go Outdated Show resolved Hide resolved

vMaroon reviewed

View reviewed changes

pkg/epp/scheduling/scheduler.go Outdated Show resolved Hide resolved

vMaroon reviewed

View reviewed changes

pkg/epp/scheduling/scorer.go Outdated Show resolved Hide resolved

vMaroon reviewed

View reviewed changes

pkg/epp/scheduling/types/types.go Outdated Show resolved Hide resolved

vMaroon reviewed

View reviewed changes

pkg/epp/scheduling/scorer.go Outdated Show resolved Hide resolved


          - Add cleanup for session->pod map

8f9785e

- Rename SessionId to SessionID
- Remove datastore from scoreTargets, add datastore to SessionAffinityScorer
- Rename ScoredPod to PodScore

vMaroon reviewed

View reviewed changes

pkg/epp/scheduling/scorer.go Outdated Show resolved Hide resolved

mayabar added 7 commits

April 14, 2025 22:09


          Export score and pod for external implementations

52af66f


          Rename session id header

90e23bc


          Separate code of Scorer interface and scorer implementations + add sc…

c946acc

…orerManager


          Remove vllmRequest from scoreTargets API since it exists in the context

137ca09


          Support negative score weights

c42f72a


          Fix fakeDataStore to be compatible with DataStore intereface

a05a573


          - Check for nils in list of available pods in main scoring function o…

aca8e07

…f ScoreMng

- If some specific scorer failed to score pods - just log the problem, skip it and continue to the next scorer

Collaborator

elevran commented Apr 16, 2025

@mayabar all done? If not, please mention when all PR comments have been addressed so I may review/approve

elevran approved these changes

View reviewed changes

Collaborator Author

mayabar commented Apr 16, 2025

All comments were addressed, all updated pushed and code is ready for merge

mayabar merged commit d8303a0 into neuralmagic:dev

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet