Repository navigation
Add job management APIs #302
Description
We should add a job management to the core API for
- tracking status of jobs
- cleaning up jobs
For retrieval, we could possibly retrieve jobs based on the existing GetFeatureSets and GetStores filters:
rpc GetJobs(GetJobsRequest) returns (GetJobsResponse);
// Retrieves matching jobs given feature set and/or store filters. If none are provided,
// all active jobs will be returned.
message GetJobsRequest {
// Filter by feature set name and version
GetFeatureSetsRequest.Filter feature_set = 1;
// Filter by store name
GetStoresRequest.Filter store = 2;
}
message GetJobsResponse {
enum JobStatus {
// Job state unknown
UNKNOWN = 0;
// New feature set or feature set version created
PENDING = 1;
// Error occurred while trying to apply changes
RUNNING = 2;
COMPLETED = 3;
/** When user sent abort command, but it's still running */
ABORTING = 4;
/** User initiated abort job */
ABORTED = 5;
/**
* Runner’s reported that the import job failed to run or there is a failure during job
* submission.
*/
ERROR = 6;
/** job has been suspended and waiting for cleanup */
SUSPENDING = 7;
/** job has been suspended */
SUSPENDED = 8;
}
message Job {
string name = 1;
JobStatus status = 2;
repeated feast.core.FeatureSet feature_sets = 3;
feast.core.Store store = 4;
feast.core.Source source = 5;
}
repeated Job jobs = 1;
}
Jobs should be aborted by name.
rpc AbortJob(AbortJobRequest) returns (AbortJobResponse);
message AbortJobRequest {
string job_name = 1;
}
message AbortJobResponse {
enum Status {
INVALID = 0;
SUCCESS = 1;
ERROR = 2;
}
// Feature set response has been enriched with version and source information
Status status = 1;
}
Activity
woop commentedon Nov 13, 2019
Thanks for this @zhilingc!
It seems like there are three types of "gets":
Can I suggest that we start with a Job RPC and a ListJobs RPC instead of a single RPC that does both? It would mirror this structure from the Kubernetes API
I know (1) and (2) have use cases, but does (3) have a use case? If it does, then we can add it to ListJobsRequest. I also think that we should consider updating feature retrieval to this structure since it more closely follows usage of clients.
zhilingc commentedon Nov 13, 2019
I would like to be consistent with our existing APIs as much as possible, which happen to overload a single
getRPC.I think looking up jobs by store/featureset is probably more useful to users than looking up jobs by name... since
(1) they would want to filter jobs to only find those relevant to their interests
(2) job names are not usually exposed to users
[-][0.3] Add job management APIs[/-][+]Add job management APIs[/+]woop commentedon Nov 13, 2019
I am happy staying consistent for the time being. I don't want to derail this request with a debate on naming, but I think ListJobs expresses intent more clearly than GetJobs. In terms of functionality there, I think we are proposing the same thing.
I do think we need a GetJob method by Id, but I guess that isn't as pressing for now so it can be punted.
woop commentedon Nov 13, 2019
Also, can we use Job Id instead of name?
zhilingc commentedon Nov 14, 2019
I agree. Like i mentioned, I'm just gunning for internal consistency as much as possible.
we could add it as one of the possible fields of the GetJobsRequest, but it doesn't really spark joy haha.
woop commentedon Nov 14, 2019
I mean I would like to have an RPC that only returns a single item, or fails otherwise. The example above is not clean.
The equivalent of
feast.example.com/v1/jobsvsfeast.example.com/v1/jobs/1234fix: Operator Cluster Role grants excess permissions (feast-dev#302)
fix: Operator Cluster Role grants excess permissions (feast-dev#302)