Allow services to handle many contests at the same time

[lw]

At the moment many services require a contest ID as a command line argument. I think that their use of this ID is very limited (except in some cases like CWS). I think that they could manage multiple contest simultaneously. For example AWS already does that and, in fact, it doesn't require that command line argument. That would mean, for example, that Workers could accept job requests for different contests, that ES should manage submissions of many contests (all in the same queue?), and the same for SS.

I therefore suggest to check exactly how services use the contest ID and check if it's possible to remove it, making them able to handle data from all contests at the same time.

What do you think?

PS: the reason for this is that, for example, for training purposes it seems convenient to run many contest simultaneously on the same machine, and it seems excessive to require dedicated services for each of them (actually, I'm not even sure if, for example, many ESs on the same machine would work correctly...)

[stefano-maggiolo]

many ESs on the same machine would work correctly

Why wouldn't they?

Anyway, I see the point too. I thought we could have some problems doing it, but thinking a bit about it I can't see why it shouldn't be not-too-hard (TM) to make the services managing multiple contests. Maybe the most troublesome would be SS, which needs to keep things separated (whereas ES can share the same queue for everything).

+1 for me.

[giomasce]

Ciao.

Il 27/11/2012 19:52, Stefano Maggiolo ha scritto:

many ESs on the same machine would work correctly

Why wouldn't they?

Probably because they wouldn't know how to coordinate the distribution
of jobs on the same set of workers.

Gio.

Giovanni Mascellani mascellani@poisson.phc.unipi.it
Pisa, Italy

Web: http://poisson.phc.unipi.it/~mascellani
Jabber: g.mascellani@jabber.org / giovanni@elabor.homelinux.org

[stefano-maggiolo]

Probably because they wouldn't know how to coordinate the distribution of jobs on the same set of workers.

Ok, I meant with the contest completely detached (i.e., not even shared workers). After a private talk with Luca, it turned out that the two issues met by the tester with the setup I was thinking of (no shared services) were

1. the tester did not know it is easy to run services with different cms.conf by setting the shell variable
2. RS has some trouble figuring out what to kill and what not to kill :) unlike from 1, it seems this is a real issue, but it should go away if we allow shareable services.

[giomasce]

Personally, I think that it would make sense anyway to allow every CMS service to manage more than one contest at the same time. Particularly, I would like all services not to require any contest ID to start, but just serve every contest they have available (though we anyway need tools to enable or disable selectively each contest; it may be a flag on the database). This, in my opinion, is cleaner and easier to manage.

[bblackham]

👍 from me also. This would be very useful for our training camps where we have two streams, or on a demo server serving several contests.

I've done an attempt at removing the contest ID from most of the code base, here: https://github.com/bblackham/cms/commits/no-contest (changesets
bblackham@4f6aec4 and bblackham@0fb71e9)

The idea is that only some contests are "active", as per a flag on each contest (in the database). The active contests are those which get polled by the CMS service loops. With these changes, all services can be started without a contest id, except for ContestWebServer (and therefore the flag has not yet been removed from ResourceService).

Unfortunately, with these patches, CMS will still honour requests for non-active contests. Preventing that seems to be quite invasive. It could be useful to ensure that a non-active contest is effectively read-only and/or hidden, but I don't see this as a priority.

Please flame me on that patch set, or provide other comments :)

[lw]

My idea to solve this issue is to make each service able to serve a certain subset of contests, ranging from none to all of them. This subset could be different for each service and is specified as a command line argument when the service is started. I think it'd also be great to be able to change this set dynamically over time, without the need to restart the service, for example through an interface in AWS.

I'm not sure of the pros and cons of my approach vs. Bernard's. Yet, I think that my idea introduces a lot of difficulties in inter-service communication and queue management (the various ESs need to know which Workers serve the contests they're serving, they need to coordinate on sending jobs to them and they need to have a queue that, as soon as a Worker is ready, is able to get the oldest job suited for that Worker, even if there are older jobs in the queue belonging to contests that that Worker doesn't serve... in short, it's a mess). A centralized publisher/subscriber messaging hub (like an AMQP broker...) may help with that.

An observation that applies to both proposals, though, is that it would be nice to set priorities on contests to have the jobs related to them be evaluated faster. That would help, for example, if one's having an onsite contest with an identical online one for the public to "play along" and one wants to use the same servers and services for both but wants the submissions of the onsite contest to get prioritized in the evaluation (i.e. wants the ones of the online contest to be evaluated only if there are no pending jobs for the onsite contest).

[stefano-maggiolo]

1. Why one would want to have ES but not the workers to manage a contest? (Provided one can set priorities on the contests.) In my opinion we should care of selecting potentially different sets of contest in three places:

1a. contestants: which contest users should be able to login;
1b. backend: which contest we process the submissions in the backend;
1c. internet: which contest goes into which ranking.

1. One of our key functionality is that services die gracefully, if not needed we should avoid to complicate things to manage dynamical changes of parameters when a simple restart is fine.

[lw]

Well, my idea was again the onsite/online contest. An admin could, for example, have four Workers: two dedicated only to the onsite contest, one only to the online contest and one shared. Then, one could have two ESs serving this same set of Workers.

I'm not sure if this scenario makes any sense. One could just have one ES instead of two and it should work equally fine, and probably with contest priorities one doesn't even need to assign Workers exclusively to one contest...

[bblackham]

Luca's idea can be useful if you have machines of different specifications as workers. You might want to some contests to use only faster workers, and other contests only slow ones. But it's a lot of effort for an unlikely scenario. I think until somebody actually needs it, it's not worth spending time on, IMHO.

I do like the idea of prioritising contests (with some kind of fairness guarantees to ensure that one contest can't starve the others).

[lw]

It can also be useful for ScoringServices (if you want a public RWS only for the onsite contest then you cannot have a single SS for both contests) and for ContestWebServers (similar reasoning).

One idea I had for the "prioritization" is to assign a "rudeness" parameter to each contest, a non-negative integer. A value of zero means "don't evaluate unless there are no other jobs in the queue belonging to a contest with non-zero rudeness" (i.e. "let me starve"). In case the queue is empty and there are many contests with a zero rudeness choose one randomly. If there are many contests with non-zero rudeness that have jobs in the queue choose one randomly based on their rudeness (i.e. the probability of a contest being chosen is its rudeness divided by the sum of the rudeness of all contests in the queue). What do you think?

[bblackham]

You can still have a single SS for all contests, but have RWS register only for updates on a subset of contests (with nothing to prevent a rogue RWS getting all contest data in this model, but I don't think it's an issue).

I like the rudeness idea.

[lw]

You can still have a single SS for all contests, but have RWS register only for updates on a subset of contests

Are you saying that it's already possible or that we could implement it? In the former case: how?

[bblackham]

Oh, I meant the latter, speaking hypothetically.

My motivation is to have as few contest-specific services to configure as possible. Just (CWS, RWS) being configured per-contest would be ideal.

[lw]

Done for SS in 948f167. No contest prioritization/rudeness was introduced there because scoring should be so fast that jobs are unlikely to starve in the queue.