Preparing and running the internal competition...

It has been a very exciting, instructive, and exhausting experience:

  • the organizing team at Univ. Paris Diderot (Pietro Abate, Ralf Treinen, Stefano Zacchiroli and myself) selected two realistic optimization criteria (Paranoid, for the conservative sysadm, and Trendy, for the reckless user always excited about new releases; more on this in a forthcoming post), published a detailed set of rules, selected a couple of hundreds significant problems to exercise the solvers, put in place the infrastructure to run the submitted solvers and wrote code to score and rank the solvers according to the competition rules, and all of this, over the Christmas holidays! (You can get all the code from the forge here, using mancoosi/mancoosi as user/pwd : access is open, but the admin likes to avoid bots);
  • the competition entrants read the rules, fiddled with CUDF syntax and semantics, reported bugs in some of the optimization criteria, and coded their own version of an efficient strategy to find solutions to the problems according to one or both of the two optimization criteria proposed; there were entrants from most of the project partners: Caixa Magica, Inesc ID, UNSA; but there was also a very welcome external participant, the Eclipse P2 solver adapted by Daniel Le Berre, and Pascal Rapicault, which have been working closely with us over the last months;
  • finally, the Mancoosi project coordinator brought to the meeting two nice bottles of the finest french Champagne, one in a classic box (for the Paranoid, conservative competition track), an the other in a more fashionable style (for the Trendy competition track).

During the meeting in Nice, the servers were still running the solvers through all the problems, and suspense was at its maximum when we decided to stop the execution before full completion to be able to attribute the bottles before the end of the meeting: after some final analysis of the data, we found two winners!

The P2 solver won in the Paranoid track, so Daniel Le Berre was awarded the conservative Champagne bottle.

The INESC solver won in the Trendy track, so Josep Algerich got the fashionable Champagne bottle.

... was just the beginning ...

But that was not the end of the story: in the weeks after that exciting moment, we put a significant amount of energy in double checking the results, and analysing the solutions. This led us to uncover a series of glitches almost everywhere: in the infrastructure we used to run the competition in Nice, in the code implementing the scoring and ranking of the solvers, in the instructions that we had given to the competition entrants.

After some serious extra work, we now believe we have fixed every issue found up to now, and we are finally ready for the International competition.

As a result of the fixes, after rerunning all the originally submitted solvers, it turns out that Eclipse P2 should have actually been declared winner in both tracks, so, congratulations to the P2 team, that, I have been told, dedicated significant energy to getting their solver CUDF-ready, while not being an official Mancoosi partner.

You can see the actual final scores online here (winners are the ones with lowest numbers, and they are on a green background).

... and now MiSC goes live!

Running this internal round also allowed the participants to uncover glitches in practically all the solvers submitted to the competition, so we decided to make available a live page with the results updated every time an improved version of one or the other solver is submitted, and with a detailed presentation of the results found by each solver on each individual problem. Some of the entrants of the competition also made available the full code of their solver.

We can now confirm that organizing a solver competition for a new kind of problems, using a new data format and with sophisticated optimization criteria is a complex endeavor, and the lesson learned is that it was a wise decision, and not a luxury, to invest all this energy in organizing a preliminary, full scale, realistic run to fine tune the tools and spot the hidden issues.

We believe that the huge effort made in setting up this internal realistic competition, and all the material we are making publicly available, will be of great help for the teams preparing to compete in the international competition, MISC 2010, over the next months.