Our competitors at Gurobi just published a new online game, called Gurobean. It is supposed to demonstrate how well Gurobi handles uncertainty.
Unfortunately for them, it mostly highlights a pattern. We’ve seen this before:
- See their infamous Avocado pricing example reviewed in code here and as a video review here. Spoiler: Gurobi claims left and right their solution was optimal yet loses the client $1 million per week.
- See their pricing and resource allocation for competing products here and our take on the problem here. Spoiler: Gurobi again claimed optimality while treating forecasts as deterministic, losing over 5.5% margin in retail grocery.
As the saying goes: "Bean there, done that."
The Setup: Gurobean Rules
The coffee shop game is again a problem of pricing and resource allocation. We get to set production targets for hot and cold coffee. We can set the markup for both (weirdly in absolutes rather than relatives). And we can pay our barista more to make her serve coffee faster.
We operate in a somewhat uncertain environment. The game does not tell us how many clients to expect (although that can be found out by test-running the environment sufficiently often). At best, we are being told or can infer averages, such as the relative interest in hot vs. cold coffee. However, the game does not tell us how these values vary over time or the functional connection.
The game proceeds in 8 rounds. The first two are very boring, because there is no cost associated with brewing coffee, so we simply max out our resources.
Wait! Did I say boring? Because already in the second round, Gurobi returns a provably suboptimal solution while, of course, boasting about optimality as usual:
Obviously, when there are no costs involved and there is uncertainty in demand, then making more is always better. So producing 19 hot and 8 cold coffees is strictly worse than making 19 and 9, respectively, if resources permit. So no, Gurobi, your solution is not optimal, and your sub-optimality has zilch to do with randomness and everything to do with your method of handling uncertainty.
Going Toe-to-Toe With A Stochastic Solver
As soon as we get to set prices in Round 5, things become more interesting. While there is a little pop-up on arrival rates of clients, we lack enough information to model the demand patterns from the simulation. For example, contrary to what the pop-up suggests, there must be additional limits, because no matter how much we lower prices, there are no more than 22 clients per hour on average. Gurobi has a clear home-court advantage here since we lack the exact simulation model.
Nevertheless, with the help of a tool like InsideOpt Seeker(TM), we can devise strategies for all 8 rounds. (We rushed to get these results out as the game came out yesterday, and we wanted to give a timely review. The solutions below are based on quick models and we are sure the readers can improve them further, especially in Round 5.)
- Make 25 hot coffees.
- Make 19 hot and 9 cold coffees.
- Make 17 hot coffees.
- Make 17 hot and 5 cold coffees.
- Make 16 hot and 5 cold coffees. Set markup to $2.90.
- Make 17 hot and 5 cold coffees. Set markup to $2.80.
- Make 37 hot and 10 cold coffees. Set markup to $2.80.
- Make 40 hot and 12 cold coffees. Set markup to $2.60. Pay the barista $36 per hour.
Gurobi chooses the same solutions for Rounds 1 and 3, so comparing those is not interesting. The other six rounds are informative, though. For each game we play against Gurobi, we therefore get 6 head-to-head comparisons.
We repeated the game with the Seeker-found solutions given above ten times. Even though we had to make guesses regarding the uncertainty while Gurobi did not:
- Seeker won all ten games. Seven of them are 6:0, the other three 5:1. If we compare the results on the full version of the problem, in Round 8 Seeker(TM) delivers 2.3% more net revenue on average that go directly to the bottom line. Assuming other costs for rent, cleaning, etc, the gains in profit are easily over 10%. If our shop opens 300 days a year, that corresponds to over $14,000 more in the pocket of our store owner.
- Out of the 60 head-to-head comparisons, Seeker(TM) won 57, Gurobi won 3. In a Binomial test, the odds that Gurobi is as good or better than Seeker(TM) are therefore 1 in 32 trillion.
Screenshots of the results are given below.
It takes a lot of nerve to claim "optimality" three times after every round and twice after every game when you are losing this badly.
This is obviously a simple game, nowhere near the complexity of running decisions under uncertainty in a real business. And yet, even in this contrived setting, Gurobi manages to deliver suboptimally with overwhelming statistical evidence. On a simple two-variable problem, even provably so.
In German, we would call that "kalter Kaffee."
When it comes to decision support under real-world uncertainty, do you want your business run by a solver blind to true stochastic variation using a rigid, deterministic framework - or by the absolute best in class?
Don't settle. Use InsideOpt Seeker(TM) next time.
Results of all Ten Games
