Common tools, ratios and figures to illustrate a tournament outcome and provide a base for its interpretation.
Number of games
The total number of games played by an engine in a tournament.
Score
The score is a representation of the tournament-outcome from the viewpoint of a certain engine.
Win & Draw Ratio
These two ratios depend on the strength difference between the competitors, the average strength level, the color and the drawishness of the opening book-line. Due to the second reason given, these ratios are very much influenced by the timecontrol, what is also confirmed by the published statistics of the testing orgnisations CCRL and CEGT, showing an increase of the draw rate at longer time controls. This correlation was also shown by Kirill Kryukov, who was analyzing statistics of his test-games [2] . The program playing white seems to be more supported by the additional level of strength. So, although one would expect with increasing draw rates the win ratio to approach 50%, in fact it is remaining about equal.
The likelihood of superiority (LOS) denotes how likely it would be for two players of the same strength to reach a certain result - in other fields called a p-value, a measure of statistical significance of a departure from the null hypothesis[6]. Doing this analysis after the tournament one has to differentiate between the case where one knows that a certain engine is either stronger or equally strong (directional or one-tailed test) or the case where one has no information of whether the other engine is stronger or weaker (non-directional or two-tailed test). The latter due to the reduced information results in larger confidence intervals.
The probability of the null hypothesis being true can be calculated given the tournament outcome. In other words, how likely would it be for two players of the same strength to reach a certain result. The LOS would then be the inverse, 1 - the resulting probability.
For this type of analysis the trinomial distribution, a generalization of the binomial distribution, is needed. Whilest the binomial distribution can only calculate the probability to reach a certain outcome with two possible events, the trinominal distribution can account for all three possible events (win, draw, loss).
The following functions gives the probability of a certain game outcome assuming both players were of equal strength:
This calculation becomes very inefficient for larger number of games. In this case the standard normal distribution can give a good approximation:
where N(1 - draw_ratio) is the sum of wins and losses:
To calculate the LOS one needs the cumulative distribution function of the given normal distribution. However, as pointed out by Rémi Coulom, calculation can be done cleverly, and the normal approximation is not really required [7] . As further emphasized by Kai Laskos[8] and Rémi Coulom [9][10] , draws do not count in LOS calculation and don't make a difference whether the game results were obtained when playing Black or White. It is a good approximation when the two players played the same number of games with each color:
Sample Program
A tiny C++11 program to compute Elo difference and LOS from W/L/D counts was given by Álvaro Begué[14] :
#include <cstdio>#include <cstdlib>#include <cmath>int main(int argc, char**argv){if(argc !=4){
std::printf("Wrong number of arguments.\n\nUsage:%s <wins> <losses> <draws>\n", argv[0]);return1;}int wins = std::atoi(argv[1]);int losses = std::atoi(argv[2]);int draws = std::atoi(argv[3]);double games = wins + losses + draws;
std::printf("Number of games: %g\n", games);double winning_fraction =(wins +0.5*draws)/ games;
std::printf("Winning fraction: %g\n", winning_fraction);double elo_difference =-std::log(1.0/winning_fraction-1.0)*400.0/std::log(10.0);
std::printf("Elo difference: %+g\n", elo_difference);double los =.5+.5* std::erf((wins-losses)/std::sqrt(2.0*(wins+losses)));
std::printf("LOS: %g\n", los);}
Statistical Analysis
The trinomial versus the 5-nomial model
As indicated above a match between two engines is usually modeled as a sequence of independent trials taken from a trinomial distribution with probabilities (win_ratio,draw_ratio,loss_ratio). This model is appropriate for a match with randomly selected opening positions and randomly assigned colors (to maintain fairness). However one may show that under reasonable elo models the trinomial model is not correct in case games are played in pairs with reversed colors (as is commonly the case) and unbalanced opening positions are used.
This was also empirically observed by Kai Laskos[15] . He noted that the statistical predictions of the trinomial model do not match reality very well in the case of paired games. In particular he observed that for some data sets the variance of the match score as predicted by the trinomial model greatly exceeds the variance as calculated by the jackknife estimator. The jackknife estimator is a non-parametric estimator, so it does not depend on any particular statistical model. It appears the mismatch may even occur for balanced opening positions, an effect which can only be explained by the existence of correlations between paired games - something not considered by any elo model.
Over estimating the variance of the match score implies that derived quantities such as the number of games required to establish the superiority of one engine over another with a given level of significance are also over estimated. To obtain agreement between statistical predictions and actual measurements one may adopt the more general 5-nomial model. In the 5-nomial model the outcome of paired games is assumed to follow a 5-nomial distribution with probabilities
These unknown probabilities may be estimated from the outcome frequencies of the paired games and then subsequently be used to compute an estimate for the variance of the match score. Summarizing: in the case of paired games the 5-nomial model handles the following effects correctly which the trinomial model does not:
Unbalanced openings
Correlations between paired games
For further discussion on the potential use of unbalanced opening positions in engine testing see the posting by Kai Laskos[16] .
SPRT
The sequential probability ratio test (SPRT) is a specific sequential hypothesis test - a statistical analysis where the sample size is not fixed in advance - developed by Abraham Wald[17] . While originally developed for use in quality control studies in the realm of manufacturing, SPRT has been formulated for use in the computerized testing of human examinees as a termination criterion [18]. As mentioned by Arthur Guez in this 2015 Ph.D. thesis Sample-based Search Methods for Bayes-Adaptive Planning[19], Alan Turing assisted by Jack Good used a similar sequential testing technique to help decipher enigma codes at Bletchley Park[20]. SPRT is applied in Stockfish testing to terminate self-testing series early if the result is likely outside a given elo-window [21] . In August 2016, Michel Van den Bergh posted following Python code in CCC to implement the SPRT a la Cutechess-cli or Fishtest: [22][23]
from__future__import division
importmathdef LL(x):
return1/(1+10**(-x/400))def LLR(W,D,L,elo0,elo1):
"""
This function computes the log likelihood ratio of H0:elo_diff=elo0 versus
H1:elo_diff=elo1 under the logistic elo model
expected_score=1/(1+10**(-elo_diff/400)).
W/D/L are respectively the Win/Draw/Loss count. It is assumed that the outcomes of
the games follow a trinomial distribution with probabilities (w,d,l). Technically
this is not quite an SPRT but a so-called GSPRT as the full set of parameters (w,d,l)
cannot be derived from elo_diff, only w+(1/2)d. For a description and properties of
the GSPRT (which are very similar to those of the SPRT) see
http://stat.columbia.edu/~jcliu/paper/GSPRT_SQA3.pdf
This function uses the convenient approximation for log likelihood
ratios derived here:
http://hardy.uhasselt.be/Toga/GSPRT_approximation.pdf
The previous link also discusses how to adapt the code to the 5-nomial model
discussed above.
"""# avoid division by zeroif W==0or D==0or L==0:
return0.0
N=W+D+L
w,d,l=W/N,D/N,L/N
s=w+d/2
m2=w+d/4
var=m2-s**2
var_s=var/N
s0=LL(elo0)
s1=LL(elo1)return(s1-s0)*(2*s-s0-s1)/var_s/2.0def SPRT(W,D,L,elo0,elo1,alpha,beta):
"""
This function sequentially tests the hypothesis H0:elo_diff=elo0 versus
the hypothesis H1:elo_diff=elo1 for elo0<elo1. It should be called after
each game until it returns either 'H0' or 'H1' in which case the test stops
and the returned hypothesis is accepted.
alpha is the probability that H1 is accepted while H0 is true
(a false positive) and beta is the probability that H0 is accepted
while H1 is true (a false negative). W/D/L are the current win/draw/loss
counts, as before.
"""
LLR_=LLR(W,D,L,elo0,elo1)
LA=math.log(beta/(1-alpha))
LB=math.log((1-beta)/alpha)if LLR_>LB:
return'H1'elif LLR_<LA:
return'H0'else:
return''
^Arthur Guez (2015). Sample-based Search Methods for Bayes-Adaptive Planning. Ph.D. thesis, Gatsby Computational Neuroscience Unit, University College London, pdf
the statistics of chess tournaments and matches, that is a collection of chess games and the presentation, analysis, and interpretation of game related data, most common game results to determine the relative playing strength of chess playing entities, here with focus on chess engines. To apply match statistics, beside considering statistical population, it is conventional to hypothesize a statistical model describing a set of probability distributions.
Table of Contents
Ratios / Operating Figures
Common tools, ratios and figures to illustrate a tournament outcome and provide a base for its interpretation.Number of games
The total number of games played by an engine in a tournament.Score
The score is a representation of the tournament-outcome from the viewpoint of a certain engine.Win & Draw Ratio
These two ratios depend on the strength difference between the competitors, the average strength level, the color and the drawishness of the opening book-line. Due to the second reason given, these ratios are very much influenced by the timecontrol, what is also confirmed by the published statistics of the testing orgnisations CCRL and CEGT, showing an increase of the draw rate at longer time controls. This correlation was also shown by Kirill Kryukov, who was analyzing statistics of his test-games [2] . The program playing white seems to be more supported by the additional level of strength. So, although one would expect with increasing draw rates the win ratio to approach 50%, in fact it is remaining about equal.
Doubling Time Control
As posted in October 2016 [3] , Andreas Strangmüller conducted an experiment with Komodo 9.3, time control doubling matches under Cutechess-cli, playing 3000 games with 1500 opening positions each, without pondering, learning, and tablebases, Intel i5-750 @ 3.5 GHz, 1 Core, 128 MB Hash [4] , see also Kai Laskos' 2013 results with Houdini 3 [5] and Diminishing Returns:
vs 1
10+0.1
20+0.2
40+0.4
80+0.8
160+1.6
320+3.2
640+6.4
1280+12.8
Elo-Rating & Win-Probability
see Pawn Advantage, Win Percentage, and EloGeneralization of the Elo-Formula:
win_probability of player i in a tournament with n players
Likelihood of Superiority
See LOS TableThe likelihood of superiority (LOS) denotes how likely it would be for two players of the same strength to reach a certain result - in other fields called a p-value, a measure of statistical significance of a departure from the null hypothesis [6]. Doing this analysis after the tournament one has to differentiate between the case where one knows that a certain engine is either stronger or equally strong (directional or one-tailed test) or the case where one has no information of whether the other engine is stronger or weaker (non-directional or two-tailed test). The latter due to the reduced information results in larger confidence intervals.
Two-tailed Test
Null- and alternative hypothesis:
The probability of the null hypothesis being true can be calculated given the tournament outcome. In other words, how likely would it be for two players of the same strength to reach a certain result. The LOS would then be the inverse, 1 - the resulting probability.
For this type of analysis the trinomial distribution, a generalization of the binomial distribution, is needed. Whilest the binomial distribution can only calculate the probability to reach a certain outcome with two possible events, the trinominal distribution can account for all three possible events (win, draw, loss).
The following functions gives the probability of a certain game outcome assuming both players were of equal strength:
This calculation becomes very inefficient for larger number of games. In this case the standard normal distribution can give a good approximation:
where N(1 - draw_ratio) is the sum of wins and losses:
To calculate the LOS one needs the cumulative distribution function of the given normal distribution. However, as pointed out by Rémi Coulom, calculation can be done cleverly, and the normal approximation is not really required [7] . As further emphasized by Kai Laskos [8] and Rémi Coulom [9] [10] , draws do not count in LOS calculation and don't make a difference whether the game results were obtained when playing Black or White. It is a good approximation when the two players played the same number of games with each color:
[11] [12] [13]
One-tailed Test
Null- and alternative hypothesis:
Sample Program
A tiny C++11 program to compute Elo difference and LOS from W/L/D counts was given by Álvaro Begué [14] :
Statistical Analysis
The trinomial versus the 5-nomial modelAs indicated above a match between two engines is usually modeled as a sequence of independent trials taken from a trinomial distribution with probabilities (win_ratio,draw_ratio,loss_ratio). This model is appropriate for a match with randomly selected opening positions and randomly assigned colors (to maintain fairness). However one may show that under reasonable elo models the trinomial model is not correct in case games are played in pairs with reversed colors (as is commonly the case) and unbalanced opening positions are used.
This was also empirically observed by Kai Laskos [15] . He noted that the statistical predictions of the trinomial model do not match reality very well in the case of paired games. In particular he observed that for some data sets the variance of the match score as predicted by the trinomial model greatly exceeds the variance as calculated by the jackknife estimator. The jackknife estimator is a non-parametric estimator, so it does not depend on any particular statistical model. It appears the mismatch may even occur for balanced opening positions, an effect which can only be explained by the existence of correlations between paired games - something not considered by any elo model.
Over estimating the variance of the match score implies that derived quantities such as the number of games required to establish the superiority of one engine over another with a given level of significance are also over estimated. To obtain agreement between statistical predictions and actual measurements one may adopt the more general 5-nomial model. In the 5-nomial model the outcome of paired games is assumed to follow a 5-nomial distribution with probabilities
These unknown probabilities may be estimated from the outcome frequencies of the paired games and then subsequently be used to compute an estimate for the variance of the match score. Summarizing: in the case of paired games the 5-nomial model handles the following effects correctly which the trinomial model does not:
For further discussion on the potential use of unbalanced opening positions in engine testing see the posting by Kai Laskos [16] .
SPRT
The sequential probability ratio test (SPRT) is a specific sequential hypothesis test - a statistical analysis where the sample size is not fixed in advance - developed by Abraham Wald [17] . While originally developed for use in quality control studies in the realm of manufacturing, SPRT has been formulated for use in the computerized testing of human examinees as a termination criterion [18]. As mentioned by Arthur Guez in this 2015 Ph.D. thesis Sample-based Search Methods for Bayes-Adaptive Planning [19], Alan Turing assisted by Jack Good used a similar sequential testing technique to help decipher enigma codes at Bletchley Park [20]. SPRT is applied in Stockfish testing to terminate self-testing series early if the result is likely outside a given elo-window [21] . In August 2016, Michel Van den Bergh posted following Python code in CCC to implement the SPRT a la Cutechess-cli or Fishtest: [22] [23]Tournament Manager
See also
Publications
1920 ...
1960 ...
1980 ...
1990 ...
2000 ...
2005 ...
2010 ...
2015 ...
Forum & Blog Postings
1996 ...
2000 ...
2005 ...
2010 ...
- Engine Testing - Statistics by Edmund Moshammer, CCC, January 14, 2010
- Chess Statistics by Edmund Moshammer, CCC, June 17, 2010
- Do You really need 1000s of games for testing? by Jouni Uski, CCC, November 04, 2010
- GUI idea: Testing until certainty by Albert Silver, CCC, December 07, 2010
- SPRT and Engine testing by Adam Hair, CCC, December 13, 2010 » SPRT
2011Re: Engine Testing - Statistics by John Major, CCC, January 14, 2010
- Ply vs ELO by Andriy Dzyben, CCC, June 28, 2011
- One billion random games by Steven Edwards, CCC, August 27, 2011
- Increase in Elo ..Question For The Experts by Steve B, CCC, December 05, 2011
2012- Advantage for White; Bayeselo (to Rémi Coulom) by Edmund Moshammer, CCC, March 03, 2012
- Human Elo ratings: averages and standard deviations by Jesús Muñoz, CCC, March 18, 2012 [39]
- Elo uncertainties calculator by Jesús Muñoz, CCC, March 24, 2012
- Elo versus speed by Peter Österlund, CCC, April 02, 2012
- Rybka odds matches and the strength of engines by Kai Laskos, CCC, June 09, 2012 » Rybka
- A new way to compare chess programs by Larry Kaufman, CCC, June 21, 2012 » Komodo
- EloStat, Bayeselo and Ordo by Kai Laskos, CCC, June 24, 2012 » EloStat, Bayeselo, Ordo
- about error margins? by Fermin Serrano, CCC, August 01, 2012
- normal vs logistic curve for elo model by Daniel Shawul, CCC, August 02, 2012
- Derivation of bayeselo formula by Rémi Coulom, CCC, August 07, 2012 [40]
- Yet Another Testing Question by Brian Richardson, CCC, September 15, 2012
- margin of error by Larry Kaufman, CCC, September 16, 2012
- Average number of plies in {1-0, ½-½, 0-1} by Jesús Muñoz, CCC, September 21, 2012
- Another testing question by Larry Kaufman, CCC, September 23, 2012
- LOS calculation: Does the same result is always the same? by Marco Costalba, CCC, October 01, 2012
- LOS (again) by Ed Schröder, CCC, October 30, 2012
- Elo points gain from doubling time by Kai Laskos, CCC, December 10, 2012
- A word for casual testers by Don Dailey, CCC, December 25, 2012
2013- A poor man's testing environment by Ed Schröder, CCC, January 04, 2013 [41] » Engine Testing
- Noise in ELO estimators: a quantitative approach by Marco Costalba, CCC, January 06, 2013
- Updated Dendrogram by Kai Laskos, CCC, February 02, 2013
- Experiment: influence of colours at fixed depth by Jesús Muñoz, CCC, March 10, 2013
- LOS by BB+, OpenChess Forum, March 31, 2013
- Fishtest Distributed Testing Framework by Marco Costalba, CCC, May 01, 2013
- The influence of the length of openings by Kai Laskos, CCC, July 14, 2013
- Scaling at 2x nodes (or doubling time control) by Kai Laskos, CCC, July 23, 2013 » Doubling TC, Diminishing Returns, Playing Strength, Houdini
- Type I error in LOS based early stopping rule by Kai Laskos, CCC, August 06, 2013 [42]
- How much elo is pondering worth by Michel Van den Bergh, CCC, August 07, 2013 » Pondering
- Contempt and the ELO model by Michel Van den Bergh, CCC, September 05, 2013 » Contempt Factor
- 1 draw=1 win + 1 loss (always!) by Michel Van den Bergh, CCC, September 19, 2013
- SPRT and narrowing of (elo1 - elo0) difference by Jesús Muñoz, CCC, October 05, 2013 » SPRT
- sprt and margin of error by Larry Kaufman, CCC, October 15, 2013 » SPRT
- How (not) to use SPRT ? by BB+, OpenChess Forum, October 19, 2013
- Houdini, much weaker engines, and Arpad Elo by Kai Laskos, CCC, November 29, 2013 » Houdini, Pawn Advantage, Win Percentage, and Elo [43]
- Testing on time control versus nodes | ply by Ed Schröder, CCC, December 04, 2013
20142015 ...
- 2-SPRT by Michel Van den Bergh, CCC, January 28, 2015 » SPRT
- Script for computing SPRT probabilities by Michel Van den Bergh, CCC, April 05, 2015
- Maximum ELO gain per test game played? by Forrest Hoch, CCC, April 20, 2015
- Getting SPRT right by Alexandru Mosoi, CCC, April 22, 2015 » SPRT
- SPRT questions by Uri Blass, CCC, May 15, 2015 » SPRT
- Adam Hair's article on Pairwise comparison of engines by Charles Roberson, CCC, May 19, 2015 [44]
- computing elo of multiple chess engines by Alexandru Mosoi, CCC, August 09, 2015
- Some musings about search by Ed Schröder, CCC, August 14, 2015 » Automated Tuning, Search
- Bullet vs regular time control, say 40/4m CCRL/CEGT by Ed Schröder, CCC, August 29, 2015
- The SPRT without draw model, elo model or whatever... by Michel Van den Bergh, CCC, September 01, 2015 » SPRT
- Name for elo without draws? by Marcel van Kervinck, CCC, September 02, 2015
- The future of chess and elo ratings by Larry Kaufman, CCC, September 20, 2015 » Opening Book
- Depth of Satisficing by Ken Regan, Gödel's Lost Letter and P=NP, October 06, 2015 » Depth, Match Statistics, Pawn Advantage, Win Percentage, and Elo, Stockfish, Komodo [45]
- ELO error margin by Fabio Gobbato, CCC, October 17, 2015
- testing multiple versions & elo calculation by Folkert van Heusden, CCC, October 27, 2015
- A simple expression by Kai Laskos, CCC, December 09, 2015
- Counting 1 win + 1 loss as 2 draws by Kai Laskos, CCC, December 15, 2015
2016Re: The SPRT without draw model, elo model or whatever.. by Michel Van den Bergh, CCC, August 18, 2016
- A Chess Firewall at Zero? by Ken Regan, Gödel's Lost Letter and P=NP, January 21, 2016
- Ordo 1.0.9 (new features for testers) by Miguel A. Ballicora, CCC, January 25, 2016
- Why the errorbar is wrong ... simple example! by Frank Quisinsky, CCC, February 23, 2016
- a direct comparison of FIDE and CCRL rating systems by Erik Varend, CCC, February 22, 2016 » FIDE, CCRL
- Some properties of the Type I error in p-value stopping rule by Kai Laskos, CCC, March 01, 2016
- A Visual Look at 2 Million Chess Games - Thinking Through the Party by Buğra Fırat, March 02, 2016
- Type I error for p-value stopping: Balanced and Unbalanced by Kai Laskos, CCC, June 16, 2016
- Empirically Logistic ELO model better suited than Gaussian by Kai Laskos, CCC, July 12, 2016
- Testing resolution and combining results by Daniel José Queraltó, CCC, July 28, 2016
- Error margins via resampling (jackknifing) by Kai Laskos, CCC, August 12, 2016 [46] [47]
- Properties of unbalanced openings using Bayeselo model by Kai Laskos, CCC, August 27, 2016 » Opening Book
- ELO inflation ha ha ha by Henk van den Belt, CCC, September 16, 2016 » Delphil, Stockfish, Playing Strength, TCEC Season 9 [48]
- The scaling with time of opening books by Kai Laskos, CCC, September 23, 2016 » Opening Book
- Perfect play by Patrik Karlsson, CCC, September 28, 2016
- Stockfish underpromotes much more often than Komodo by Kai Laskos, CCC, October 05, 2016 » Komodo, Promotions, Stockfish
- Differences between top engines related to "style" by Kai Laskos, October 07, 2016
- SPRT when not used for self testing by Andrew Grant, CCC, October 21, 2016
- Doubling of time control by Andreas Strangmüller, CCC, October 21, 2016 » Doubling TC, Diminishing Returns, Playing Strength, Komodo
- Stockfish 8 - Double time control vs. 2 threads by Andreas Strangmüller, CCC, November 15, 2016 » Doubling TC, Diminishing Returns, Playing Strength, Stockfish
- When Data Serves Turkey by Ken Regan, Gödel's Lost Letter and P=NP, November 30, 2016
- Magnus and the Turkey Grinder by Ken Regan, Gödel's Lost Letter and P=NP, December 08, 2016 » Pawn Advantage, Win Percentage, and Elo [49]
- Regan's conundrum by Carl Lumma, CCC, December 09, 2016
- Statistical Interpretation by Dennis Sceviour, CCC, December 10, 2016
- Absolute ELO scale by Nicu Ionita, CCC, December 17, 2016
- A question about SPRT by Andrew Grant, CCC, December 25, 2016 » SPRT
- Diminishing returns and hyperthreading by Kai Laskos, CCC, December 27, 2016 » Diminishing Returns, Playing Strength, Thread
2017About expected scores and draw ratios by Jesús Muñoz, CCC, September 17, 2016
- Progress in 30 years by four intervals of 7-8 years by Kai Laskos, CCC, January 19, 2017 » Playing Strength
- sprt tourney manager by Richard Delorme, CCC, January 24, 2017 » Amoeba Tournament Manager, SPRT
- Binomial distribution for chess statistics by Lyudmil Antonov, CCC, March 03, 2017
- Higher than expected by me efficiency of Ponder ON by Kai Laskos, CCC, March 06, 2017 » Pondering
- What can be said about 1 - 0 score? by Kai Laskos, CCC, March 28, 2017
- 6-men Syzygy from HDD and USB 3.0 by Kai Laskos, CCC, April 04, 2017 » Komodo, Playing Strength, Syzygy Bases, USB 3.0
- Scaling of engines from FGRL rating list by Kai Laskos, CCC, April 07, 2017 » FGRL
- Low impact of opening phase in engine play? by Kai Laskos, CCC, April 18, 2017 » Opening
- How to simulate a game outcome given Elo difference? by Nicu Ionita, CCC, April 25, 2017
- Wilo rating properties from FGRL rating lists by Kai Laskos, CCC, May 01, 2017 » FGRL
- MATCH sanity by Ed Schroder, CCC, May 03, 2017 » Portable Game Notation
- Symmetric multiprocessing (SMP) scaling - SF8 and K10.4 by Andreas Strangmüller, CCC, May 05, 2017 » Lazy SMP, Komodo, Stockfish
- Symmetric multiprocessing (SMP) scaling - K10.4 Contempt=0 by Andreas Strangmüller, CCC, May 11, 2017 » SMP, Komodo, Contempt Factor
- Symmetric multiprocessing (SMP) scaling - SF8 Contempt=10 by Andreas Strangmüller, CCC, May 13, 2017 » SMP, Stockfish, Contempt Factor
- Likelihood Of Success (LOS) in the real world by Kai Laskos, CCC, May 26, 2017
- Opening testing suites efficiency by Kai Laskos, CCC, June 21, 2017 » Engine Testing, Opening
- Testing A against B by playing a pool of others by Andrew Grant, CCC, June 24, 2017
- Engine testing & error margin ? by Mahmoud Uthman, CCC, July 05, 2017
- Invariance with time control of rating schemes by Kai Laskos, CCC, July 22, 2017 [50]
- Ways to avoid "Draw Death" in Computer Chess by Kai Laskos, CCC, July 25, 2017
- SMP NPS measurements by Peter Österlund, CCC, August 06, 2017 » Lazy SMP, Parallel Search, Nodes per second
- What is a Match? by Henk van den Belt, CCC, September 01, 2017
- Scaling from FGRL results with top 3 engines by Kai Laskos, CCC, September 26, 2017 » FGRL, Houdini, Komodo, Stockfish
- Statistical interpretation of search and eval scores by J. Wesley Cleveland, CCC, November 18, 2017 » Pawn Advantage, Win Percentage, and Elo, Score
- "Intrinsic Chess Ratings" by Regan, Haworth -- seq by Kai Middleton, CCC, November 19, 2017
- ELO progression measured by year by Ed Schroder, CCC, December 13, 2017
2018ELO measurements by Peter Österlund, CCC, August 06, 2017 » Playing Strength
Re: "Intrinsic Chess Ratings" by Regan, Haworth -- by Kenneth Regan, CCC, November 20, 2017 » Who is the Master?
External Links
Rating Systems
Tools
Statistics
Data Visualization
Misc
ARMS Charity Concert, Madison Square Garden, December 08, 1983
References
What links here?
Up one level