criterion performance measurements
overview
want to understand this report?
fib1/10
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.1948541052545269e-6 | 1.1963876464594169e-6 | 1.1980507327045165e-6 |
| Standard deviation | 2.964859054711564e-9 | 4.061078526332601e-9 | 6.515836770018346e-9 |
Outlying measurements have no (5.263010554015876e-3%) effect on estimated standard deviation.
fib1/20
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.4563597621189685e-4 | 1.4581975820150044e-4 | 1.460448079334158e-4 |
| Standard deviation | 3.8119869895589255e-7 | 5.195070915727966e-7 | 7.792177603651192e-7 |
Outlying measurements have slight (1.0868252626494244e-2%) effect on estimated standard deviation.
fib1/30
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.7913937863513876e-2 | 1.793440244990244e-2 | 1.7963283570631906e-2 |
| Standard deviation | 2.9946233973022967e-5 | 4.744815829531345e-5 | 7.547360801503968e-5 |
Outlying measurements have slight (6.632653061224489e-2%) effect on estimated standard deviation.
fib1/31
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.9005916155711787e-2 | 2.9031581617517566e-2 | 2.9067324976448837e-2 |
| Standard deviation | 2.4616538254445622e-5 | 4.815587694498758e-5 | 6.904645985353252e-5 |
Outlying measurements have slight (8.264462809917356e-2%) effect on estimated standard deviation.
fib1/32
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.690236754094745e-2 | 4.6947869860618496e-2 | 4.700250577819514e-2 |
| Standard deviation | 5.1941382736446665e-5 | 7.464441730575787e-5 | 9.579431026418075e-5 |
Outlying measurements have moderate (0.10937500000000001%) effect on estimated standard deviation.
fib1/33
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.592200507054589e-2 | 7.598744610363528e-2 | 7.603308038916048e-2 |
| Standard deviation | 3.800635874596015e-5 | 7.169659871713805e-5 | 1.0873597923527458e-4 |
Outlying measurements have moderate (0.13888888888888878%) effect on estimated standard deviation.
fib1/34
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.12286678376642193 | 0.12296382563881499 | 0.12306481563944396 |
| Standard deviation | 3.12905511025835e-5 | 1.1291335204548393e-4 | 1.5244500692987117e-4 |
Outlying measurements have moderate (0.15999999999999998%) effect on estimated standard deviation.
fib1/35
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.1989545459823275 | 0.19903974573733607 | 0.19908962273939848 |
| Standard deviation | 0.0 | 7.694710889079077e-5 | 8.638950170132701e-5 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
fib1/36
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.3202900569876656 | 0.320840914903684 | 0.32168001874127367 |
| Standard deviation | 0.0 | 7.384529115659125e-4 | 8.024947926088977e-4 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
fib1/37
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.5192291629879425 | 0.5203321789035237 | 0.5209785064880416 |
| Standard deviation | 0.0 | 9.966945053692781e-4 | 1.1194722147181728e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
fib1/38
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.8403694394823688 | 0.8464887180704576 | 0.8505546708234678 |
| Standard deviation | 0.0 | 6.099030974527389e-3 | 7.042436749388366e-3 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
fib1/39
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.3715258682407072 | 1.3764856106816108 | 1.3813303120958669 |
| Standard deviation | 2.7194799110210365e-16 | 8.293435983947122e-3 | 8.3912689969922e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
fib1/40
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.224040356983518 | 2.2259061069321953 | 2.2274691727681297 |
| Standard deviation | 0.0 | 2.486972426785467e-3 | 2.7073094434136085e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
fib1/41
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.5910069969883285 | 3.594542289487538 | 3.5968461119885515 |
| Standard deviation | 0.0 | 3.458480091658616e-3 | 3.990337623375977e-3 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
fib1/42
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.808229568241797 | 5.817351495237443 | 5.823378865317633 |
| Standard deviation | 0.0 | 9.042140148697244e-3 | 1.0439711214908132e-2 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
fib2/10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.933442933507513e-4 | 6.956046627245967e-4 | 6.981667941426397e-4 |
| Standard deviation | 5.0726468103869515e-6 | 6.171013574663284e-6 | 8.587175896184636e-6 |
Outlying measurements have slight (1.63888888888892e-2%) effect on estimated standard deviation.
fib2/100000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.722059156549107e-2 | 3.7339751331082624e-2 | 3.7441655223727784e-2 |
| Standard deviation | 1.3919958317192493e-4 | 1.866035070330769e-4 | 2.222639491963453e-4 |
Outlying measurements have slight (9.876543209876543e-2%) effect on estimated standard deviation.
fib2/200000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.16632729232058044 | 0.1672504540146207 | 0.16774350840179975 |
| Standard deviation | 0.0 | 8.050378181046402e-4 | 8.539952494888356e-4 |
Outlying measurements have moderate (0.18749999999999994%) effect on estimated standard deviation.
fib2/300000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.39099952698276325 | 0.3937089456816886 | 0.39519223993438773 |
| Standard deviation | 0.0 | 2.377923245318885e-3 | 2.569141008249796e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
fib2/400000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.7060221849828522 | 0.707322241848388 | 0.7084579856010774 |
| Standard deviation | 0.0 | 1.8414356459779864e-3 | 1.96716588403704e-3 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
fib2/500000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.1084999733214402 | 1.1096492260979494 | 1.1106117131512776 |
| Standard deviation | 0.0 | 1.5311791266332559e-3 | 1.6670764779916434e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
fib2/600000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.5918317624823102 | 1.6021360400704416 | 1.608037892905643 |
| Standard deviation | 0.0 | 9.202838841967736e-3 | 1.0222308969362538e-2 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
fib2/700000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.190081318990662 | 2.203424456264557 | 2.212247913761501 |
| Standard deviation | 0.0 | 1.323650691173846e-2 | 1.528267668313162e-2 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
fib2/800000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.8495261304819572 | 2.850556289404066 | 2.8511864915728187 |
| Standard deviation | 0.0 | 9.566606995427467e-4 | 1.0915421753192316e-3 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
fib2/900000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.62590776074124 | 3.628749903904364 | 3.6331789189830155 |
| Standard deviation | 0.0 | 3.8866511808311128e-3 | 4.198005168209701e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
fib2/1000000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.449817350490987 | 4.476902813931905 | 4.4946877847626645 |
| Standard deviation | 0.0 | 2.668681474769245e-2 | 3.080447309000606e-2 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
fib3/1000000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.711726257491624e-3 | 9.726223188327167e-3 | 9.735938738379748e-3 |
| Standard deviation | 1.620025333373791e-5 | 2.4102670133663463e-5 | 3.57127454802883e-5 |
Outlying measurements have slight (4.986149584487535e-2%) effect on estimated standard deviation.
fib3/10000000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.1355018553212867 | 0.13571079309794246 | 0.13581785498806856 |
| Standard deviation | 0.0 | 1.8111255840034174e-4 | 1.8543663325278374e-4 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
fib3/20000000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.2923753539821191 | 0.29267511618157654 | 0.29293823693441584 |
| Standard deviation | 0.0 | 4.275535095133336e-4 | 4.557385124434665e-4 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
fib3/30000000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.5137438673218421 | 0.5139800450984305 | 0.5141247729853179 |
| Standard deviation | 0.0 | 2.1959435974102358e-4 | 2.5067605336084783e-4 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
fib3/40000000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.6814420847415312 | 0.6819009081823945 | 0.6822921074295979 |
| Standard deviation | 0.0 | 6.272687197561494e-4 | 6.775769720389792e-4 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
fib3/50000000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.9603743297404725 | 0.962725039571121 | 0.9639174536539988 |
| Standard deviation | 0.0 | 2.036417606413073e-3 | 2.0653217752050115e-3 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
fib3/60000000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.1750068376546938 | 1.178046977126341 | 1.1801763698218952 |
| Standard deviation | 0.0 | 3.2051998734235233e-3 | 3.6882163379659133e-3 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
fib3/70000000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.3516466779824112 | 1.3777103307093073 | 1.3919187322668376 |
| Standard deviation | 0.0 | 2.284610377694944e-2 | 2.4609673391983188e-2 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
fib3/80000000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.5841668059877605 | 1.5896100071540802 | 1.5931930049885608 |
| Standard deviation | 0.0 | 5.375814005712346e-3 | 6.205934292729485e-3 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
fib3/90000000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.6515933774830627 | 1.6632674105709777 | 1.6744627291569405 |
| Standard deviation | 0.0 | 1.89898629106032e-2 | 1.9390860597807452e-2 |
Outlying measurements have moderate (0.18749999999999994%) effect on estimated standard deviation.
fib3/100000000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.307685259482663 | 2.3102182369873656 | 2.311842331656743 |
| Standard deviation | 0.0 | 2.4419089267466785e-3 | 2.813014483663255e-3 |
Outlying measurements have moderate (0.18749999999999994%) effect on estimated standard deviation.
fib3/200000000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.860844151322128 | 4.878806298348639 | 4.8920682253219425 |
| Standard deviation | 0.0 | 2.017202026246113e-2 | 2.297033132402966e-2 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.