criterion performance measurements

overview

want to understand this report?

fib1/10

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.1948541052545269e-6 1.1963876464594169e-6 1.1980507327045165e-6
Standard deviation 2.964859054711564e-9 4.061078526332601e-9 6.515836770018346e-9

Outlying measurements have no (5.263010554015876e-3%) effect on estimated standard deviation.

fib1/20

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.4563597621189685e-4 1.4581975820150044e-4 1.460448079334158e-4
Standard deviation 3.8119869895589255e-7 5.195070915727966e-7 7.792177603651192e-7

Outlying measurements have slight (1.0868252626494244e-2%) effect on estimated standard deviation.

fib1/30

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.7913937863513876e-2 1.793440244990244e-2 1.7963283570631906e-2
Standard deviation 2.9946233973022967e-5 4.744815829531345e-5 7.547360801503968e-5

Outlying measurements have slight (6.632653061224489e-2%) effect on estimated standard deviation.

fib1/31

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.9005916155711787e-2 2.9031581617517566e-2 2.9067324976448837e-2
Standard deviation 2.4616538254445622e-5 4.815587694498758e-5 6.904645985353252e-5

Outlying measurements have slight (8.264462809917356e-2%) effect on estimated standard deviation.

fib1/32

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 4.690236754094745e-2 4.6947869860618496e-2 4.700250577819514e-2
Standard deviation 5.1941382736446665e-5 7.464441730575787e-5 9.579431026418075e-5

Outlying measurements have moderate (0.10937500000000001%) effect on estimated standard deviation.

fib1/33

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 7.592200507054589e-2 7.598744610363528e-2 7.603308038916048e-2
Standard deviation 3.800635874596015e-5 7.169659871713805e-5 1.0873597923527458e-4

Outlying measurements have moderate (0.13888888888888878%) effect on estimated standard deviation.

fib1/34

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.12286678376642193 0.12296382563881499 0.12306481563944396
Standard deviation 3.12905511025835e-5 1.1291335204548393e-4 1.5244500692987117e-4

Outlying measurements have moderate (0.15999999999999998%) effect on estimated standard deviation.

fib1/35

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.1989545459823275 0.19903974573733607 0.19908962273939848
Standard deviation 0.0 7.694710889079077e-5 8.638950170132701e-5

Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.

fib1/36

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.3202900569876656 0.320840914903684 0.32168001874127367
Standard deviation 0.0 7.384529115659125e-4 8.024947926088977e-4

Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.

fib1/37

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.5192291629879425 0.5203321789035237 0.5209785064880416
Standard deviation 0.0 9.966945053692781e-4 1.1194722147181728e-3

Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.

fib1/38

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.8403694394823688 0.8464887180704576 0.8505546708234678
Standard deviation 0.0 6.099030974527389e-3 7.042436749388366e-3

Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.

fib1/39

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.3715258682407072 1.3764856106816108 1.3813303120958669
Standard deviation 2.7194799110210365e-16 8.293435983947122e-3 8.3912689969922e-3

Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.

fib1/40

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.224040356983518 2.2259061069321953 2.2274691727681297
Standard deviation 0.0 2.486972426785467e-3 2.7073094434136085e-3

Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.

fib1/41

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.5910069969883285 3.594542289487538 3.5968461119885515
Standard deviation 0.0 3.458480091658616e-3 3.990337623375977e-3

Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.

fib1/42

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.808229568241797 5.817351495237443 5.823378865317633
Standard deviation 0.0 9.042140148697244e-3 1.0439711214908132e-2

Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.

fib2/10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 6.933442933507513e-4 6.956046627245967e-4 6.981667941426397e-4
Standard deviation 5.0726468103869515e-6 6.171013574663284e-6 8.587175896184636e-6

Outlying measurements have slight (1.63888888888892e-2%) effect on estimated standard deviation.

fib2/100000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.722059156549107e-2 3.7339751331082624e-2 3.7441655223727784e-2
Standard deviation 1.3919958317192493e-4 1.866035070330769e-4 2.222639491963453e-4

Outlying measurements have slight (9.876543209876543e-2%) effect on estimated standard deviation.

fib2/200000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.16632729232058044 0.1672504540146207 0.16774350840179975
Standard deviation 0.0 8.050378181046402e-4 8.539952494888356e-4

Outlying measurements have moderate (0.18749999999999994%) effect on estimated standard deviation.

fib2/300000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.39099952698276325 0.3937089456816886 0.39519223993438773
Standard deviation 0.0 2.377923245318885e-3 2.569141008249796e-3

Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.

fib2/400000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.7060221849828522 0.707322241848388 0.7084579856010774
Standard deviation 0.0 1.8414356459779864e-3 1.96716588403704e-3

Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.

fib2/500000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.1084999733214402 1.1096492260979494 1.1106117131512776
Standard deviation 0.0 1.5311791266332559e-3 1.6670764779916434e-3

Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.

fib2/600000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.5918317624823102 1.6021360400704416 1.608037892905643
Standard deviation 0.0 9.202838841967736e-3 1.0222308969362538e-2

Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.

fib2/700000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.190081318990662 2.203424456264557 2.212247913761501
Standard deviation 0.0 1.323650691173846e-2 1.528267668313162e-2

Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.

fib2/800000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.8495261304819572 2.850556289404066 2.8511864915728187
Standard deviation 0.0 9.566606995427467e-4 1.0915421753192316e-3

Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.

fib2/900000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.62590776074124 3.628749903904364 3.6331789189830155
Standard deviation 0.0 3.8866511808311128e-3 4.198005168209701e-3

Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.

fib2/1000000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 4.449817350490987 4.476902813931905 4.4946877847626645
Standard deviation 0.0 2.668681474769245e-2 3.080447309000606e-2

Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.

fib3/1000000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 9.711726257491624e-3 9.726223188327167e-3 9.735938738379748e-3
Standard deviation 1.620025333373791e-5 2.4102670133663463e-5 3.57127454802883e-5

Outlying measurements have slight (4.986149584487535e-2%) effect on estimated standard deviation.

fib3/10000000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.1355018553212867 0.13571079309794246 0.13581785498806856
Standard deviation 0.0 1.8111255840034174e-4 1.8543663325278374e-4

Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.

fib3/20000000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.2923753539821191 0.29267511618157654 0.29293823693441584
Standard deviation 0.0 4.275535095133336e-4 4.557385124434665e-4

Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.

fib3/30000000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.5137438673218421 0.5139800450984305 0.5141247729853179
Standard deviation 0.0 2.1959435974102358e-4 2.5067605336084783e-4

Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.

fib3/40000000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.6814420847415312 0.6819009081823945 0.6822921074295979
Standard deviation 0.0 6.272687197561494e-4 6.775769720389792e-4

Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.

fib3/50000000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.9603743297404725 0.962725039571121 0.9639174536539988
Standard deviation 0.0 2.036417606413073e-3 2.0653217752050115e-3

Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.

fib3/60000000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.1750068376546938 1.178046977126341 1.1801763698218952
Standard deviation 0.0 3.2051998734235233e-3 3.6882163379659133e-3

Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.

fib3/70000000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.3516466779824112 1.3777103307093073 1.3919187322668376
Standard deviation 0.0 2.284610377694944e-2 2.4609673391983188e-2

Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.

fib3/80000000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.5841668059877605 1.5896100071540802 1.5931930049885608
Standard deviation 0.0 5.375814005712346e-3 6.205934292729485e-3

Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.

fib3/90000000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.6515933774830627 1.6632674105709777 1.6744627291569405
Standard deviation 0.0 1.89898629106032e-2 1.9390860597807452e-2

Outlying measurements have moderate (0.18749999999999994%) effect on estimated standard deviation.

fib3/100000000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.307685259482663 2.3102182369873656 2.311842331656743
Standard deviation 0.0 2.4419089267466785e-3 2.813014483663255e-3

Outlying measurements have moderate (0.18749999999999994%) effect on estimated standard deviation.

fib3/200000000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 4.860844151322128 4.878806298348639 4.8920682253219425
Standard deviation 0.0 2.017202026246113e-2 2.297033132402966e-2

Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.

understanding this report

In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.

Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.

We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)

A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.