Methodology

Scientific validation report

How accurate Carbon-Eye's maps are, measured on 2026-10-15 at 7 Kenyan test areas - from a 0.3 ha plot to a whole county - on land the models never trained on.

In short

  • The model beats a do-nothing baseline (predicting the median everywhere) at every site, removing 30%-74% of its error.
  • The 90% ranges held the true reference value 90% of the time on average (sites 87%-95%), against a 90% target.
  • Dense vegetation is under-estimated: errors grow with carbon density, as with every satellite biomass map. The tables below show by how much.
  • A benchmark of tuned settings and other algorithms led to new Random Forest and Gradient Tree Boosting settings (adopted 2026-10-16, after the figures on this page were measured); gradient-boosting libraries not available in Earth Engine did 6% better than the previous ensemble.
  • Since 15 November 2026, land the ESA map leaves blank (below its detection limit) counts as zero biomass in training and testing, which removed an over-estimate in drylands (e.g. Machakos: +12.8% to −0.9% against ESA's own county total). The site figures on this page were measured before that change.
  • Since 2026-11-21, new runs correct the forest under-estimate when the models compress dense areas: on held-out test blocks at 6 areas the 50-100 t C/ha band went from −10.6 to −6.6 t C/ha on average, with overall error almost unchanged. Details.
  • Against independent maps (2026-11-21, 6 areas), Carbon-Eye agrees on where the carbon is but reads lower than NASA/ORNL's 2010 map - conservative totals. Details.
  • Accuracy here means agreement with the ESA CCI reference map, not with field plots. An independent lidar check is reported separately.

How it was measured

  • Reference. ESA CCI Above-Ground Biomass v6 (2020), converted to carbon with the IPCC fraction 0.47. Imagery window 2024-10-01 to 2026-10-01.
  • Spatial blocks. Each area is cut into squares and whole squares are held out: held-out spatial blocks (~0.71 km squares, ~101 blocks; test areas never trained on). Test points are never next to training points, which is stricter than a random split.
  • Three-way split. Models are fitted on one part, the ensemble is weighted on a second, and every number here comes from the third - the test blocks.
  • Baseline. Predicting the median of the held-out reference points everywhere. “Error removed” is 1 - RMSE(model) / RMSE(baseline).
  • 90% ranges. Calibrated on held-out points, then checked on the test blocks: the share of test points whose reference value fell inside its range.

Results by site

SiteMean (t C/ha)R² RF / GTB / SVMEnsemble RMSEBaseline RMSEError removed90% coverageTest points
Kitui smallholder farm (277 ha)9.00.53 / 0.54 / 0.507.912.034%91%530
0.25 ha plot, Kitui4.80.57 / 0.56 / 0.557.513.142%87%461
Mau farmland (~970 ha)6.00.50 / 0.47 / 0.406.69.430%88%625
Kakamega rainforest (~2,500 ha)96.80.89 / 0.89 / 0.9114.555.974%92%269
Nairobi CBD (built-up)--9.27 / -12.95 / -11.99--n/a-4
Kirinyaga county23.90.90 / 0.90 / 0.898.028.171%89%1095
Makueni county10.00.86 / 0.88 / 0.784.714.868%95%952
Kitui farm, GEDI lidar training on9.90.04 / 0.59 / 0.568.012.034%95%530

R² rewards variety: small uniform farms score lower R² even when their error in t C/ha is small. The built-up control (Nairobi CBD) has almost no reference biomass, so there is nothing to score - it is included to show the map correctly reports no carbon there. The lidar-training row uses the same farm with the opt-in GEDI setting.

Cyclists on a red earth road through dense green rainforest
A road through Kakamega Forest, Kenya's last stretch of Guineo-Congolian rainforestPhoto: Nao Iizuka, CC BY 3.0, via Wikimedia Commons (resized)

Errors by carbon density

Average error (predicted minus reference, t C/ha) on the test points, by the reference carbon density. Negative means the map is too low. Bands with fewer than 5 points are not scored (count shown).

Site0-10 t C/ha10-25 t C/ha25-50 t C/ha50-100 t C/ha100+ t C/ha
Kitui smallholder farm (277 ha)+2.8 (n=319)-2.5 (n=141)-14.1 (n=65)-22.5 (n=5)-
0.25 ha plot, Kitui+3.0 (n=271)-1.4 (n=135)-12.0 (n=50)-26.8 (n=5)-
Mau farmland (~970 ha)+0.8 (n=484)-4.5 (n=119)-18.6 (n=17)-47.7 (n=5)-
Kakamega rainforest (~2,500 ha)+10.7 (n=12)+5.6 (n=33)+3.3 (n=35)-0.6 (n=27)-2.1 (n=162)
Kirinyaga county+0.8 (n=677)+3.7 (n=181)-0.6 (n=144)-8.1 (n=56)-17.3 (n=37)
Makueni county+0.8 (n=673)+1.2 (n=154)+0.1 (n=106)-14.1 (n=19)-
  • Kitui smallholder farm (277 ha): in the 25-50 t C/ha band the map is 14.1 t C/ha too low on average (65 points).
  • 0.25 ha plot, Kitui: in the 25-50 t C/ha band the map is 12.0 t C/ha too low on average (50 points).
  • Mau farmland (~970 ha): in the 25-50 t C/ha band the map is 18.6 t C/ha too low on average (17 points).
  • Kirinyaga county: in the 100+ t C/ha band the map is 17.3 t C/ha too low on average (37 points).
  • Makueni county: in the 50-100 t C/ha band the map is 14.1 t C/ha too low on average (19 points).
  • This is signal saturation and the pull of any model towards the average: satellites stop seeing more biomass in dense canopy. Totals for dense woodland and forest are therefore likely under-estimates; sparse farmland tends to read slightly high. Every run now shows this breakdown in its Validation tab.

Forest correction (2026-11-21)

Measured 2026-11-20 on held-out spatial test blocks. The correction: actual = a + b x ensemble, fitted on the calibration points (never trained or tested on); applied only when b > 1 and there are at least 100 calibration points. Bias is predicted minus reference (t C/ha); negative means the map is too low. Before → after.

AreaApplied25-50 t C/ha50-100 t C/haRMSE (t C/ha)Test points
Kiambu countyYes−6.0 → −4.3 (n=69)−4.7 → +0.1 (n=87)6.00 → 6.01959
Nyeri countyYes−2.2 → +1.1 (n=180)−13.0 → −7.1 (n=97)6.83 → 6.501119
Kakamega countyYes−7.0 → −6.5 (n=49)−2.6 → +0.2 (n=37)4.88 → 4.91808
Machakos countyYes−6.9 → −3.6 (n=55)-3.31 → 3.331386
Elgeyo-Marakwet countyYes−9.1 → −8.0 (n=156)−16.3 → −14.6 (n=80)9.64 → 9.541132
Kitui smallholder farm (277 ha)No (slope 0.97)−16.6 → −16.6 (n=13)-7.05 → 7.05177
Kabarak University campus (94 ha)Yes−16.6 → −14.4 (n=20)−28.7 → −24.4 (n=22)7.15 → 6.49633
Kirinyaga countyYes−2.2 → −0.9 (n=140)−7.0 → −3.0 (n=75)7.08 → 7.191091
  • Across the 6 areas with enough 50-100 t C/ha test points (398 points), the average bias in that band went from −10.6 to −6.6 t C/ha.
  • Overall RMSE changed by −0.7 to +0.1 t C/ha where the correction applied - it moves the dense end without making the rest worse.
  • It narrows the gap rather than closing it everywhere: Elgeyo-Marakwet and the Kabarak campus are still well under the reference in dense forest. Field plots and lidar are the way to close the rest.
  • A farm whose models don't compress (slope at or below 1) is left as it is, as are runs made before 2026-11-21 and runs weighted on GEDI lidar. Each run's Validation tab shows whether the correction applied and its own before/after.

Comparison with independent maps (2026-11-21)

The same areas, mapped by groups that had nothing to do with Carbon-Eye: NASA/ORNL's 2010 global biomass map and NASA GEDI L4B spaceborne lidar (2019-2023), plus the ESA CCI map Carbon-Eye learns from. Means are compared on the same land; the correlation is over 1 km cells and shows whether the maps agree on where the carbon is. Runs: current method (forest correction, spatial test), Balanced preset; the Kitui farm with the gate's settings.

AreaCarbon-EyeNASA/ORNL 2010GEDI L4B lidarESA CCI (reference)
Kiambu county17.5 t C/ha24.4 (−28.0%, r 0.90)15.6 (−16.0%, r 0.46, 57% covered)19.6 (−8.4%, r 0.95)
Kirinyaga county22.5 t C/ha27.5 (−18.1%, r 0.94)19.5 (+2.9%, r 0.80, 32% covered)22.9 (−1.9%, r 0.97)
Kakamega county14.8 t C/ha33.7 (−56.1%, r 0.76)15.7 (−21.1%, r 0.74, 20% covered)15.2 (−2.4%, r 0.94)
Machakos county4.2 t C/ha10.8 (−61.2%, r 0.75)5.7 (−27.3%, r 0.43, 89% covered)5.7 (−21.6%, r 0.85)
Kabarak University campus (94 ha)4.8 t C/ha19.2 (−74.8%)2.2 (+114.8%, 89% covered)9.2 (−47.2%)
Kitui smallholder farm (277 ha)13.0 t C/ha30.3 (−57.0%, r 0.87)25.6 (−45.3%, 33% covered)12.1 (+7.5%, r 0.89)

t C/ha; in brackets, Carbon-Eye minus that map on the same land, and the 1 km correlation where there were enough cells. GEDI samples tracks, not every hectare - its coverage is shown when under 90%.

  • Carbon-Eye reads lower than NASA/ORNL at every area (−74.8% to −18.1%), while agreeing on where the carbon is (r 0.75-0.94). NASA/ORNL describes 2010 - before a decade of tree-cover loss - with a different method.
  • Against GEDI lidar, where it covers at least half the area, the difference is −27.3% to +114.8%.
  • Most of the gap to NASA/ORNL is already in the ESA reference the models learn from (Carbon-Eye minus ESA: −47.2% to +7.5%). Totals are therefore conservative compared with NASA/ORNL; field plots, not another satellite map, settle which is right for a given site.
  • Chloris, Planet and Sylvera maps are commercial and not openly available to compare against.
Tall forest trees with a vine-covered trunk and hills of forest behind
Inside the Mau Forest, Kenya's largest closed-canopy forest and a water tower for the Rift ValleyPhoto: Kaa.rie, CC BY-SA 4.0, via Wikimedia Commons (resized)

Independent lidar check

NASA GEDI spaceborne lidar measures canopy structure directly and was never used for training (unless the opt-in setting is on). Mean difference, model minus lidar, over footprints in the area:

SiteFootprintsModel - lidar (t C/ha)Correlation
Kitui smallholder farm (277 ha)731-8.80.46
0.25 ha plot, Kitui402-6.60.35
Mau farmland (~970 ha)1134-6.20.33
Kakamega rainforest (~2,500 ha)16--
Nairobi CBD (built-up)15--
Kirinyaga county1450+8.60.62
Makueni county1482+4.60.19
Kitui farm, GEDI lidar training on188-6.80.13

The ESA reference - and any model trained only on it - reads lower than lidar on farms and higher in some counties. That gap is the reference map's, which is why the opt-in “Calibrate with GEDI lidar” setting exists.

Model benchmark

Nested spatial cross-validation at 5 sites (Kakamega rainforest: 978 points, 119 blocks; Kirinyaga county: 3835 points, 127 blocks; Kitui farm: 1688 points, 122 blocks; Makueni county: 3259 points, 131 blocks; Mau farmland: 2530 points, 119 blocks). Three outer folds of whole blocks; inside each, two inner block folds choose settings from a small grid, so the outer fold that scores them never influenced the choice. All models see the same points and folds.

ModelPooled RMSEPooled R²Error removed vs medianBias
Median baseline25.970.3080.0%-4.77
Ensemble (app defaults)8.570.92567.0%-0.60
RF default9.110.91564.9%-0.01
RF tuned8.430.92767.5%+0.10
GTB default8.690.92266.5%-0.93
GTB tuned8.480.92667.3%-0.75
SVM default9.050.91665.1%-0.82
SVM tuned8.950.91865.6%-0.88
Ensemble (tuned)8.300.92968.0%-0.50
Extra Trees8.190.93168.5%+0.36
XGBoost8.090.93368.8%+0.15
LightGBM8.230.93068.3%+0.11
  • Settings adopted. The rule, fixed before looking: change a default only if better at 4 of 5 sites, by at least 2% on average, with no site more than 2% worse. Each candidate was then re-checked as a fixed setting on every outer fold: RF {min_leaf:15,vps:6} → {min_leaf:5,vps:10}: -8.4% RMSE, better at 5/5 sites (adopted). GTB {shrinkage:0.05,max_nodes:6} → {shrinkage:0.1,max_nodes:12}: -4.4% RMSE, better at 5/5 sites (adopted). SVM tuning failed the rule (better at 4/5, worst site 2.5%), so it is unchanged.
  • Other algorithms. Extra Trees, XGBoost and LightGBM beat the previous ensemble at every site (by 5.8%, 6.7% and 5.9% lower RMSE on average). They don't run inside Google Earth Engine, where the maps are made; the new settings close about 63% of that gap. Bringing tree models trained outside Earth Engine into the map is a possible next step.

Limitations

  • Accuracy is measured against a modelled reference map (ESA CCI), not field plots. Its own errors are in the labels.
  • High-biomass bands have few test points at farm sites; their numbers are indicative.
  • Seven test areas in Kenya are not every landscape: drylands, mangroves and plantations are not separately tested yet.
  • Field-plot validation (e.g. national forest inventory plots) would be the next independent check.

Reproduce it

Every run records its dataset versions, settings, code revision and an area fingerprint, and gives the same numbers for the same inputs. Saved areas can be recomputed from scratch by a second person, who sees whether the result fingerprint matches. Area fingerprints of the sites above:

  • Kitui smallholder farm (277 ha): sha256:bc58f5445cabdc3e
  • 0.25 ha plot, Kitui: sha256:9766a077160bb648
  • Mau farmland (~970 ha): sha256:e98f97d0cf2a0342
  • Kakamega rainforest (~2,500 ha): sha256:15cdb2a3baa7e35d
  • Nairobi CBD (built-up): sha256:b2f894137ae5372d
  • Kirinyaga county: sha256:2ffd15bc93f1827a
  • Makueni county: sha256:05e734606d50e4e6
  • Kitui farm, GEDI lidar training on: sha256:bc58f5445cabdc3e