About

This page contains results comparing ordered_set with and without decentralized counters.

Benchmark Description

The benchmark measures how many ETS operations per second X Erlang processes can perform on a single table. Each of the X processes repeatedly selects an operation to do from a given set of operations. The likelihood that a certain operation will be selected is also given to the benchmark. The table that the processes operate on is prefilled with 500K items before each benchmark run starts. The source code for the benchmark is located in the function ets_SUITE:throughput_benchmark/0 (see "$ERL_TOP/lib/stdlib/test/ets_SUITE.erl"). Below is a list with brief descriptions of the operations:

insert = an ets:insert/2 call with a random item within the key range as the second parameter value
remove = an ets:remove/2 call with a random item within the key range as the second parameter value
lookup = an ets:lookup/2 call with a random item within the key range as the second parameter value
lookup = an ets:lookup/2 call with a random item within the key range as the second parameter value
nextseqX = a sequence of X ets:next/2 calls. The first next call is given a random item within the key range as the second parameter value, and the other next calls are given the result of the previous next call as the second parameter value.
selectAll = an ets:select_count call that counts all the items in the table
partial_selectX = an ets:select_count call that counts all the items within a random range of size X

Benchmark Machine and Erlang Parameters

Processor: NUMA machine with four Intel(R) Xeon(R) CPU E5-4650 CPUs (2.70GHz), eight cores each (i.e., the machine has a total of 32 physical cores, each with hyperthreading, which makes a total of 64 logical cores).
Memory: 128GB of RAM
OS: Debian GNU/Linux 9

Benchmark Results

Settings:

Note that that lines can also be hidden by clicking on the corresponding label.

Code

# Each instance of the benchmark runs for 3.0 seconds:
# The result of a benchmark instance is presented as a number
representing
# the number of operations performed per second:

# To plot graphs for the results below:
# 1. Open
"$ERL_TOP/lib/stdlib/test/ets_SUITE_data/visualize_throughput.html" in a
 web browser
# 2. Copy the lines between "#BENCHMARK STARTED$" and "#BENCHMARK
ENDED$" below
# 3. Paste the lines copied in step 2 to the text box in the browser
window opened in
#    step 1 and press the Render button

#BENCHMARK STARTED$
Scenario:  50.000000% insert,  50.000000% delete | Key Range Size:
1000000$
; 1; 2; 4; 8; 16; 32; 64$
[ordered_set,public,{write_concurrency,true}] ; 652872.666667;
835821.000000; 1256706.666667; 2767671.333333; 5292470.333333;
9995096.000000; 15532162.666667$
[ordered_set,public,{write_concurrency,true},{read_concurrency,true}] ;
663275.333333; 746566.000000; 1473009.333333; 2837511.666667;
5464639.666667; 9840317.000000; 15424325.333333$
[set,public,{write_concurrency,true}] ; 941498.666667; 978429.333333;
1545221.333333; 2019850.666667; 2447201.666667; 2971504.000000;
2656963.333333$
[set,public,{write_concurrency,true},{read_concurrency,true}] ;
731715.000000; 963034.666667; 1515825.666667; 2526119.666667;
2538718.666667; 3079339.000000; 2883586.666667$
Scenario:  10.000000% insert,  10.000000% delete,  80.000000% lookup |
Key Range Size: 1000000$
; 1; 2; 4; 8; 16; 32; 64$
[ordered_set,public,{write_concurrency,true}] ; 600243.666667;
817602.666667; 1768749.666667; 3697379.000000; 6820433.666667;
12171236.333333; 19277618.333333$
[ordered_set,public,{write_concurrency,true},{read_concurrency,true}] ;
549970.000000; 933068.666667; 1610183.000000; 3070474.333333;
5698226.666667; 10093939.666667; 15422129.333333$
[set,public,{write_concurrency,true}] ; 830462.666667; 1582155.333333;
2369625.333333; 4583978.666667; 5133664.333333; 5931011.333333;
6944685.333333$
[set,public,{write_concurrency,true},{read_concurrency,true}] ;
846048.333333; 1073562.000000; 1573887.666667; 3252649.666667;
6148835.333333; 8664137.000000; 9240178.000000$
Scenario:  1.000000% insert,  1.000000% delete,  98.000000% lookup | Key
 Range Size: 1000000$
; 1; 2; 4; 8; 16; 32; 64$
[ordered_set,public,{write_concurrency,true}] ; 656812.000000;
1026749.666667; 1939832.666667; 4166140.333333; 8061738.333333;
14375336.000000; 21668750.666667$
[ordered_set,public,{write_concurrency,true},{read_concurrency,true}] ;
701200.333333; 1027669.000000; 1992324.333333; 3964703.000000;
7635268.000000; 12979743.000000; 20277949.666667$
[set,public,{write_concurrency,true}] ; 922454.333333; 1153374.333333;
2229115.000000; 5002256.666667; 11099578.666667; 18854089.000000;
15735376.333333$
[set,public,{write_concurrency,true},{read_concurrency,true}] ;
895954.666667; 1099076.666667; 1997938.333333; 4097063.000000;
8698491.000000; 14246836.666667; 18911014.333333$
Scenario:  100.000000% lookup | Key Range Size: 1000000$
; 1; 2; 4; 8; 16; 32; 64$
[ordered_set,public,{write_concurrency,true}] ; 701498.333333;
815477.333333; 1625373.000000; 1865178.000000; 2308608.666667;
2640283.666667; 3483945.333333$
[ordered_set,public,{write_concurrency,true},{read_concurrency,true}] ;
786518.333333; 1103603.666667; 1901781.000000; 4204926.666667;
8154268.333333; 16021936.333333; 25182765.000000$
[set,public,{write_concurrency,true}] ; 1085858.000000; 1345946.000000;
2128191.000000; 5075605.333333; 10203657.666667; 20118775.333333;
28284082.333333$
[set,public,{write_concurrency,true},{read_concurrency,true}] ;
950605.666667; 1325196.000000; 2252287.666667; 4354509.666667;
9613663.333333; 20007792.333333; 30256830.000000$
Scenario:  10.000000% insert,  10.000000% delete,  40.000000% lookup,
40.000000% nextseq10 | Key Range Size: 1000000$
; 1; 2; 4; 8; 16; 32; 64$
[ordered_set,public,{write_concurrency,true}] ; 318717.666667;
520777.333333; 901520.000000; 1765387.666667; 2864041.000000;
3917042.000000; 4793727.000000$
[ordered_set,public,{write_concurrency,true},{read_concurrency,true}] ;
281886.333333; 480635.666667; 838652.000000; 1569714.666667;
2596520.000000; 3639792.000000; 4525442.666667$
[set,public,{write_concurrency,true}] ; 353794.000000; 641290.333333;
1121096.666667; 1951741.000000; 2392815.333333; 2288797.666667;
2389323.333333$
[set,public,{write_concurrency,true},{read_concurrency,true}] ;
353085.333333; 559121.333333; 923814.000000; 1671587.666667;
2580325.666667; 3341850.333333; 3852132.333333$
Scenario:  10.000000% insert,  10.000000% delete,  40.000000% lookup,
40.000000% nextseq100 | Key Range Size: 1000000$
; 1; 2; 4; 8; 16; 32; 64$
[ordered_set,public,{write_concurrency,true}] ; 63413.666667;
108956.000000; 221462.333333; 413977.666667; 701819.333333;
1083058.666667; 1435329.666667$
[ordered_set,public,{write_concurrency,true},{read_concurrency,true}] ;
48088.666667; 107131.333333; 205847.666667; 389316.000000;
628157.000000; 1020123.666667; 1381612.000000$
[set,public,{write_concurrency,true}] ; 75258.666667; 150814.666667;
274198.666667; 481539.000000; 706132.333333; 567883.666667;
493596.666667$
[set,public,{write_concurrency,true},{read_concurrency,true}] ;
86136.000000; 116193.333333; 248156.666667; 454364.333333;
717659.666667; 1082514.333333; 1332467.666667$
Scenario:  10.000000% insert,  10.000000% delete,  79.999900% lookup,
0.000100% selectAll | Key Range Size: 1000000$
; 1; 2; 4; 8; 16; 32; 64$
[ordered_set,public,{write_concurrency,true}] ; 635519.333333;
940926.000000; 1760969.666667; 3522265.666667; 6516154.000000;
11754391.333333; 17384096.000000$
[ordered_set,public,{write_concurrency,true},{read_concurrency,true}] ;
686654.333333; 895173.000000; 1461285.666667; 2946554.666667;
5684068.000000; 9676382.333333; 14127639.333333$
[set,public,{write_concurrency,true}] ; 912185.000000; 1412458.333333;
1803265.666667; 3028742.000000; 3483857.000000; 4265102.333333;
5020466.666667$
[set,public,{write_concurrency,true},{read_concurrency,true}] ;
776707.666667; 796505.333333; 1605823.333333; 2794511.000000;
3864057.000000; 3975542.000000; 5078355.333333$
Scenario:  10.000000% insert,  10.000000% delete,  79.000000% lookup,
1.000000% partial_select1000 | Key Range Size: 1000000$
; 1; 2; 4; 8; 16; 32; 64$
[ordered_set,public,{write_concurrency,true}] ; 793.333333; 1409.666667;
 3539.666667; 6448.000000; 12841.666667; 25046.000000; 43687.333333$
[ordered_set,public,{write_concurrency,true},{read_concurrency,true}] ;
569.666667; 1551.333333; 3721.333333; 6044.000000; 12994.666667;
23402.000000; 43982.333333$
[set,public,{write_concurrency,true}] ; 866.666667; 1751.666667;
2711.000000; 4971.000000; 11794.333333; 21406.000000; 40060.000000$
[set,public,{write_concurrency,true},{read_concurrency,true}] ;
1499.333333; 962.333333; 2777.666667; 5735.666667; 11424.333333;
21970.000000; 38735.333333$

#BENCHMARK ENDED$
# Each instance of the benchmark runs for 3.0 seconds:
# The result of a benchmark instance is presented as a number
representing
# the number of operations performed per second:

#BENCHMARK STARTED$
Scenario:  50.000000% insert,  50.000000% delete | Key Range Size:
1000000$
; 1; 2; 4; 8; 16; 32; 64$
[ordered_set,public,{write_concurrency,true}] ; 551210.000000;
796432.666667; 1177155.666667; 1717264.333333; 2122377.333333;
2404208.000000; 3486105.000000$
[ordered_set,public,{write_concurrency,true},{read_concurrency,true}] ;
613031.000000; 714389.666667; 1137868.666667; 1706604.333333;
2149434.333333; 2431626.333333; 3500979.666667$
[set,public,{write_concurrency,true}] ; 857329.333333; 1105575.000000;
1563588.666667; 1886429.000000; 2181626.666667; 1843487.000000;
2153501.666667$
[set,public,{write_concurrency,true},{read_concurrency,true}] ;
911586.333333; 907030.333333; 1362435.333333; 1858066.000000;
2116525.000000; 2420779.000000; 1675749.000000$
Scenario:  10.000000% insert,  10.000000% delete,  80.000000% lookup |
Key Range Size: 1000000$
; 1; 2; 4; 8; 16; 32; 64$
[ordered_set,public,{write_concurrency,true}] ; 743317.333333;
866109.333333; 1838977.000000; 3213990.666667; 5047350.666667;
6947664.333333; 8967087.666667$
[ordered_set,public,{write_concurrency,true},{read_concurrency,true}] ;
694889.333333; 885474.000000; 1609609.333333; 2851503.333333;
4428585.666667; 6228946.666667; 8019331.000000$
[set,public,{write_concurrency,true}] ; 941474.000000; 1496504.000000;
2601803.666667; 4040674.000000; 4971342.000000; 5525500.666667;
6240278.333333$
[set,public,{write_concurrency,true},{read_concurrency,true}] ;
822177.666667; 1054594.666667; 1816476.666667; 3252239.666667;
4804607.666667; 5860742.000000; 6277836.333333$
Scenario:  1.000000% insert,  1.000000% delete,  98.000000% lookup | Key
 Range Size: 1000000$
; 1; 2; 4; 8; 16; 32; 64$
[ordered_set,public,{write_concurrency,true}] ; 679850.666667;
1125891.666667; 2318248.333333; 4474486.000000; 7918213.000000;
13568093.666667; 19146358.333333$
[ordered_set,public,{write_concurrency,true},{read_concurrency,true}] ;
697821.666667; 1203342.666667; 2326244.000000; 4373830.000000;
7714174.666667; 13173260.000000; 17995926.333333$
[set,public,{write_concurrency,true}] ; 898652.666667; 1700953.333333;
3359807.333333; 6520632.000000; 10421773.000000; 11298571.666667;
11993870.333333$
[set,public,{write_concurrency,true},{read_concurrency,true}] ;
895880.666667; 1669757.000000; 3226240.000000; 6086956.333333;
10589495.666667; 16320336.333333; 16297338.666667$
Scenario:  100.000000% lookup | Key Range Size: 1000000$
; 1; 2; 4; 8; 16; 32; 64$
[ordered_set,public,{write_concurrency,true}] ; 742677.000000;
1025457.666667; 1267962.666667; 1404803.333333; 1534999.333333;
1775437.000000; 2914679.333333$
[ordered_set,public,{write_concurrency,true},{read_concurrency,true}] ;
677695.333333; 1307626.333333; 2732280.666667; 5405518.000000;
10209850.000000; 19431247.666667; 28097920.333333$
[set,public,{write_concurrency,true}] ; 1069576.666667; 1887774.666667;
3531473.333333; 7188044.333333; 13276245.333333; 24593168.000000;
31709139.333333$
[set,public,{write_concurrency,true},{read_concurrency,true}] ;
997049.666667; 1847891.333333; 3628141.000000; 7267659.666667;
11042089.333333; 25746685.000000; 35628619.666667$
Scenario:  10.000000% insert,  10.000000% delete,  40.000000% lookup,
40.000000% nextseq10 | Key Range Size: 1000000$
; 1; 2; 4; 8; 16; 32; 64$
[ordered_set,public,{write_concurrency,true}] ; 287069.000000;
459084.333333; 939351.000000; 1608939.333333; 2430748.666667;
3406912.333333; 4215285.000000$
[ordered_set,public,{write_concurrency,true},{read_concurrency,true}] ;
281450.000000; 459171.666667; 783379.000000; 1462425.000000;
2264491.000000; 3176144.333333; 3976193.000000$
[set,public,{write_concurrency,true}] ; 366653.333333; 618377.333333;
1143153.000000; 1845907.333333; 2344412.333333; 2282283.000000;
2313405.666667$
[set,public,{write_concurrency,true},{read_concurrency,true}] ;
351726.333333; 563974.333333; 1008964.666667; 1704755.000000;
2384573.666667; 3041798.333333; 3380451.666667$
Scenario:  10.000000% insert,  10.000000% delete,  40.000000% lookup,
40.000000% nextseq100 | Key Range Size: 1000000$
; 1; 2; 4; 8; 16; 32; 64$
[ordered_set,public,{write_concurrency,true}] ; 73922.333333;
118070.333333; 220634.000000; 389979.666667; 662100.000000;
1006710.666667; 1347836.333333$
[ordered_set,public,{write_concurrency,true},{read_concurrency,true}] ;
58643.000000; 107708.000000; 203165.000000; 373624.666667;
614185.000000; 923179.000000; 1287867.000000$
[set,public,{write_concurrency,true}] ; 81073.666667; 145176.666667;
274305.000000; 474113.666667; 695175.333333; 654652.666667;
521729.666667$
[set,public,{write_concurrency,true},{read_concurrency,true}] ;
59011.333333; 130824.000000; 242060.000000; 437161.666667;
712853.333333; 1068380.666667; 1334069.666667$
Scenario:  10.000000% insert,  10.000000% delete,  79.999900% lookup,
0.000100% selectAll | Key Range Size: 1000000$
; 1; 2; 4; 8; 16; 32; 64$
[ordered_set,public,{write_concurrency,true}] ; 716881.000000;
907463.333333; 1773958.000000; 2970048.333333; 4579450.000000;
6515943.333333; 8805360.000000$
[ordered_set,public,{write_concurrency,true},{read_concurrency,true}] ;
668070.333333; 943032.666667; 1592447.000000; 2659500.000000;
4128187.666667; 8571624.333333; 11438172.333333$
[set,public,{write_concurrency,true}] ; 802529.000000; 1493695.666667;
1772434.333333; 2895415.333333; 3622338.000000; 3331032.333333;
5300562.333333$
[set,public,{write_concurrency,true},{read_concurrency,true}] ;
793842.000000; 991773.000000; 1683647.000000; 2546523.666667;
3466018.333333; 4815306.000000; 3897967.666667$
Scenario:  10.000000% insert,  10.000000% delete,  79.000000% lookup,
1.000000% partial_select1000 | Key Range Size: 1000000$
; 1; 2; 4; 8; 16; 32; 64$
[ordered_set,public,{write_concurrency,true}] ; 537.000000; 1277.666667;
 3347.666667; 6368.333333; 11640.000000; 23215.333333; 38687.666667$
[ordered_set,public,{write_concurrency,true},{read_concurrency,true}] ;
830.333333; 1027.666667; 3187.000000; 5947.333333; 11660.000000;
22532.333333; 39860.000000$
[set,public,{write_concurrency,true}] ; 373.000000; 1211.333333;
2471.666667; 5137.000000; 10605.666667; 21512.333333; 40374.000000$
[set,public,{write_concurrency,true},{read_concurrency,true}] ;
583.666667; 1509.666667; 3449.000000; 5598.000000; 11683.333333;
20908.666667; 40322.000000$

#BENCHMARK ENDED$

Bar Plot
Same X Spacing Between Points
Show [ordered_set,public]
Show [ordered_set,public,{write_concurrency,true}]
Show [ordered_set,public,{read_concurrency,true}]
Show [ordered_set,public,{write_concurrency,true},{read_concurrency,true}]
Show [set,public]
Show [set,public,{write_concurrency,true}]
Show [set,public,{read_concurrency,true}]
Show [set,public,{write_concurrency,true},{read_concurrency,true}]