Hardware Model Checking Competition Report
HWMCC’15 Armin Biere1 and Keijo Heljanko2 1 Johannes Kepler University, Austria 2 Aalto University, Finland
15th International Conference on
Formal Methods in Computer-Aided Design FMCAD’15 The University of Texas at Austin Austin, TX, USA Wednesday, 30th September, 2015 added tip2014bmc data after the competition on 2nd Oct. 2015
History
1/23
2nd HWMCC
AIGER format AVM’06 Ascona
1st HWMCC
Founding Lunch CAV’06 FLOC’06 Seattle
CAV’07
2006
2007
Berlin
CAV’08 Princeton
3rd HWMCC
4th
5th
6th
7th
8th
HWMCC
HWMCC
HWMCC
HWMCC
HWMCC FMCAD’15
HWMCC Lunch
CAV’10 FLOC’10
FMCAD’11
FMCAD’12
FMCAD’13
CAV’14 FLOC’14
FMCAD’08
Edinburgh
Austin
Cambridge UK
Portland USA
Vienna Austria
2012
2013
Portland
2008
2010
2011
2014
Austin USA
2015
founding lunch at CAV’06, first competition at CAV’07 HWMCC lunch at FMCAD’08 ⇒ need multiple properties !!! affilliated with either CAV (7,8,10,14) or FMCAD (11,12,13,15) HWMCC’11:
old SINGLE, new LIVEness and new MULTI property track
HWMCC’12 as HWMCC’11, new DEEP bounds track
sponsored by Oski
in essence no change in HWMCC’12 - HWMCC’15 HWMCC’15: DEEP, SINGLE, and LIVE, MULTI, 1h Hardware Model Checking Competition 2015
time limit, before 15min
Tracks
2/23
SINGLE single safety property, specified as bad state monitor SAT = bad state reachable
dual safety property does not hold
UNSAT = bad state not reachable
dual safety property does holds
restricted (pre AIGER 1.9) models only: thus zero initialized latches (actually flip-flops) and no environment constraints new Oski benchmarks went through
aigtosmv | smvtoaig-20071012
DEEP focus on hard unsolved benchmarks in SINGLE “how deep” can the model checker prove absence of bad state for instance in BMC:
print bounds for which unrolling UNSAT
LIVE single justice properties (with fairness and environment constraints) generalized Buchi automata ¨ Hardware Model Checking Competition 2015
Model Checkers
3/23
19 model checkers from 11 teams abcsimple, abcsimplive, abcsuprove from Berkeley aigbmc, blimc from JKU Linz avy from Technion+SEI+Princeton iimc from Boulder iprover,iproverdeep from Manchester nuxmv,nuxmvbmc from Trento pdtravdeep, pdtravthrd from Torino ricecnu from Rice
Brayton,Sterin,Mishchenko,. . . Biere Vizel,Gurfinkel,Malik Somenzi,Bradley,Hassan Tsarkov,Korovin Griggio,Roveri,. . . Cabodi,Quer,. . . Li,Vardi
shiftbmc from Dresden
Manthey
tip2014 from Chalmers
¨ Sorensson,Claessen
v3s from Taipei Hardware Model Checking Competition 2015
Yang,Wu,Huang
AIGER
4/23 O0
4-bit adder
50
O1
46
54
O2
52
38
48
44
36
42
40
34
14
16
L0 O3
58
30
8 62
56
28
32
60
22
26
4
10
12
24
20
y[3]
6
8
18
x[0]
y[0]
x[2]
4
12
reset
2
14
x[3]
10
y[1]
2
y[2]
Hardware Model Checking Competition 2015
Q
!Q
x[1]
enable
6
toggle flip-flop with enable & reset
Benchmarks LIVE with the identical 223 LIVE benchmarks from HWMCC’14 removed “outputs” in cu* benchmarks though SINGLE included all 79 unsolved instances from HWMCC’14 in SINGLE track: 6s(53), intel(9), oski(7), beem(5), bob(4), nusmv(1)
added 156 new benchmarks from Oski submitted by Vigyan Singhal oski15a01(80), oski15a07(6), oski15a08(18), oski15a10(18), oski15a10(34)
classified all 2075 SINGLE benchmarks on my hard disk into 19 classes removed 79 unsolved from HWMCC’14 removed 690 solved by 11 out of 16 model checkers in 36 seconds randomly selected further 313 benchmarks out of remaining 1306 as follows: varying fraction of selected benchmarks per class fraction determined to balance origin and importance in total 548 benchmarks in SINGLE Hardware Model Checking Competition 2015
5/23
SINGLE Randomly Selected Benchmarks selected
class
remaining
fraction
6/23
original
79 39 38 38 26 16 13 11 8 8 8 6 5 5 4 4 3 2 0
6s pdt bob beem intel bv eijk nusmv vis oski lmcs pj ken bjrb nec mentor cmu texas regr
237 234 76 446 53 98 26 23 16 16 16 13 10 10 9 9 7 5 2
1/3 1/6 1/2 1/12 1/2 1/6 1/2 1/2 1/2 1/2 1/2 1/2 1/2 1/2 1/2 1/2 1/2 1/3 1/inf
314 347 147 680 62 129 28 32 26 24 156 14 16 44 13 15 7 19 2
313
total
1306
˜1/4.17
2075
Hardware Model Checking Competition 2015
HWMCC'15 Cactus LIVE Track SAT+UNSAT
3000
●
●
●
nuxmv abcsimplive iimc v3s aigbmc
●●
●●● ● ● ● ●
0
1000
2000
●
● ● ●● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
0
50
100
150
HCMCC’15 Table LIVE SAT+UNSAT rank
cnt
1 2 3
223 223 223 223 223
nuxmv abcsimplive iimc v3s aigbmc
ok sat uns fld to mo unk 176 173 164 100 74
96 95 96 54 74
80 47 46 78 50 19 68 59 14 46 123 68 0 149 66
1 31 45 47 58
0 0 0 8 25
real
time max best uniq
26037 102160 5254 21294 74824 6954 8709 29772 4354 57957 117237 7000 18122 18054 4507
hors concours (not ranked): aigbmc: cnt ok sat uns fld to mo s6 s11 unk real time max best uniq
organizer model checker
number of benchmarks considered (for ranking) solved benchmarks (sat or unsat) solved satisfiable benchmarks solved unsatisfiable benchmarks unsolved benchmarks (cnt - ok) time out (3600 wall clock time) memory out (7GB main memory) signal 6 caught (abort) signal 11 caught (segmentation faulg) other signal or no result in output wall clock time process time maximum memory usage (MB) number of benchmarks solved fastest number of uniquely solved benchmarks
47 41 67 3 26
1 1 3 0 2
HWMCC'15 Cactus LIVE Track SAT
●
iimc nuxmv abcsimplive aigbmc v3s ●●
1000
2000
3000
●
●
0
● ●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
0
20
40
60
80
100
HCMCC’15 Table LIVE SAT rank
cnt ok sat fld to mo
1 iimc 2 nuxmv 3 abcsimplive aigbmc v3s
100 100 100 100 100
96 96 95 74 54
96 96 95 74 54
real
4 3 1 6149 4 4 0 8924 5 2 3 8313 26 4 22 18122 46 23 23 38969
hors concours (not ranked): aigbmc:
organizer model checker
time max best uniq 20686 34820 28860 18054 67309
2835 5254 6954 4507 2577
20 30 24 26 0
2 1 0 2 0
3000
HWMCC'15 Cactus LIVE Track UNSAT
●
●
nuxmv abcsimplive iimc v3s
●●
● ● ●
1000
2000
●
●
0
● ●● ● ●● ● ●●●● ●●● ● ●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
0
20
40
60
80
HCMCC’15 Table LIVE UNSAT rank 1 nuxmv 2 abcsimplive 3 iimc v3s
cnt ok uns fld to mo unk 84 84 84 84
80 78 68 46
80 78 68 46
4 4 0 6 2 4 16 0 16 38 31 6
real
time max best uniq
0 17113 67340 4838 0 12982 45964 6485 0 2560 9086 4354 1 18988 49927 7000
17 17 47 3
0 1 1 0
HWMCC'15 Cactus SINGLE Track SAT+UNSAT ● ●
3000
●
●
2000
●
●
1000
●
●
0
●
abcsimple abcsuprove nuxmv pdtravthrd avy iimc tip2014 v3s blimc aigbmc shiftbmc tip2014bmc nuxmvbmc iproverhc pdtravdeep ricecnu iproverdeephc iproverdeep iprover
● ●
● ● ●
● ● ●● ●
●
●
● ● ● ●
●
● ● ●
● ●
● ● ●
●
● ● ● ●
●
● ● ● ● ● ● ● ● ● ● ● ● ● ● ●
● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ●● ●● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●
0
100
200
300
HWMCC’15 Table SINGLE SAT+UNSAT rank
cnt
1
548 548 548 548 548 548 548 548 548 548 548 548 548 548 548 548 548 548 548
abcsimple abcsuprove 2 nuxmv pdtravthrd 3 avy iimc tip2014 v3s blimc aigbmc shiftbmc tip2014bmc nuxmvbmc iproverhc pdtravdeep ricecnu iproverdeephc iproverdeep iprover
ok sat uns fld 363 358 357 353 345 335 315 296 141 126 120 117 115 112 105 100 93 86 83
122 120 127 115 115 105 98 62 128 126 120 117 115 64 46 30 46 46 43
241 238 230 238 230 230 217 234 13 0 0 0 0 48 59 70 47 40 40
185 190 191 195 203 213 233 252 407 422 428 431 433 436 443 440 455 462 465
to
mo s11 s6 unk
real
154 31 0 0 0 45438 156 34 0 0 0 41596 165 26 0 0 0 47600 62 124 3 0 6 52819 171 30 0 0 2 49724 163 50 0 0 0 57840 233 0 0 0 0 46817 141 7 102 2 0 54273 287 20 0 0 100 31687 310 43 0 0 69 34210 247 12 0 0 169 37819 225 50 0 0 156 11054 256 28 0 0 149 8885 156 0 0 0 280 32269 345 86 4 0 8 14725 342 88 0 10 0 49153 177 0 0 0 278 18116 149 0 0 0 313 14899 59 0 0 0 406 8768
time max best uniq 161407 115597 186902 158951 166878 202386 46596 152125 31584 34104 37697 10911 8796 17292 14568 49054 17482 14818 8698
6785 28 6905 23 6900 30 6824 18 5924 19 6969 76 1062 144 6037 28 2241 61 3644 47 1810 28 5496 64 2504 45 6730 3 4202 11 2204 0 6728 2 6322 1 6176 3
hors concours (not ranked): aigbmc blimc: pdtravthrd: ricecnu: iprover*hc:
organizer model checkers issue catching ’FATAL’ for ’intel045’ (not counted) reports 8 instances SAT which are UNSAT (not counted) last minute (script) fixes after deadline
each team / submitter only ranked once (one medal maximum)
1 1 0 3 1 7 0 4 1 1 0 1 0 0 0 0 0 0 0
HWMCC'15 Cactus SINGLE Track SAT ●
3000
●
●
●
●
2000
●
1000
●
●
0
●
blimc nuxmv aigbmc abcsimple abcsuprove shiftbmc tip2014bmc nuxmvbmc avy pdtravthrd iimc tip2014 iproverhc v3s iproverdeep iproverdeephc pdtravdeep iprover ricecnu
● ●
●●
●
● ●
●
●●
●●
● ● ● ●● ● ●● ● ● ● ●
● ● ●
● ● ●● ● ●●● ● ●● ● ● ●●● ● ● ● ● ● ● ●●● ● ● ●●●● ● ●● ● ●● ●●● ● ● ● ●● ●●●● ● ●●● ● ● ● ● ●● ● ● ● ● ●● ● ●● ●●●●● ●●●●● ● ● ● ●●●●● ● ● ● ●●● ●● ● ●●● ●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●● ●● ●● ●● ●●●●●●●●●● ●●●●●●●● ●● ●● ●● ●● ●●● ●●●●●●● ●●●● ●● ●● ●●●●● ●●● ●● ●● ●● ●●●●●●●●●● ●● ●● ●● ●● ●● ● ●●●●●●●●● ●●●●● ●● ●● ●● ●● ● ●● ●●● ●● ●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●● ●● ●● ●● ●●●●●●●●●●●●●●●●●●● ●● ●●●●●●●●● ●●● ●● ●● ●●●●●● ●● ●● ●● ●● ●● ●● ●●●●●●●●●●●●●●●●●●● ●● ●● ●● ●● ●● ●● ●● ●● ●●
0
20
40
60
80
100
120
140
HWMCC’15 Table SINGLE SAT rank
cnt
blimc 1 nuxmv aigbmc 2 abcsimple abcsuprove 3 shiftbmc tip2014bmc nuxmvbmc avy pdtravthrd iimc tip2014 iproverhc v3s iproverdeep iproverdeephc pdtravdeep iprover ricecnu
144 144 144 144 144 144 144 144 144 144 144 144 144 144 144 144 144 144 144
ok sat fld to mo s11 s6 unk 128 127 126 122 120 120 117 115 115 115 105 98 64 62 46 46 46 43 30
128 16 14 127 17 9 126 18 16 122 22 11 120 24 15 120 24 23 117 27 23 115 29 27 115 29 21 115 29 7 105 39 19 98 46 46 64 80 19 62 82 24 46 98 20 46 98 20 46 98 83 43 101 3 30 114 75
1 8 1 11 9 0 3 1 6 15 20 0 0 0 0 0 4 0 38
0 0 0 0 0 0 0 0 0 2 0 0 0 58 0 0 3 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1
1 0 1 0 0 1 1 1 2 5 0 0 61 0 78 78 8 98 0
real 31680 18941 34210 19559 13884 37819 11054 8885 22939 23817 27651 17110 26045 26253 12214 12247 13354 6079 15532
time max best uniq 31583 74202 34104 66584 30568 37697 10911 8796 65806 57597 87239 17037 11701 75591 12164 12165 13255 6042 15500
2241 4 6899 4 3644 7 6267 4 5623 16 1810 2 5496 46 2504 5 5924 2 6108 3 6969 7 1062 43 6695 0 6037 0 6322 0 6340 0 4202 1 6176 0 2204 0
1 0 1 1 0 0 1 0 0 0 4 0 0 0 0 0 0 0 0
hors concours (not ranked): aigbmc blimc: pdtravthrd: ricecnu: iprover*hc:
organizer model checkers issue catching ’FATAL’ for ’intel045’ (not counted) reports 8 instances SAT which are UNSAT (not counted) last minute (script) fixes after deadline
HWMCC'15 Cactus SINGLE Track UNSAT
2000
3000
●
●
●
●
1000
●
0
● ●
● ●●
●
abcsimple abcsuprove pdtravthrd v3s avy nuxmv iimc tip2014 ricecnu pdtravdeep iproverhc iproverdeephc iproverdeep iprover blimc
●
●● ● ●
● ● ●
●
●
● ● ●
●
● ● ● ● ●
● ● ●
● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●●●● ● ● ● ● ●●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●●●●●●● ● ● ● ●● ● ● ● ● ● ●●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●
0
●
50
100
150
200
250
HWMCC’15 Table SINGLE UNSAT rank
cnt
1
269 269 269 269 269 269 269 269 269 269 269 269 269 269 269
abcsimple abcsuprove pdtravthrd 2 v3s 3 avy nuxmv iimc tip2014 ricecnu pdtravdeep iproverhc iproverdeephc iproverdeep iprover blimc
ok uns dis fld 241 238 238 234 230 230 230 217 70 59 48 47 40 40 13
241 238 238 234 230 230 230 217 70 59 48 47 40 40 13
0 0 0 0 0 0 0 0 8 0 0 0 0 0 0
28 31 31 35 39 39 39 52 191 210 221 222 229 229 256
to mo s11 s6 unk 22 18 15 26 30 33 31 52 167 152 82 101 80 35 147
6 13 16 3 9 6 8 0 15 57 0 0 0 0 12
0 0 0 6 0 0 0 0 0 1 0 0 0 0 0
0 0 0 0 0 0 0 0 9 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 139 121 149 194 97
real 25879 27711 29002 28020 26785 28660 30189 29707 33621 1371 6225 5869 2686 2688 7
time max best uniq 94823 85028 101354 76534 101073 112700 115147 29559 33554 1314 5592 5317 2653 2655 1
6785 24 6905 7 6824 15 5074 26 5513 17 6900 13 4613 56 431 101 1766 0 2391 2 6730 0 6728 1 5390 0 5391 0 2 7
hors concours (not ranked): aigbmc blimc: pdtravthrd: ricecnu: iprover*hc:
organizer model checkers issue catching ’FATAL’ for ’intel045’ (not counted) reports 8 instances SAT which are UNSAT (not counted) last minute (script) fixes after deadline
0 1 3 4 1 0 3 0 0 0 0 0 0 0 0
DEEP Benchmarks 135 unsolved benchmarks in SINGLE
19/23
within one hour
6s100 6s105 6s119 6s148 6s149 6s160 6s161 6s171 6s179 6s185 6s186 6s187 6s188 6s190 6s195 6s22 6s23 6s24 6s266rb2 6s267rb3 6s268r 6s274r 6s279r 6s280r 6s29 6s329rb19 6s329rb20 6s33 6s340rb27 6s340rb63 6s341r 6s342rb122 6s351rb02 6s36 6s365r 6s366r 6s376r 6s377r 6s382r 6s39 6s392r 6s398b09 6s398b16 6s399b02 6s399b03 6s402rb0342 6s44 6s45 6s514r 6s516r 6s517rb0 beembkry8b1 beemcmbrdg7f2 beemfwt4b1 beemfwt5b1 beemfwt5b3 beemhanoi4b1 beemkrebs4b1 beemldfilt7b1 beemlifts3b1 beemloyd3b1 beempgmprot1f2 beemskbn2b1 bob12s02 bobpcihm bobsmcodic bobsmminiuart intel012 intel013 intel016 intel027 intel028 intel032 intel065 intel066 intel067 nusmvdme216 oski15a01b00s oski15a01b02s oski15a01b08s oski15a01b12s oski15a01b14s oski15a01b18s oski15a01b26s oski15a01b28s oski15a01b30s oski15a01b32s oski15a01b34s oski15a01b36s oski15a01b38s oski15a01b40s oski15a01b42s oski15a01b44s oski15a01b48s oski15a01b50s oski15a01b52s oski15a01b54s oski15a01b58s oski15a01b60s oski15a01b62s oski15a01b64s oski15a01b66s oski15a01b68s oski15a01b70s oski15a01b72s oski15a01b74s oski15a01b76s oski15a01b78s oski15a07b0s oski15a07b2s oski15a07b4s oski15a08b00s oski15a08b02s oski15a08b04s oski15a08b10s oski15a08b12s oski15a10b02s oski15a10b04s oski15a10b06s oski15a10b08s oski15a10b10s oski15a10b16s oski15a14b02s oski15a14b04s oski15a14b06s oski15a14b20s oski15a14b26s oski15a14b28s oski15a14b30s oski1rub00i oski1rub01i oski1rub02i oski1rub08i oski1rub09i oski1rub10i
Hardware Model Checking Competition 2015
HWMCC'15 Cactus DEEP Bound Distribution
800
1000
● ●●●●● ●●
●
600
● ●
400
●
●
0
200
●
blimc shiftbmc aigbmc nuxmvbmc nuxmv tip2014bmc iimc tip2014 iproverhc iprover v3s pdtravdeep iproverdeephc iproverdeep
● ●
●
● ●● ● ●
●
● ● ●● ● ●● ●● ● ● ●● ●● ● ● ● ● ● ● ●● ●●●● ●● ●● ●●●●●●●● ● ● ●● ●●● ●● ●●●● ●● ●● ●● ●●●● ● ●●●●●● ●●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ●●●●● ●● ● ●● ● ● ● ●●● ●●●● ●● ●●●● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●● ●● ●●●●●●●●●●●●●●●●●● ● ●●●●●●●●
0
20
40
60
80
100
120
140
DEEP Bound Track
21/23
DEEP Bound Track award of $500 sponsored by Oski Technology 135 unsolved benchmarks in SINGLE (out of 548) reached bounds capped at 1000 (last year 100)
deep
1 135 · ∑ (1 − 1/(2 + boundi)) 135 i=1
=
boundi boundi boundi
= = =
boundi
=
boundi
=
Hardware Model Checking Competition 2015
⇒
−1 0 1
contributes contributes contributes ... 8 contributes ... 98 contributes ...
0% 50% 75% 90% 99%
boundi
HWMCC’15 Table DEEP rank
cnt
blimc 1 shiftbmc aigbmc 2 nuxmvbmc nuxmv 3 tip2014bmc iimc tip2014 iproverhc iprover v3s pdtravdeep iproverdeephc iproverdeep
135 135 135 135 135 135 135 135 135 135 135 135 135 135
to mo s11 s6 unk best 126 127 128 124 123 100 113 135 55 21 91 110 56 49
7 6 6 8 12 30 22 0 0 0 4 25 0 0
0 0 0 0 0 0 0 0 0 0 38 0 0 0
0 2 0 2 0 1 0 3 0 0 0 5 0 0 0 0 0 80 0 114 2 0 0 0 0 79 0 86
50 26 40 40 13 18 13 0 3 3 2 8 1 1
deep 93952 93733 93728 93423 93058 92031 91396 82167 79281 64248 61874 55305 32492 32398 *1e-5
’tip2014bmc’ was added after competition and solved uniquely ’6s299b685’ which lead to ’nuxmvbmc’ and ’shiftbmc’ swapping 1st and 2nd place hors concours (not ranked): aigbmc blimc: organizer model checkers iprover*hc: last minute (script) fixes after deadline others not participating and / or producing incorrect bounds
Conclusion changes this year smaller machines larger time limit of 1 hour two model checkers per team maximum changes for next year use AIGER version 1.9 format in SINGLE too reencode benchmarks with environment constraints and unitialized latches strict policy for discrepancies after final deadline more powerful HW? MULTI? plans regression, fuzzing and delta-debugging suite continue with word-level format please help us getting more benchmarks Hardware Model Checking Competition 2015
23/23
