# Experimental Evaluation Scripts

## Requirements and Assumptions
The scripts assume that the benchmark files are located in
the `../downloaded_benchmarks/drivers` directory.

To download the drivers benchmarks, run the following command:

```python download_benchmark.py --blist drivers_list.txt ```

<!--- #################################################### -->

## How to obtain the data for tables in Section 4

Table 2 summarizes the data of Table 13 in the Appendix.

The data for Tables 2 and 13 come from corresponding
protected sheets in file `tables/table-havoc-transform-effect.ods`;
we have the following mapping (sheet ==> table):
  - table-havoc-transform-effect       ==> Table 13
  - table-havoc-transform-effect-descr ==> Table 2

The contents of these protected sheets are obtained by the raw data
contained in the (non-protected) sheets named:
  - pos-nonrel
  - def-nonrel
  - pos-rel
  - def-rel

These are obtained by running ```python havoc_stats.py```

The script will first compute text files in directory
`DRIVER-NUM-HAVOC` and then produce from them the excel file
`DRIVER-NUM-HAVOC/DRIVER-NUM-HAVOC.xlsx`,
which contains sheets with the same name (but using the drivers
full names rather than their id).

<!--- #################################################### -->

## How to obtain the data for tables in Section 5

### How to obtain the data for tables in Section 5.1

Section 5.1 contains Tables 3, 4 and 5 for havoc analysis
accuracy evaluation; Tables 3 and 4 summarize Tables 14 and 15
in the Appendix.

The data for Tables 14, 15, 3, 4, 5 come from corresponding
protected sheets in file `tables/table-havoc-analysis-accuracy.ods`;
we have the following mapping (sheet ==> table):
  - table-nonrel       ==> Table 14
  - table-rel          ==> Table 15
  - table-descr-nonrel ==> Table 3
  - table-descr-rel    ==> Table 4
  - table-acc          ==> Table 5

The contents of these protected sheets are obtained by the raw data
contained in the (non-protected) sheets named:
  - pos-nonrel
  - def-nonrel
  - pos-rel
  - def-rel

The raw data for these sheets is obtained by calling python script
`compute_havoc_accuracy.py` (explained below).

<!--- #################################################### -->

### How to obtain the data for tables in Section 5.2

Section 5.2 contains Tables 6, 7 and 8 for havoc transform
accuracy evaluation; Tables 6 and 7 summarize Tables 16 and 17
in the Appendix.

The data for Tables 16, 17, 6, 7, 8 come from corresponding
protected sheets in file
```tables/table-havoc-transform-accuracy.ods```
which has the same structure as table-havoc-analysis-accuracy.ods.
Hence, we have the following mapping (sheet ==> table):
  - table-nonrel       ==> Table 16
  - table-rel          ==> Table 17
  - table-descr-nonrel ==> Table 6
  - table-descr-rel    ==> Table 7
  - table-acc          ==> Table 8

As before, the contents of these protected sheets are obtained
by the raw data contained in the (non-protected) sheets named:
  - pos-nonrel
  - def-nonrel
  - pos-rel
  - def-rel

The raw data for these sheets is obtained by calling python script
`compute_havoc_accuracy.py` (explained below).

### How to use script `compute_havoc_accuracy.py`

This script, which should be called by passing an oracle name, e.g.,

```python compute_havoc_accuracy.py --oracle=pos-nonrel```

will perform the accuracy evaluation for *both* the havoc analysis
(Section 5.1) and the havoc transform (Section 5.2),
producing two output files, having the same name `pos-nonrel.txt`,
in directories `DRIVER-HAVOC-ANALYSIS-ACCURACY` and
`DRIVER-HAVOC-TRANSFORM-ACCURACY`.
The contents of these files, interpreted as csv files

```libreoffice --calc all_drivers.txt ```

is used to provide the raw data for the corresponding sheet
in files

```tables/table-havoc-analysis-accuracy.ods```

 and

```tables/table-havoc-transform-accuracy.ods```

 (see above).

For convenience, the call to

```python compute_havoc_accuracy_all.py```

will run the script above for all the four oracles
(warning: this will take a very long time):
 - pos-nonrel
 - def-nonrel
 - pos-rel
 - def-rel

Once all the text files have been obtained, running the script

```python havoc_accuracy_excel_file.py --path=DRIVER-HAVOC-ANALYSIS-ACCURACY ```

and

```python havoc_accuracy_excel_file.py --path=DRIVER-HAVOC-TRANSFORM-ACCURACY ```

will produce an excel file in each directory.

<!--- #################################################### -->

## How to obtain the data for tables in Section 6.1

Section 6.1 contains Tables 9 and 10, which summarize Tables 18 and 19
in the Appendix, providing the precision comparison for
target analysis results.

We first describe how to obtain precision data,
then how to compare precision data,
and finally how to produce the tables.

### How to compare precision of target analysis results

The script `compute_target_precision.py` should be called with
appropriate values for parameters `oracle` and `domain`;
for instance, the call

```python compute_target_precision.py --oracle=none --domain=int ```

will create directory

```TARGET-PRECISION/none-int```

and populate it with
a subdirectory for each driver, containing a .json file with the
invariants computed for each function in the driver.

For convenience, the call to

```python compute_target_precision_all.py ```

will run the script above for all the needed parameter combinations
(warning: this will take a long time):
  - none + int
  - pos-nonrel + int
  - def-nonrel + int
  - none + pk-ap-pplite
  - pos-rel + pk-ap-pplite
  - def-rel + pk-ap-pplite
  - pos-nonrel + pk-ap-pplite
  - def-nonrel + pk-ap-pplite

### How to compare precision of target analysis results

After having computed precision data (see above), the script
`compare_target_precision.py` should be called with
appropriate values for parameters
`oracle1`, `domain1`, `oracle2` and `domain2`;
for instance, the call

```python compare_target_precision.py --oracle1=pos-nonrel --domain1=int --oracle2=none --domain2=int ```

assuming that directories

  `TARGET-PRECISION/pos-nonrel-int`

and

  `TARGET-PRECISION/none-int`

have been populated before, will create directory

  `TARGET-PRECISION/CMP/pos-nonrel-int-vs-none-int`

and populate it with a file for each driver (containing the corresponding
results of the precision comparison, with function level granularity)
as well as file

  ```all_drivers.txt```

containing the precision comparison with driver level granularity.
This file can be opened as a csv file

```libreoffice --calc all_drivers.txt ```

The call to

```python compare_target_precision_all.py ```

will run the script above for all the needed parameter combinations
(warning: this may take a long time):

  - (pos-nonrel + int) vs (none + int)
  - (def-nonrel + int) vs (none + int)
  - (pos-rel + pk-ap-pplite) vs (none + pk-ap-pplite)
  - (def-rel + pk-ap-pplite) vs (none + pk-ap-pplite)
  - (pos-nonrel + pk-ap-pplite) vs (none + pk-ap-pplite)
  - (def-nonrel + pk-ap-pplite) vs (none + pk-ap-pplite)

### How to obtain target precision tables from raw data

The data for Tables 9, 10, 18, 19 come from corresponding
protected sheets in file `tables/table-target-precision.ods`;
we have the following mapping (sheet ==> table):
  - table-nonrel-int-precision       ==> Table 18
  - table-nonrel-pk-precision        ==> Table 19
  - table-nonrel-int-precision-descr ==> Table 9
  - table-nonrel-pk-precision-descr  ==> Table 10

The contents of these protected sheets are obtained by the raw data
contained in the (non-protected) sheets, which match the raw data
produced by script `compare_target_precision_all.py` (sheet name ==> raw data filename):
  - pos-nonrel-int ==> TARGET-PRECISION/CMP/pos-nonrel-int-vs-none-int/all_drivers.txt
  - def-nonrel-int ==> TARGET-PRECISION/CMP/def-nonrel-int-vs-none-int/all_drivers.txt
  - pos-nonrel-pk ==> TARGET-PRECISION/CMP/pos-nonrel-pk-vs-none-pk/all_drivers.txt
  - def-nonrel-pk ==> TARGET-PRECISION/CMP/def-nonrel-pk-vs-none-pk/all_drivers.txt

<!--- #################################################### -->

## How to obtain the data for tables in Section 6.2

Section 6.2 contains Table 11, which summarizes Table 20
in the Appendix, providing the efficiency comparison for
the target analysis.

The data for Tables 11 and 20 come from corresponding
protected sheets in file `tables/table-target-efficiency.ods`;
we have the following mapping (sheet ==> table):
  table-target-efficiency-nonrel-poly ==> Table 20
  table-target-efficiency-nonrel-poly-descr ==> Table 11

The contents of these protected sheets are obtained by the raw data
contained in the (non-protected) sheets named:
  - none-pk-ap-pplite
  - pos-nonrel-pk-ap-pplite
  - def-nonrel-pk-ap-pplite

The raw data for these sheets is obtained by calling python script
`compute_target_efficiency.py` as follows:
  - ```python compute_target_efficiency.py --oracle=none --domain=pk-ap-pplite```
  - ```python compute_target_efficiency.py --oracle=pos-nonrel --domain=pk-ap-pplite```
  - ```python compute_target_efficiency.py --oracle=def-nonrel --domain=pk-ap-pplite```

These calls will populate directory `TARGET-EFFICIENCY` with corresponding text files:
  - none-pk-ap-pplite.txt
  - pos-nonrel-pk-ap-pplite.txt
  - def-nonrel-pk-ap-pplite.txt

