# anonymous-auditing-dpsgd
Anonymous Code for "Auditing Differentially Private Machine Learning:How Private is Private SGD?", NeurIPS submission 1902.

To reproduce results, first modify code/auditing_args.py to set the appropriate variables, based on the examples from <code>
code/args/</code>. For example, getting the ClipBKD plot from Figure 1a uses the following setting:

<pre><code>
args = {
        "pois_ct": [1, 2, 4, 8],
        "clip_norm": [1],
        "init_mult": [1],
        "trials": (0, 1000),
        "dataset": "fmnist",
        "old_bkd": False,
        "model": "2f",
        "save_dir": "../../test/fmnist_2f",
        "data_dir": "../datasets"
}
</code></pre>

<code>"pois_ct"</code>: list of poisoning counts, a subset of [1, 2, 4, 8]

<code>"clip_norm"</code>: list of clipping norms, can take any float

<code>"init_mult"</code>: list of initialization randomness scales, can take any float

<code>"trials"</code>: (trial_start_index, trial_end_index)

<code>"dataset"</code>: a string representing the dataset

<code>"old_bkd"</code>:  True if using old backdoor, False else

<code>"model"</code>: "2f" is neural networks, "lr" is logistic regression

<code>"save_dir"</code>: results directory

<code>"data_dir"</code>: directory holding data



Then, run

<code>
python code/exp_run.py
</code>

This will run a lot of experiments, each of which calls audit.py. This will take a while, it this many trials, using the default 6 poisoning rates:

<code>6*len(args["pois_ct"])*len(args["clip_norm"])*len(args["init_mult"])*(args["trials"][1]-args["trials"][0])</code>

You may want to use nohup. It parallelizes on CPU.


After those experiments finish, run

<code>
python code/run_make.py
</code>

This computes the test statistics over all models in <code>args["save_dir"]</code>, in batches of 50. This takes some time, but is much faster than <code>code/exp_run.py</code>.


When run_make.py finishes, run

<code>
python code/combine_files.py
</code>

This combines the batches of experiments. This is fast.


After that finishes, run 

<code>
python code/bkd_parser.py
</code>

This computes the lower bounds: it prints the results, but also produces a file with the results:

<code>
args["save_dir"]/results/results.npy
</code>
        
which stores a dict containing a mapping from parameter settings to results.
