# AbstractRendering

## Abstract Rendering of NeRF-based Scenes

### Prerequisite
Setup the conda environment from [`environment.yaml`](environment.yaml)
```
conda env create -f environment.yaml --name absrend
# activate the environment
conda activate absrend
```

Download necessary data for NeRF scene
```
cd ./AbstractNerf/data
gdown --folder https://drive.google.com/drive/folders/1w4olgQXQIda4X71PzcYTeNXFMGEH1Ea6 -O .
```


---

### Visualize Rendered Images and Evaluate Reconstruction Quality

To visualize rendered images and evaluate reconstruction quality from a specific camera pose, run:

```bash
cd ./AbstractNerf
python nerf_vis.py \
    --dataname <data_filename_in_data_folder> \
    --n_iters <number_of_training_iterations> \
    --testimgidx <index_of_estimated_image>
```

Example:
```
cd ./AbstractNerf
python nerf_vis.py
```

---

### Run Abstract Rendering with Perturbation

To perform abstract rendering with input perturbations, run:

```bash
cd ./AbstractNerf
python nerf_forward.py \
    --dataname <data_filename_in_data_folder> \
    --n_iters <number_of_training_iterations> \
    --testimgidx <index_of_estimated_image> \
    --input_type <input_dimension_involving_perturbation> \
    --xyz_eps <translation_perturbation_magnitude> \
    --ry_eps <rotation_perturbation_magnitude> \
    --xyz_step <translation_step_per_spatial_dimension> \
    --ry_step <rotation_step_per_rotational_dimension>
```

Example:
```
cd ./AbstractNerf
python nerf_forward.py
```

---

### Notes
- `--input_type` specifies the type of input subject to perturbation (e.g., position, orientation, or both).
- `--xyz_eps` and `--ry_eps` define the bounds of translation and rotation perturbations, respectively.
- `--xyz_step` and `--ry_step` define the resolution of the abstract domain for each dimension.

### Verify ResNet Classifier on NeRF-based Scenes

To verify classification robustness using a ResNet model on rendered NeRF scenes, run:

```bash
cd ./AbstractNerf
python nerf_forward_classification_resnet.py \
    --dataname <data_filename_in_data_folder> \
    --n_iters <number_of_training_iterations> \
    --testimgidx <index_of_estimated_image> \
    --input_type <input_dimension_involving_perturbation> \
    --xyz_eps <translation_perturbation_magnitude> \
    --ry_eps <rotation_perturbation_magnitude> \
    --xyz_step <translation_step_per_spatial_dimension> \
    --ry_step <rotation_step_per_rotational_dimension>
```

### Verify Gatenet Pose esimator on NeRF-based Scenes

To verify pose estimation robustness using GateNet on NeRF-rendered images, run:

```bash
cd ./AbstractNerf
python nerf_forward_pose_estimation_gatenet.py \
    --dataname <data_filename_in_data_folder> \
    --n_iters <number_of_training_iterations> \
    --testimgidx <index_of_estimated_image> \
    --checkpoint_path <path_to_gatenet_weights> \
    --input_type <input_dimension_involving_perturbation> \
    --xyz_eps <translation_perturbation_magnitude> \
    --ry_eps <rotation_perturbation_magnitude> \
    --xyz_step <translation_step_per_spatial_dimension> \
    --ry_step <rotation_step_per_rotational_dimension>
```

### Verify object detection on NeRF-based Scenes
To evaluate the robustness of object detection in NeRF-based scenes using AbstractRendering, first run the following command:
```bash
cd ./AbstractNerf
python nerf_forward_yolo.py \
    --datanamne <data_filename_in_data_folder> \
    --n_iters <number_of_training_iterations> \
    --testimgidx <index_of_estimated_image> \
    --input_type <input_dimension_involving_perturbation> \
    --xyz_eps <translation_perturbation_magnitude> \
    --ry_eps <rotation_perturbation_magnitude> \
    --xyz_step <translation_step_per_spatial_dimension> \
    --ry_step <rotation_step_per_rotational_dimension> \
    --manual_focal_factor <manually set focal factor of rendering> \
    --tile_height <image height> \
    --tile_width <image width>
```
This will generate a result folder under [`AbstractNerf/yolo_results`](AbstractNerf/yolo_results/) and save the computed bounds in it. Then, post-process the results with
```bash
python post_process_yolo.py \
    --datanamne <data_filename_in_data_folder> \
    --n_iters <number_of_training_iterations> \
    --testimgidx <index_of_estimated_image> \
    --input_type <input_dimension_involving_perturbation> \
    --xyz_eps <translation_perturbation_magnitude> \
    --ry_eps <rotation_perturbation_magnitude> \
    --xyz_step <translation_step_per_spatial_dimension> \
    --ry_step <rotation_step_per_rotational_dimension> \
    --manual_focal_factor <manually set focal factor of rendering> \
    --tile_height <image height> \
    --tile_width <image width> \
    --conf_threshold <threshold of confidence score>
```
More examples can be found in [`AbstractNerf/yolo_example.sh`](AbstractNerf/yolo_example.sh).
