---
title: "Reproducibility Package for 'A Data Quality Approach to the Identification of Discrimination Risk in Automated Decision Making Systems' _ RQ2 analysis"
author: "Antonio Vetrò, Marco Torchiano, Mariachiara Mecati"
date: 2021-04-12  
output:
 html_document: default
 output_dir: '../results/'
---

This documents contains the reproducibility scripts for the paper

**A Data Quality Approach to the Identification of Discrimination Risk in Automated Decision Making Systems**

by Mariachiara Mecati, Antonio Vetrò, Marco Torchiano

submitted/accepted for publication at Government Information Quarterly.

The goal of our study is to anticipate possible discrimination risks on the basis of the balance features of the training data employed by ADM systems. To this end we formulated two research questions that will drive our investigation:

RQ 1. _How are existing measures able to detect imbalance among the classes of a given attribute in a dataset?_

RQ 2. _Are existing measures able to reveal a discrimination risk when an ADM system is trained with such data?_

In this script we present our code to answer the second Research Question by analyzing data obtained from the investigation in the script Risks_Datasets.Rmd and summarized in unfair-balance.xlsx .

```{r Setup, include=FALSE}
library(ggplot2)
library(tidyverse)
library(ggrepel)
library(latex2exp)
library(readODS)
knitr::opts_chunk$set(echo = FALSE)
```

```{r load, include=FALSE}

#run = 'run1'
base_paper = 'all attributes measures/'  # base paper measures/  OR 'all attributes' (for no base paper)
model = 'lr'
cardinality = 8 #2,4,5,8

run_correlation <- function(run) { 
imbalance_measures <- readODS::read_ods(paste(base_paper,"balance measures.ods", sep=''), run) %>%
        slice(10:38) %>% # 29 or 38 (38 -> without Compas, juvenile, community and Age category)
        filter(Attribute != 'Age category' & Attribute != 'Age cateogry') %>%
        filter(m==cardinality) 
#%>% select("Gini","Shannon","Simpson", "IIR",
#            
#                "Renyi_2", "Renyi_4", "Renyi_6",
#                "Renyi_8", "Renyi_10", "Renyi_12",
#                "Renyi_14", "Renyi_16", "Renyi_18",
#                "Renyi_20", "Renyi_22", "Renyi_24",
#                "Renyi_50", "Renyi_100","Renyi_200",
#                "Renyi_500",
#            
#                "Hill_3", "Hill_5", "Hill_7",
#                "Hill_9", "Hill_11", "Hill_13",
#                "Hill_15", "Hill_17", "Hill_19",
#                "Hill_21", "Hill_23", "Hill_25",
#                "Hill_50", "Hill_100", "Hill_200",
#                "Hill_500")

# or 'measures smote'                                      # ATTENTION HERE
unfairness_measures <- readODS::read_ods(paste(base_paper,"measures/unfairness measures ", model, ".ods", sep=''),run) %>%
        slice(10:38) %>%
        filter(Attribute != 'Age category' & Attribute != 'Age cateogry') %>% 
        filter(m==cardinality) %>% 
        select("Diff Ind" ,"Diff TPR","Diff FPR", "Diff PPV","Diff NPV", "Diff OAE")

correlation_matrix = round(cor(unfairness_measures,imbalance_measures,method="spearman"), 2)
return(correlation_matrix)
}

```


```{r }
run1 = run_correlation('run1')
run2 = run_correlation('run2')
run3 = run_correlation('run3')
run4 = run_correlation('run4')
run5 = run_correlation('run5')
```


```{r }
round((run1 + run2 + run3 + run4 + run5)/5, 2) %>%
knitr::kable()
```

```{r box-plot}
imbalance_measures <- readODS::read_ods(paste(base_paper,"balance measures.ods", sep=''), 'run2') %>%
        slice(10:38) %>% # 29 or 38 (38 -> without Compas, juvenile, community and Age category)
        filter(Attribute != 'Age category' & Attribute != 'Age cateogry')
```