

This chapter explains the background information on the development of semantic segmentation and Neural Networks, the technical details of deep learning and its application in semantic segmentation tasks. It covers Convolutional Neural Networks (CNNs) and their variations, particularly, the cutting-edge models such as SegNet, U-Net, U-Net++ used for Semantic Segmentation. In addition, metrics such as Dice coefficient, Intersection over Union and pixel accuracy, which are widely used for semantic segmentation, and loss functions are explained.

\section{Artificial Intelligence}
Basic knowledge of artificial intelligence (AI) and in particular machine learning and deep learning is essential to understand and successfully perform a pixel-level component recognition task.

Artificial intelligence (AI) is a branch of computer science that aims to create systems that perform tasks normally carried out by humans, such as pattern recognition, understanding of natural languages, making decisions etc. Variety of techniques are utilized including machine learning and deep learning, developing systems that perform tasks previously thought to be the exclusive domain of humans\cite{chollet2021deep}.

\section{Machine Learning}
Machine Learning (ML) is a branch of Artificial Intelligence (AI) that concentrates on creating algorithms and statistical models that allow machines to learn from data without human intervention. These algorithms allow systems to automatically improve their performance by detecting patterns and connections in the data, making predictions or decisions based on that knowledge. Machine Learning algorithms can be divided into three main categories: supervised, unsupervised, and reinforcement learning\cite{chollet2021deep}. 

In supervised learning, a model is trained using labelled data. The goal of the training process is learning a mapping from inputs to outputs so that when the model receives new, unknown inputs, it can predict the correct output. Supervised learning is typically used in image classification. On the other hand, in unsupervised learning, a dataset of unlabeled examples is given to the model and the goal is learning some basic structure or distribution from the data. Unsupervised learning is widely used in areas such as computer vision where a lot of unlabeled data is available.

\section{Deep Learning}

Deep Learning is a technique within Machine Learning (ML) that employs deep neural networks (DNNs) with multiple layers to learn representations of data, particularly in computer vision applications. These DNNs are able to learn increasingly complex representations of the input data by training on large amounts of labeled data, and they have been able to achieve state-of-the-art performance on a wide range of computer vision tasks such as image classification, object detection and semantic segmentation\cite{chollet2021deep}.

\section{Neural Networks}

Neural Networks are machine learning model based on the structure and function of the human brain. It consists of interconnected processing units known as artificial neurons arranged in layers. These layers process the input data and produce output through a series of mathematical operations. The neurons in the neural network are trained using a large set of labeled data, and the goal is to learn the basic relationships between input and output. An example of layer representation is shown in Figure \ref{cnn_fig}.
Neural networks can be used for a wide variety of tasks such as image classification, natural language processing speech recognition etc. The architecture of neural networks can be divided into feed forward, convolutional and recurrent neural networks, each serving a specific purpose and suitable for different types of tasks\cite{chollet2021deep}.

\begin{figure}[h!]
\includegraphics[width=150mm,scale=0.8]{nn.png} 
\centering
\caption{ Example of deep representations that is learned by a digit classification model\cite{chollet2021deep}. }
\label{cnn_fig}
\end{figure}
Layers in a neural network can be broadly classified into three types as input layers, hidden layers and output layer. The input layers are the first layers of the network and the main functionality is receiving the input data and forwarding them to the next layer. The upcoming layers are called hidden layers and their main functionality is extracting features from the input data. These layers typically consist of several artificial neurons that are connected to each other. They are responsible for the non-linear transformation of the input data. Moreover, these layers are called hidden layers because they are not directly involved with the input or output of the data. The final layer in a network is output layer and their main task is producing the final output. This layer takes the extracted  features from the hidden layers and uses these features to make a decision or prediction. The number of layers and neurons in each layer can be varied and this custom variation is depending on the task and data complexity. The more layers and neurons there are in a model, the more complex it is, and the more data must be used for training.
During the training phase, the network regulates these parameters for minimizing the error that occurred between the predicted output and the ground truth. This is referred to as back-propagation. The training process is being iterated over and over again so that the network learns from the data and improves its state\cite{chollet2021deep}. 

A representation of a neural network is shown in the Figure \ref{pipeline_nn}  that is defined by its weights, and evaluation of a loss metric which serves as a feedback signal to update the network's weights

\begin{figure}[h!]
\includegraphics[width=145mm,scale=0.8]{nn2.png} 
\centering
\caption{An illustration of a neural network that is parameterized with its weights and loss score calculation in order to use as a feedback signal to regulate the weights\cite{chollet2021deep}.}
\label{pipeline_nn}
\end{figure}

A neural network is parameterized by its weights which are used for calculations and generating the predictions. The weights are essentially the values assigned to the connections between the neurons of a network. Weights are adjusted during the process of training in order to optimize the performance of the network.
 The error evaluation between predicted output and the ground truth is referred as the loss score that is used as a feedback mechanism to fine-tune the weights. The loss score is a network measure that indicates how well the network is performing.
The goal of the training is to minimize the loss score which means that the network predictions are as close as possible to the ground truth. The most common loss functions used in neural networks are Cross Entropy Loss and Mean Squared Error. 
The training process initiates by feeding the network with input data. Then the weights of the network are used to make calculations and thus generate predictions. The estimated output (prediction) is then compared with the actual output (ground truth) by a loss function, this comparison provides a loss score. The loss score is then used as a feedback signal and the weights are adjusted, in a way to minimize the loss score, by using an optimization algorithm (such as Stochastic Gradient Descent (SGD) or Adam). This process is being iterated until the network converges in other words reaches a sufficient level of accuracy\cite{chollet2021deep}.



\subsection{ The History of Neural Networks } 

The neural networks history is traced back to the 1940s-1950s when researchers initially considered creating artificial networks of simple elements, or "neurons," to mimic the behavior of biological neurons.
Warren McCulloch and Walter Pitts proposed the first artificial neural network concept in 1943, proposing a simple mathematical model of a biological neuron that could be used to execute simple logical functions\cite{mcculloch1943logical}. 

In the 1950s and 1960s, researchers such as Frank Rosenblatt and Bernard Widrow continued to develop early neural network models, known as perceptrons \cite{rosenblatt1958perceptron}. These models were based on a single layer of artificial neurons and were able to perform simple linear classification tasks. However, they were not able to solve more complex problems due to the limitations of their architecture.

In the 1980s, David Rumelhart introduced the concept of backpropagation, an algorithm for training multi-layer neural networks\cite{rumelhart1986learning}.This allowed for the training of more complex networks, known as multi-layer perceptrons (MLPs), which were capable of solving more complex problems. However, the training of these networks was still difficult and computationally expensive, which limited their practical application.


In 1998, Yann LeCun published a detailed paper on convolutional neural networks, graph transformer networks, and a discriminative training method for sequence labeling, finalizing the training algorithm\cite{726791}. However, it wasn not until 2012, with the advancements in optimized GPUs, that convolutional neural networks saw a significant breakthrough. In that year, a team led by Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton, won the ImageNet Large Scale Visual Recognition Challenge with their model, AlexNet \cite{krizhevsky2017imagenet}. The model used a convolutional neural network and achieved a Top-5 error rate of 15.3 \%, significantly outperforming the next best result of 26.2\%. Ever since the breakthrough in 2012, convolutional neural networks (CNN) have consistently been the leading method in many computer vision tasks and have yet to be surpassed by other techniques.
In the last decade, with the availability of large data-sets and powerful computational resources, neural networks have experienced a resurgence of interest and have been used to achieve state-of-the-art performance in a wide range of applications such as computer vision, natural language processing, speech recognition and generation, among others.

In the course of the history of neural networks, complex and powerful models has been developed as well as new training algorithms. Therefore, wide variety of real-world problems became feasible to solve by neural networks.

\subsection{Convolutional Neural Network (CNN)}

Convolutional Neural Network (CNN) is a sort of neural network that excels at analyzing image and video tasks. CNNs were inspired by animal vision systems and are meant to handle data having a grid-like layout, such as images. CNNs are composed of various layers, these layers are convolutional layers, pooling layers, and fully connected layers. 
Convolutional layers are the core of CNNs that perform convolution operations on the input data. They are used to extract features from the input data, then these features are passed on to the next layers in the network.
Pooling layers are used to reduce the spatial size of the data with the given strategy. It essentially allows the network to focus on the important features while reducing the dimensionality of the data.
 Fully connected layers (also known as dense layers) refers to the layers whose inside neurons connect to every neuron in the preceding layer. They are used to make predictions (or decisions) based on the extracted features of the convolutional and pooling layers.
CNNs can learn hierarchical visual representations. The characteristics learnt may be used to a variety of image analysis tasks, including picture classification, object recognition, and semantic segmentation. CNNs have been utilized to achieve cutting-edge performance in a variety of computer vision applications.

\subsubsection{Convolution Operation}
Convolution is a mathematical procedure that CNNs employ to extract features from incoming data. Convolution is done between input data, such as an image, and a kernel which is a small matrix and is also known as a filter.  The input data is scanned by the filter (kernel). An element-wise multiplication between the filter values and the corresponding input values at each position is performed. After this element-wise multiplication, a summation operation is performed to generate a single output value for that position. This process is repeated for each position of the filter across the input data resulting in a new output matrix called activation map or feature map. A feature map or activation map is a multi-dimensional array that contains the convolution operation output for each position of the filter. A representation of convolution operation is simply shown in the Figure \ref{conv_op_fig}.
Each element in the feature map represents the filter activation or  response to a specific region in the input data. The feature maps are a simplified representation of the input data that emphasizes specific features such as edges, patterns etc.





\begin{figure}[h!]
\includegraphics[width=140mm,scale=0.8]{images_thesis/convopr.png} 
\centering
\caption{An illustration of convolution operation\cite{chollet2021deep}} 
\label{conv_op_fig}
\end{figure}

Multiple features can be extracted from the input data by using multiple filters, each filter responding to different patterns in the input data, resulting in multiple feature maps. These feature maps are then passed through additional CNN layers such as pooling layers and activation layers to extract more abstract features and reduce data dimensionality.

\subsubsection{Convolution Layers}

Convolutional layers are the basic building block of Convolutional Neural Networks and are used to extract features from input data, which are then passed through other layers in the network to make classifications or predictions.

The input data is passed through a series of filters, each of which is a small matrix, in convolutional layers. The input data is scanned by these filters and performing an element-wise multiplication with the corresponding input values in each position. This process is repeated for each filter position across the input data, after all, producing a new output matrix known as a feature map. The important point is that each neuron in the feature map is connected to only a small and localized region of input data that is defined by the size and stride of the kernel. This local connectivity allows the network to extract patterns or features from the input data. This structure is called as translation-invariant: regardless of their position in the input data, the features will be detected. This property is crucial in order to obtain a good performance of CNNs in tasks such as image recognition, where the objects of interest may appear in different positions in a given input image.

\subsubsection{Non-linear Layer and Activation Functions}
A nonlinear layer consists of an activation function that takes the feature map generated by the convolutional layer and generates the activation map by applying the activation function to each element-wise value of the feature map. This layer allows the network to model complex, nonlinear data relationships. It is worth to say that rectified linear unit (ReLU) function is the most widely employed non-linear function and this function converts any negative input values to 0 while leaving positive values unchanged, introducing non-linearity into the network.  Since linear functions cannot model non-linear relationships, this non-linearity is critical for the network to learn complex representations of the data. Typically, Relu is typically used as activation function in the intermediate layers, and sigmoid is typically used activation function in the final layer for binary outputs that outputs a probability (a score between 0 and 1). Representation of ReLu and Sigmoid functions are shown in the Figure. Additionally, softmax is another popular activation function that is used in final layer in multi-class classifications.
\begin{figure}[h!]
\includegraphics[width=120mm,scale=0.8]{images_thesis/relusigmoid.png}
\centering
\caption{ReLu and Sigmoid functions respectively\cite{chollet2021deep}. }
\label{relu_sigm}
\end{figure}

\subsubsection{Pooling Layers}
A pooling layer is a type of layer in Neural Networks which performs down-sampling operations on the incoming data while reducing spatial dimension with the chosen strategy such as maximum pooling and average pooling. Max pooling selects the maximum value in a small window of the input data and outputs it as the new value for that position. In average pooling, the average value of the input data in a small window is calculated and the result outputs as the new value for that position. 
A pooling layer is typically applied after one or more convolutional layers, and the pooling operation is typically performed on the convolutional layer's output feature maps. The pooling operation reduces the size of the feature maps, resulting in a more computationally efficient network.
The primary purpose of the pooling layers is utilizing the network more resistant to small translations in the input. Pooling layers are used to extract the most important features from input data and make the network more robust to minor translations of the objects in the input data.

\subsubsection{Dense Layers}
A dense layer (also known as a fully connected layer) are connected to every neuron in the preceding layer instead of connecting to just a specific subset of neurons as seen in other architectures such as convolutional neural networks. Each neuron receives incoming data from all the neurons in the previous layer and applies a set of parameters (weights and biases) to the input data to produce an output. Then the output is sent to the next layer. Dense layers are commonly used in feed-forward neural networks and are employed for extracting features from the input data.  Dense layers are typically positioned between the input and output layers to extract features from the input data. They can be stacked one on top of the other to enable the extraction of high-level features from the input data.



\subsection{Encoder Decoder Structure}
Encoder-decoder structure is a type of neural network architecture that is typically used in computer vision, particularly in image and video analysis tasks. 

The encoder blocks of the network extracts features from the input image (or video). On the other hand, the decoder blocks of the network uses these features to generate the output image (or video). This architecture is important since it enables the network to learn a compact representation of input data, which can then be used to produce more accurate output. Additionally, It is also used in tasks like image captioning, semantic segmentation, and object detection.


\subsubsection{Encoder}
An encoder architecture usually consists of multiple layers of convolutional neural networks (CNNs). These layers extract increasingly complex features from the data as the input image or video passes through them. Convolutional layers, pooling layers, and normalization layers are common CNN layers in encoders.
By applying a set of filters to the input data, convolutional layers start extracting local features from images, pooling layers are used to reduce the spatial dimensions of feature maps while keeping the most important information intact. Normalization layers, optionally, are used to ensure that the input data is consistent and to mitigate the impact of any outliers.
The final output of an encoder is a compact, fixed-length feature vector representing the input image (or video). 

\subsubsection{Decoder}
Decoder architecture typically is consists of multiple decoder blocks. Each decoder block receives the features from the encoder as fixed-length, compact feature vector that os used to generate the output image (or video) by applying a set of filters to the input feature vector. Transposed convolutional layers, also known as up-convolutional layers, are used to increase the spatial dimensions of feature maps. These layers are the inverse of the encoder's convolutional layers and are used to generate a higher-resolution output image (or video).
    


\section{Semantic Segmentation}
Semantic segmentation is a Computer Vision task that essentially is based on interpretation and understanding the visual information from the world. It entails developing algorithms and methods for assessing photos and videos, and interpreting the scene. The objective of computer vision is building machines that can perceive and understand the visual world like humans and apply this understanding to a wide range of applications, such as autonomous cars, medical imaging, security systems etc.\cite{krig2014computervision}.

Semantic Segmentation is the process of assigning a semantic label, such as "road", "sky", or "car", to each pixel in an image and creating a dense prediction that classifies every pixel in the image and provides a complete representation of its semantic content. Semantic Segmentation tasks differs from other tasks of computer vision, such as Image Classification, Object Detection, Instance Segmentation so on. More specifically, image classification only assigns a single label to the whole image, whereas Object Detection involves not only classification but also the localization of objects within an image and the drawing of bounding boxes around them. Instance Segmentation takes it a step further, as it not only detects objects but also separates individual instances of the same object class. In contrast, the main focus of Semantic Segmentation is to label each pixel in the image with its semantic class, producing a dense label map that provides a more fine-grained representation of the image with respect to the sparse representation of bounding boxes in Object Detection.
Moreover, semantic segmentation task is useful for several applications such as image editing, and scene understanding. Finally, it's worth noting that semantic segmentation is a very active research area and new techniques and architectures are being proposed frequently. Different approaches to the semantic segmentation problem.
\par \textbf{Binary semantic segmentation} aims classifying each pixel in an image as belonging to one of two classes. For instance, in an image of a road scene, the classes could be "road" and "not road". The model would output an image where each pixel is labeled as either one or other.

\par \textbf{Multi-class semantic segmentation} aims to classify each pixel in an image as belonging to one of multiple classes which provides to do pixel-wise labelling of multiple objects by using only one model. For example, in an image of a road scene, the classes could be "road", "vehicle", "pedestrian", and "sky". The model would output an image where each pixel is labeled with one of the classes. Multi-class semantic segmentation is relatively more challenging than binary semantic segmentation because it requires the model to distinguish among different classes, whereas binary semantic segmentation only requires the model to distinguish between two classes. Selecting the one of these two strategies are depend on the need of the specific problem.


\subsection{Comparisons of different computer vision tasks}


\begin{figure}[h!]
\includegraphics[width=150mm,scale=0.8]{images_thesis/compvis.png} 
\centering
\caption{An illustration of several computer vision tasks \cite{kirillov2019panoptic}
\label{compVis}
\cite{hao2020brief} }
\end{figure}

The goal of image classification is to assign a whole image to one or more category labels. In other words, an algorithm for image classification identifies the objects in a picture. 
The objective of Object Detection is to identify objects in an image and determine their location. This computer vision task involves both object classification and localization. The system outputs a collection of bounding boxes that surround the objects found in the image and provide their corresponding class labels that is shown in the Figure \ref{compVis} (d). Semantic segmentation, on the other hand, has higher requirements because it aims to precisely separate each object region from the background region.

An example is given in  Figure \ref{compVis}, showing that a segmentation algorithm is able to not only find all the desired objects, but also accurately define their boundaries. It is mentioned that semantic segmentation is more difficult than the previous tasks because it involves connecting low-level features with high-level meaning.
Two advanced tasks, instance segmentation and panoptic segmentation, have become the focus of new research. Instance segmentation aims to identify each individual object within an image, as shown in  Figure \ref{compVis} (e) where cars are labeled with different labels representing each instance. Panoptic segmentation goes one step further by assigning both a semantic label and an instance label to every pixel. These tasks are more demanding than traditional semantic segmentation as they focus on identifying specific instances of objects and not just general class labels\cite{hao2020brief}.

\subsection{Traditional Computer Vision-based Methods for Semantic Segmentation}

Before the advent of deep neural networks, semantic segmentation was primarily approached using traditional computer vision techniques such as graph-based methods and region-based methods. 

Graph-based methods involve representing the image as a graph, with each pixel or super-pixel as a node and edges between neighboring pixels or super-pixels. The purpose is to divide the graph into segments (or regions), each of which corresponds to a particular object or area of the picture.
Region-based methods, on the other hand, involve grouping pixels or super-pixels that are similar in terms of color, texture, or other features. 
Edge-based image segmentation divides an image into areas or objects by recognizing edges or boundaries in the picture. This method is based on the assumption that picture edges or borders represent the transition between various regions or objects. They serve as the foundation for dividing a picture into separate sections or items.
Pixel-based image segmentation divides a picture into various areas or objects by evaluating the attributes of each pixel and it assumes that pixels of image that have similar color, texture etc. correspond to the same region or object\cite{graph_region_based_ss}. \newline

These methods are less precise than deep learning-based solutions and are easily influenced by changes in lighting, perspective, and occlusions, among other difficulties.
\par Figure \ref{trad_ss} illustrates the application of the mentioned methods to a sample image.
\begin{figure}[h!]
\includegraphics[width=145mm,scale=0.8]{images_thesis/traditional_ss_methods.png} 
\centering
\caption{Representation of pixel-based, edge-based, graph-based image segmentation \cite{graph_region_based_ss}}
\label{trad_ss}
\end{figure}
In addition, these traditional methods are not able to handle large variations in object shape, size, pose and viewpoint, which makes them less robust.


Overall, traditional computer vision-based methods for semantic segmentation were less accurate and less robust than deep learning-based methods, and they were not able to handle large variations in object shape, size, pose and viewpoint.

\subsection{Deep Learning-based Methods for Semantic Segmentation}

Deep learning-based methods for semantic segmentation have become the state-of-the-art in recent years that are based on convolutional neural networks (CNNs). Most popular deep learning-based application for semantic segmentation, including: 
Fully convolutional networks (FCNs) are a sort of CNN that can accept any size input picture and create an output segmentation map of the same size. FCNs may leverage the spatial information in an image by extracting features at different scales using a sequence of convolutional and pooling layers.\newline

U-Net is a form of FCN that can accurately segment small structures in pictures, making it particularly well-suited for biomedical image segmentation applications. U-Net is built on an encoder-decoder architecture, with the encoder extracting features from the picture and the decoder reconstructing the segmentation map from the features.\newline


These approaches outperformed various benchmarks and have been employed in a variety of applications, including self-driving automobiles, robot navigation, picture editing, medical imaging, and scene interpretation, among others.
It's worth mentioning that these systems are always evolving and improving, with new methodologies and architectures being offered on a regular basis.



\subsection{FCN for Semantic Segmentation}
Fully Convolutional Networks (FCNs), a version of CNNs, were a significant advancement in the area of image segmentation. The Fully Convolutional Network (FCN) for Semantic Segmentation was the first model that adapted and fine-tuned classification networks for direct, dense prediction of semantic segmentation\cite{long2015fully}. A Fully Convolutional Neural Network(FCN) is represented in Figure \ref{fcn_fig}.

\begin{figure}[h!]
\includegraphics[width=145mm,scale=0.8]{FCN.png} 
\centering
\caption{ A Fully Convolutional Neural Network (FCN) make dense predictions for per-pixel tasks, allowing the network to take in inputs of arbitrary size and produce output with corresponding spatial dimensions\cite{long2015fully}. }
\label{fcn_fig}
\end{figure}

A Fully Convolutional Network (FCN) is a sort of convolutional neural network (CNN) that is intended to make dense predictions across a whole picture rather than identifying a particular patch or portion of an image. This is accomplished by replacing the fully connected layers at the end of a CNN with convolutional layers, which enables the network to accept inputs of any size and provide output with commensurate spatial dimensions.
In other words, a Fully Convolutional Network makes dense predictions over an image while processing the entire image at once, rather than processing small regions of the image at a time. This allows the network to consider the context of the entire image when making predictions, which is beneficial for tasks such as object detection and semantic segmentation. \newline



In traditional classification networks, an input image is downsized and processed through convolution layers and fully connected layers to produce a predicted label. But by replacing the fully connected layers with 1x1 convolutional layers and not downsize the image, the output becomes a feature map with a smaller resolution than the input image, due to max pooling. By upsampling the output to the original resolution, a pixel-wise label map is obtained.  \newline

\begin{figure}[h!]
\includegraphics[width=145mm,scale=0.8]{fcnStride.png} 
\centering
\caption{ Combining information from layers with different strides in a fully convolutional network and observation of enhanced accuracy of segmentation\cite{long2015fully}. }
\label{fcn_strides}
\end{figure}

In addition, a Fully Convolutional Network (FCN) has the effect of strides on the convolutional layers in terms of reducing the spatial resolution of the feature maps. This is done by skipping over some of the elements in the input feature maps during the convolution operation. Larger strides result in a larger reduction of spatial resolution, while smaller strides result in a smaller reduction. A representation is shown in Figure \ref{fcn_strides}. This can be useful for reducing the computational cost of the network and preventing overfitting, but it also means that the network may lose some fine-grained information about the input.


\subsection{SegNet}

SegNet, developed by the University of Cambridge, \cite{badrinarayanan2017segnet} was initially submitted to the 2015 CVPR conference but was not officially published until 2017 in the TPAMI journal.  \newline 
\par SegNet is a convolutional neural network architecture designed for semantic image segmentation tasks. It is an encoder-decoder type of architecture where the encoder is used to extract features from the input image and the decoder is used to make predictions based on these features.

It consists of an encoder network and a corresponding decoder network, followed by a final pixel-wise classification layer as shown in Figure \ref{segnet_arc_fig}. The encoder is made up of 13 convolutional layers, which are based on the VGG16 network\cite{simonyan2014very}. VGG16 network is a pre-trained network that has been trained on a large dataset of images. The encoder extracts features from the input image and compresses them into a lower-dimensional feature space. The decoder is an up-sampling network that expands the compressed features back to the original resolution of the input image. Each layer in the encoder has a corresponding layer in the decoder, resulting in a total of 13 layers in the decoder network and the final output from the decoder is then fed into a multi-class soft-max classifier to produce class probabilities for each pixel independently.






\begin{figure}[h!]
\includegraphics[width=145mm,scale=0.8]{segNet.png} 
\centering
\caption{ Illustration of SegNet architecture \cite{badrinarayanan2017segnet} }
\label{segnet_arc_fig}
\end{figure}









The encoder network in SegNet uses a filter bank to produce a set of feature maps. These feature maps are batch-normalized and then an element-wise rectified linear non-linearity (ReLU) is applied. After that, max-pooling with a 2x2 window and a stride of 2 (non-overlapping window) is performed to reduce the resolution of the feature maps by a factor of 2. This is done to achieve translation invariance over small spatial shifts in the input image. However, as a result, the spatial resolution of the feature maps is reduced. This is not ideal for segmentation, where boundary delineation is important. To address this, the authors of SegNet capture and save boundary information in the encoder feature maps before subsampling is performed. However, due to memory limitations, the stored information was reduced to only max-pooling indices, i.e., the locations of the maximum feature value in each pooling window is recorded for each encoder feature map.\newline

The decoder network is in charge of upsampling the encoder network feature maps to the original picture resolution and constructing the final segmentation mask.

SegNet decoder network employs a reverse design of the encoder network, with pooling indices from the corresponding encoder layer utilized to accomplish non-linear up-sampling of feature maps. This permits the decoder network to preserve spatial information that was lost during the encoder network's down-sampling operation.
During upsampling, the feature maps are processed through a succession of convolutional and non-linear activation layers to generate the final segmentation mask. SegNet decoder network is meant to reconstruct the spatial resolution of the original image while keeping the encoder high-level semantic information of the network .


The performance of SegNet was evaluated on two scene segmentation benchmarks. The first task was road scene segmentation with CamVid dataset \cite{brostow2009semantic} which has practical applications in various autonomous driving-related problems. The second task was indoor scene segmentation with SUN RGB-D dataset \cite{song2015sun} which is relevant to several augmented reality tasks. The model performance was compared to several other widely adopted deep architectures for segmentation such as FCN, DeepLab, and DeconvNet. All the architectures were trained with the same hyperparameters and output the same resolution output feature maps. According to the results of the original paper, SegNet outperforms all other architectures in all metrics (Class average, Global average, mean Intersection over Union, and Boundary F1 measure) on the CamVid dataset. The experimental results on the SUN RGB-D dataset reveal some interesting points. All the deep architectures have very low mean IoU and boundary metrics, as well as small global and class averages. In terms of mIoU, SegNet outperformed FCN and DeconvNet but had a slightly lower mIoU than DeepLab-LargeFOV, which is probably due to the CRF post-processing used in the DeepLab model. The authors attribute the overall poor performance to a large number of classes in this segmentation task, many of which occupy a small part of the image and appear infrequently.

\subsection{U-Net}
The U-Net is a modified fully convolutional network (FCN) that is mostly used in semantic segmentation tasks. Ronneberger and Fischer originally identified the U-Net in 2015, and it was especially built for image segmentation and localization in biomedical applications\cite{ronneberger2015u}. \newline

U-Nets have an advantage over regular CNNs because U-Net can both identify where an object is in an image and what it is. Thus, it provides both localization and classification. In this context, localization means that each pixel in an image is assigned a corresponding class label.

U-Nets have an advantage over Fully Convolutional Networks (FCNs) because they can produce accurate segmentation even with a limited number of training images. This is achieved by using up-sampling layers with a large number of feature channels, which helps conveying more contextual information to the higher resolution layers.

The U-Net segmentation network is the one of the best choice for this thesis application because it is a scalable and flexible Fully Convolutional Network (FCN) that does not compromise on localization and context information. 


\subsubsection{Network Architecture}
It has a specific structure that includes a contracting path on the left and an expansive path on the right, which is essentially an encoder-decoder architecture. The contracting path follows a typical convolutional network format, including repeated use of 3x3 convolutions, ReLU activation, and 2x2 max pooling with a stride of 2. 
The network also contains skip connections that is in between the encoder and decoder. Skip connections allow the decoder to learn  the high-level features from the encoder, and allows the network to retain spatial information (height and width) throughout the network. More specifically, skip connections contribute the decoder blocks with the intermediate encoder activations that is composed of high resolution details.

At each downsampling step, the number of feature channels is doubled. Each step in the expansive path includes an up-sampling of the feature map, followed by a 2x2 convolution that reduces the number of feature channels by half. This is combined with the correspondingly cropped feature map from the contracting path, and two 3x3 convolutions with a ReLU activation function are applied. The cropping is necessary due to the loss of border pixels during each convolution. The final layer uses a 1x1 convolution to map each component feature vector to the desired number of classes. The network architecture is illustrated in the Figure \ref{label_unet}.


\begin{figure}[h!]
\includegraphics[width=150mm,scale=0.8]{unet.png} 
\centering
\caption{ Illustration of U-net architecture\cite{ronneberger2015u}. }
\label{label_unet}
\end{figure}

The U-Net model is able to capture the overall context of an image by using a contracting path, while at the same time it allows for precise localization by using an expanding path that is symmetric to the contracting path. The high resolution features from the contracting path are combined with the upsampled output to enable localization. Additionally, the upsampling section of the model has a large number of feature channels which helps the network to propagate context information to higher resolution layers.

\subsection{U-Net ++}

UNet++\cite{zhou2018unet++} is a type of deeply-supervised encoder-decoder network that utilizes an encoder-decoder structure with multiple layers of nested and dense connections, called skip pathways, that link the encoder and decoder sub-networks. These pathways are designed specifically to minimize the difference between the feature maps generated by the encoder and decoder, thus improving the final output of the segmentation. The experiments that is done in "UNet++: A Nested U-Net Architecture for Medical Image Segmentation" study , the suggested architecture provided an important performance gain compared to standard U-Net model on medical images.
UNet++ uses a hierarchical architecture, where multiple levels of encoders-decoders are stacked on top of each other to extract features at different scales. Each level of the encoder extracts features at a different resolution, and each level of the decoder uses these features to make predictions at a corresponding resolution. This allows the network to incorporate both low-level and high-level features, which improves the accuracy of the segmentation. 

The network also uses a skip connection between the encoder and decoder, which allows the network to combine features from different levels of the encoder. This helps to preserve fine details in the segmentation while still incorporating contextual information from the encoder.
Additionally, UNet++ uses attention gates to selectively propagate information from the encoder to the decoder, which helps to reduce noise and improve the accuracy of the segmentation.

\begin{figure}[h!]
\includegraphics[width=150mm,scale=0.8]{unetPP.png} 
\centering
\caption{ High-level overview of Unet++\cite{zhou2018unet++} }
\label{unetPP}
\end{figure}

With respect to the Figure \ref{unetPP}: (a) UNet++ is an architecture that uses an encoder and a decoder that are connected through a series of nested dense convolutional blocks. The aim of UNet++ is to reduce the semantic gap between the feature maps generated by the encoder and the decoder before they are combined. As seen in the graphical abstract, UNet++ is composed of the original U-Net represented by black blocks, the dense convolution blocks on the skip pathways which are represented by green and blue blocks, and deep supervision represented by red blocks. These red, green, and blue components distinguish UNet++ from U-Net.
\par (b) Analysis of the first skip pathway detailed for UNet++. 
\par (c) UNet++ can be more efficient during the inference phase if it is trained with deep supervision, by removing less important parts of the network..

Overall, UNet++ is an elaborated version of the UNet architecture that addresses some of its limitations and in some cases it allows to achieve better performance in image segmentation tasks.
\subsubsection{Difference Between UNet and UNet++}
The primary distinction between UNet and UNet++ is in their design and approach to feature extraction and feature fusion. UNet employs a single encoder-decoder structure, whereas UNet++ employs a hierarchical design with many encoder and decoder layers. UNet employs the same set of filters at all scales, whereas UNet++ employs multiple sets of filters at various sizes.

UNet combines features from several layers of the encoder, whereas UNet++ selectively propagates information from the encoder to the decoder via skip connections and it is designed to extract more contextual and fine details information than UNet. It is designed to reduce the semantic gap between the feature maps of the encoder and decoder sub-networks, while UNet uses skip connections to preserve fine details in the segmentation.
The re-designed skip pathways also connect the use of deep supervision which gives possibility to network to work in two different modes. First mode is called as accurate mode that averages outputs from all segmentation branches. The second mode is called the fast mode and provides the final segmentation map selected from only one of the segmentation branches. The mode is selected according to the need.



\section{Evaluation Metrics}
When assessing the performance of various semantic segmentation network architectures, it is necessary to use an convenient metric to precisely describe the network capability of identifying a class.
In this study three different metrics are employed to evaluate model performance that are widely used in semantic segmentation. These are mean Intersection over union, dice coefficient and pixel accuracy.

\begin{figure}[h!]
\includegraphics[width=145mm,scale=0.8]{metrics.png} 
\centering
\caption{Auxiliary image to define metrics.}
\label{metric}
\end{figure}


\subsubsection{mIoU}
mIoU stands for "mean Intersection over Union" which is a commonly used metric in computer vision, particularly for semantic segmentation tasks where the evaluation is based on the overlapping between the ground truth image and the predicted image. mIoU is in fact, used to evaluate the degree of overlap between the predicted segmentation map produced by a model and the ground truth segmentation map. It is calculated by taking the overlap of the predicted and target masks and dividing it by the union of the two masks and taking the average of this value of all classes.

Regarding the image shown in Figure \ref{metric}, it highlights three different regions, namely FN, TP, and FP, which correspond to False Negative, True Positive, and False Positive, respectively:\newline 

$$ IoU = \frac{\text{TP}}{\text{TP} + \text{FP} + \text{FN}} $$ 
This is the formulation of Intersection over Union and mean of IoU is: 
$$mIoU = \frac{1}{N} \sum_{i=1}^{N} \frac{\text{TP}_i}{\text{TP}_i + \text{FP}_i + \text{FN}_i}$$
where $N$ is the number of classes and mIoU is the average Intersection over Union for all classes. 

mIoU is a scalar value between 0 and 1, with 1 representing a perfect match and 0 representing no overlap.

It is used to assess the model's overall performance by considering the performance of all classes, not just the most common ones.

\subsubsection{Dice Coefficient}
The Dice coefficient is used to compare the overlap between two image (ground truth and the prediction). It provides a scalar value between 0 and 1. Value 1 indicates identical images and 0 indicates no elements in common. Therefore, values closer to 1 means a better match. \newline
With respect to the image in Figure \ref{metric} Dice Coefficient: \newline 

$$DC = \frac{2 \cdot \text{TP}}{2 \cdot \text{TP} + \text{FP} + \text{FN}}$$


 It is calculated by taking twice the intersection of the two masks and dividing it by the total number of pixels in both masks. 

\subsubsection{Pixel accuracy}
Pixel accuracy is a widely used metric for evaluating the performance of image classification and semantic segmentation models and is defined as the ratio of correctly classified pixels in an image to the total pixels.
It provides a scalar value between 0 and 1. While value 1 is representing a absolute match, value 0 is representing zero-match (no match at all). Therefore, this metric only considers whether pixels are correctly classified or not However, it does not take class imbalance into account.\newline 
With respect to the image in Figure \ref{metric}: \newline 

$$PA = \frac{\text{TP} + \text{TN}}{\text{TP} + \text{TN} + \text{FP} + \text{FN}}$$


