


    This chapter helds the techniques used to perform semantic segmentation on mesh and wire. First, the data-set is analyzed, and the necessary pre-processing steps to prepare the data and run the models effectively will be explained. Then, different model structures are utilized, along with the metrics and loss functions used to refine the models for comparison. 
    The following section covers the procedure to evaluate the performance of the final models on test data and determine the overall accuracy of the models. Finally, grid search is employed to find the optimal parameters for the employed models.


\section{Data processing for Semantic Segmentation}

    Pre-processing stands for employed techniques that assures the data used in training and evaluation of the model is suitable for the given task. The aim of this step is improving to quality of data and removing unwanted features and doing feature extraction from raw data and so improving the reliability of the model.    
    The semantic segmentation pre-process typically involves collecting images and annotations to assign pixel-level labels, cleaning data and converting it into a suitable format to use in the model. Optionally, augmenting data to increase diversity and possibly avoid overfitting, and splitting data as training, validation, and test sets (typically). These steps are essential to prepare the data to feed the deep models so that can be effectively utilized by semantic segmentation models, leading to more accurate and reliable results. 


\subsection{Data collection and labeling}

    Data collection contains acquiring images and annotations for each class of interest. Approximately 500 high-resolution images with dimension of 4928 x 3264 were captured at three different locations in Sicily in order to construct the dataset. These images are representing rocky walls covered by meshes and, in many cases, also contain spikes and wires.

    The annotations define the region of interest of the images belong to a particular class. They can be done manually by some devolopment tools that have a user interface for manually annotating image data. In this thesis, the labeling process is carried out using a Python library named \emph{LabelMe}\cite{wada2018labelme}. For each image in the dataset, the masks corresponding to wires and meshes are created using polygons, thus the data that is suitable for semantic segmentation is obtained. LabelMe is an open-source image annotation tool and it is designed for image segmentation and object recognition tasks. Users can manually label and annotate objects in images with this tool. Creating segmentation masks and annotated data can be used for model training such as semantic segmentation and object detection models. Moreover, LabelMe is easy to use due to its graphical interface for annotating images, which makes it a widely adopted tool for preparing data in computer vision tasks. \\ \\ Figure \ref{labelme1} shows an example of the LabelMe interface used for annotation and the labelMe interface.\\ Figure \ref{labelme2} shows the corresponding mask of an image that is annotated by labelMe.

    \begin{figure}[H]
    \includegraphics[width=150mm,scale=0.99]{labelme.jpg} 
    \centering
    \caption{ An example of how semantic segmentation annotations done by LabelMe\cite{wada2018labelme}. }
    \label{labelme1}
    \end{figure}



    \begin{figure}[H]
    \includegraphics[width=150mm,scale=0.99]{images_thesis/labelme2.png} 
    \centering
    \caption{ Semantic segmentation mask creation is demonstrated with its corresponding image after annotations done by LabelMe\cite{wada2018labelme}. }
    \label{labelme2}
    \end{figure}

    
    In this study, only wire and mesh are considered since they are more suitable for semantic segmentation application. Figure \ref{annotation_show} displays an image with annotations to demonstrate the process of annotation.


    \begin{figure}[H]
    \includegraphics[width=150mm,scale=0.99]{annotation_demo.png} 
    \centering
    \caption{ A demonstration of the labeling process performed using the LabelMe\cite{wada2018labelme} annotation tool is given. Original image and annotated version is shown respectively. The yellow boxes indicate spikes(that are not considered in this thesis), the green polygons represent wires, and the red polygons are for mesh. }
    \label{annotation_show}
    \end{figure}

\subsection{Image Pre-processing}
    Pre-processing includes cleaning and transforming the data so that it can be fed into model and this can involve image resizing, pixel values normalization,encoding class labels to numerical values etc.
    In this thesis, acquired images are captured in both horizontal and vertical orientations. Since the models used require a fixed input size, three cuts for each horizontal image (left, middle, right) and two cuts for each vertical image (bottom, middle) are taken to create dataset. The vertical orientation was limited to two crops because the top portions were often too distant from the point the pic was taken (likely because the images were taken from the ground) and resulting in unclear visibility of the elements in image. This cropping operation is shown in the Figure \ref{image_cut_oriantation}. These cropped images are square in shape with a size of 3264x3264 pixels. The mesh and wire images are then resized as 512x512.


    \begin{figure}[H]
        \includegraphics[width=150mm,scale=0.99]{hor_oriantation.jpg} 
        \centering
        \caption{ A demonstration is provided of the horizontal and vertical cuts applied to the input images. It shows the vertical cuts identified as `center' and `bottom' and the horizontal cuts identified as `center', `left', and `right'.}
        \label{image_cut_oriantation}
    \end{figure}

    

\subsection{Data Splitting}
    At the beginning, there were 451 images (horizontal and vertical), these images are composed of thee different sites of a region, in this thesis these sites are named as site 1 (154 images), site 2 (111 images) and site 3 (186 images). Then, these images are cropped, from these pieces two different data-set are obtained for wire and mesh separately.
    Mesh data-set contains 1137 images and wire data-set contains 1036 images because the images that does not contain any piece of wire and mesh are discarded. Therefore the number of images are different for mesh and wire.
    The Figure \ref{split_num_data} shows the process of how the mesh and wire data-set obtained.
    
    
    Splitting data-set involves dividing the data as training, validation, and test sets (typically). The training set is to train the model, the validation set is for the model selection and lastly, the test set is used for evaluating the performance of the model. 

        \begin{figure}[H]
        \includegraphics[width=150mm,scale=0.99]{SPLIT-DATA-NUM.png} 
        \centering
        \caption{ Data-set creation step by step and indication of the number of images of each data-set.  }
        \label{split_num_data}
    \end{figure}
    
    
    For the following experiments the data for mesh and wire are evenly split into three sets: 75\% for training, 15\% for validation, and 10\% for testing. The images were divided in such a way that the cuts of the same image were not placed in the same set and were distributed fairly among the sites. 
    At the end for mesh data-set: training set contains 854 images, validation set has 163 and test set has 120 images.
    For wire set: training set contains 772 images, validation set has 152 and test set has 112 images.

\subsection{Data Augmentation}
    Data augmentation involves typically generating additional data to add training set by applying transformations to the existing images. This transformation may include flipping, rotation, scaling, adding random noise into images etc. Data augmentation most possibly help preventing overfitting and increases the diversity of the training data.
    
    Data augmentation is employed to assess its impact on accuracy, and the same method used for both wire and mesh. For each image in the training and validation sets, a modified version was created by randomly altering lighting and with a specified probability, adding noise and flipping the image horizontally. This doubled the number of examples in the training and validation sets.
    The Figure \ref{aug_sample} shows an example of data augmentation that is employed for the dataset.

    \begin{figure}[H]
        \includegraphics[width=145mm,scale=0.99]{aug.png} 
        \centering
        \caption{An example of the original image (left) and the augmented image (right).}
        \label{aug_sample}
    \end{figure}


    The Figure \ref{split_num_data_aug} shows the process of how the mesh and wire data-set with data augmentation employed are obtained.


    \begin{figure}[H]
        \includegraphics[width=150mm,scale=0.99]{SPLIT-DATA-NUM-AUG.png} 
        \centering
        \caption{ Data-set creation with data augmentation step by step and indication of the number of images of each data-set.  }
        \label{split_num_data_aug}
    \end{figure}

    After data augmentation employed on validation and training sets, two different data set is created for mesh and wire separately. At the end for mesh, training set contains 1708 images, validation set has 326 and test set has 120 images.
    For wire set: training set contains 1544 images, validation set has 304 and test set has 112 images.








    

\section{Binary Semantic Segmentation with U-Net }

    In this thesis, separate U-Net models are used to conduct further research on semantic segmentation of wire images and mesh images using a neural network.

    Standard U-Net model that was proposed in the original paper \cite{ronneberger2015u} is taken as base model. The proposed model is starting with 64 filters in the first down-sampling block and in each block it doubles the filter size until bottleneck layer. After bottleneck layer, number of filters are halved in each layer until it gets a symmetric correspondence of expansive path.
    Standard U-Net model is composed of two paths: contracting path and expansive path. The down-sampling (contracting) consists of 4 blocks that contain 2 convolutional layers followed by max-pool layer. 
    After the contracting path, there is a bottleneck layer and it is followed by the expansive path (up-sampling blocks). In the expansive path, the feature maps are upsampled by transposed convolutions. Then, the up-sampled feature maps are concatenated with the corresponding feature maps from the contracting path to preserve the spatial details. The final part of expansive path contains 3 convolutional layers where the last layer is an activation function acting on each pixel. As loss function binary cross entropy and for optimizer Adam is employed. \\

    In this experimental setup, sigmoid is chosen as activation function in the output layer. Thus, each pixel of the output is a value between 0 and 1. In the prediction phase, values lower than 0.5 are set to 0 and those higher are set to 1. 

    The model takes as input an image of size (512,512,3) with target the mask of size (512,512,1). Since the problem is binary segmentation mask has one color channel. The standard U-Net that is proposed in the paper written by Ronneberger\cite{ronneberger2015u} has dimension 572x572. Additionally there are cropping layer to build a robust network because the spacial dimension varies in each up-sampling down-sampling blocks and if the dimension of the images are not power of two the output image that is obtained at the end does not give the same spacial dimension. In our experiment to prevent this issue 512x512 images are employed and cropping layers (crop2D) are removed. Output returns the predicted mask with the same size as the target. Image values are normalized to [0,1].

    As evaluation metrics, mean intersection over union (IoU), Dice Coefficient and pixel accuracy is indicated. However selecting the best model will be only considered on mIoU. Since it gives the overlapping rate of the ground truth label and the prediction label (also dice coefficient works in a similar way), it is the most widely used and reliable evaluation metric. It is worth to mention that, pixel accuracy may be miss-leading since the amount of background in an image may be unbalanced and mostly much higher compared to wire and mesh. For instance if the amount of background pixels in an image are higher than mesh/wire amount, and the model predicts that all the image is composed of only background then pixel accuracy would give high result that is not reliable. That is why pixel accuracy only for mesh/wire is considered that gives that accurately predicted pixels only for mesh/wire which provides a robust evaluation.

    This experiment utilized a data augmentation employed data-set of images, rotation, flipping,adding random noise and saturation adjustment are applied randomly to images too obtain new data-set with data augmentation.




    \begin{figure}[H]
        \includegraphics[width=130mm,scale=0.99]{images_thesis/wire_standard_unet_images.png} 
        \centering
        \caption{Samples of wire prediction that is trained by U-Net model with and without data augmentation employed wire images. Input image in the left, prediction in the center and target mask in the right.}
        \label{wire_aug_vs_no}
    \end{figure}

    \begin{figure}[H]
        \includegraphics[width=140mm,scale=0.99]{images_thesis/mesh_standard_unet_images.png} 
        \centering
        \caption{Samples of mesh prediction that is trained by U-Net model with and without data augmentation employed mesh images. Input image in the left, prediction in the center and target mask in the right.}
        \label{mesh_aug_vs_no}
    \end{figure}

    \begin{table}[H]
        \centering
        \resizebox{\textwidth}{!}{%
        \begin{tabular}{
        >{\columncolor[HTML]{DAE8FC}}l l
        >{\columncolor[HTML]{DAE8FC}}l l
        >{\columncolor[HTML]{DAE8FC}}l }
        \cellcolor[HTML]{FFFFFF} &
          \cellcolor[HTML]{DAE8FC}Wire &
         Wire with augmentation &
         \cellcolor[HTML]{DAE8FC}Mesh  &
         Mesh with augmentation \\
        Mean IoU                           & 51.1\% & \textbf{52.1\%} & 87.2\% & \textbf{87.8\%} \\
        Dice coeff.                        & 67.7\% & 68.5\%                        & 93.1\% & 93.5\%                        \\
        Pixel accuracy                     & 96.9\% & 96.8\%                        & 91.7\% & 92.1\%                        \\
        Pixel accuracy wire or mesh   only & 60.4\% & 63.3\%                        & 90.0\% & 94.4\%                       
        \end{tabular}%


        }
     \caption{standard U-Net results on test set of mesh and wire images separately for mesh and wire models.}

     \label{table_1}
    \end{table}

    The obtained results on test set of standard U-Net model trained on wire and mesh images, with and without data augmentation employed are reported in Table \ref{table_1}. Model performance on a sample composition of original image, prediction and ground truth mask is shown in the Figure \ref{wire_aug_vs_no} for wire images with and without data augmentation employed. In the Figure \ref{mesh_aug_vs_no} for mesh images with and without data augmentation employed is demonstrated. \\

   

     In conclusion, this binary semantic segmentation experiment showed that data augmentation improves performance of the model and the results are reasonably good. Although evaluation results are high the prediction sample images are noisy. In order to reduce this noisy view, model improvement is crucial. A dropout layer may help model to be less complex and regularize it. 
     Dropout is a regularization technique that is widely used in deep neural Networks. For  reducing the neurons co-adaptation effect, dropout layers are working in a way that it randomly removes a specified ratio of neurons so that remaining neurons can learn more robust features in a neural network during the training procedure. Thus this technique is used for reducing the effect of overfitting the training data and to obtain less complex models.
     
     In the following experiment, the same data-sets will be exploited with dropout layer added standard U-Net model. 

      \begin{figure}[H]
        \includegraphics[width=130mm,scale=0.99]{images_thesis/loss_std_unet_mesh.png} 
        \centering
        \caption{Training and validation loss behavior during the training on mesh images.}
        \label{losses_mesh}
    \end{figure}

      For observation purpose the validation loss - training loss plot will be compared after employing dropout layers, previous model behaviour is visualized as a graph in the figure \ref{losses_mesh} on mesh images. Since early stopping procedure was employed over-fitting is prevented but with dropout layers it is expected to have longer training and model to learn more features. The graph shows that after 150th epoch model started to give higher distance between training loss and validation loss.
    

\clearpage

\section{Binary Semantic Segmentation with U-Net, with dropout layers }

    In this experiment the previous model is taken as base and added dropout layers in each encoder decoder blocks. First and second block have dropout rate 0.1, third and forth blocks have 0.2. Bottleneck layer has dropout rate : 0.3 and expansive path have the same values symmetrically (respectively 0.2, 0.2, 0.1, 0.1). 


    \begin{table}[h!]
        \centering
        \resizebox{\textwidth}{!}{%
        \begin{tabular}{
            >{\columncolor[HTML]{DAE8FC}}l l
            >{\columncolor[HTML]{DAE8FC}}l l
            >{\columncolor[HTML]{DAE8FC}}l }
                \cellcolor[HTML]{FFFFFF} &
                \cellcolor[HTML]{DAE8FC}Wire &
                 Wire with augmentation &
                \cellcolor[HTML]{DAE8FC}Mesh &
            Mesh with augmentation \\
        Mean IoU                           & 58.1\% & \textbf{58.8\%} & 81.1\% & \textbf{89.2\%} \\
        Dice coeff.                        &73.4\% & 74.1\%                        & 89.6\% & 94.3\%                        \\
        Pixel accuracy                     & 97.3\% & 97.4\%                        & 87.4\% & 93.1\%                        \\
        Pixel accuracy wire or mesh   only & 68.2\% & 69.1\%                        & 90.6\% & 95.2\%                       
    \end{tabular}%
    }
        \caption{results of standard U-Net with dropout layers on test set of mesh and wire images separately for mesh and wire models.}
        \label{table_2}
    \end{table}






    The obtained results on test set of standard U-Net model with dropout layers added, trained on wire and mesh images, with and without data augmentation employed are reported in Table \ref{table_2}. Model performance on a sample composition of original image, prediction and ground truth mask is shown in the Figure \ref{wire_dropout_img} for wire images with and without data augmentation employed. In the Figure \ref{mehs_dropout_img} for mesh images with and without data augmentation employed is demonstrated. \\



    \begin{figure}[H]
        \includegraphics[width=140mm,scale=0.99]{images_thesis/wire_dropout.png} 
        \centering
        \caption{Samples of wire prediction that is trained by standard U-Net model with dropout layers on with and without data augmentation employed wire images. Input image in the left, prediction in the center and target mask in the right.}
        \label{wire_dropout_img}
    \end{figure}
        \begin{figure}[H]
        \includegraphics[width=140mm,scale=0.99]{images_thesis/mesh_droput_img.png} 
        \centering
        \caption{Samples of mesh prediction that is trained by standard U-Net model with dropout layers on with and without data augmentation employed mesh images. Input image in the left, prediction in the center and target mask in the right.}
        \label{mehs_dropout_img}
    \end{figure}


    \begin{figure}[H]
        \includegraphics[width=130mm,scale=0.99]{images_thesis/loss_dropout_std_unet_mesh.png} 
        \centering
        \caption{Training and validation loss behavior of U-Net with dropout layers during the training on mesh images.}
        \label{losses_mesh_dropout}
    \end{figure}

    Figure \ref{losses_mesh_dropout} shows the validation loss - training loss plot for U-Net model on mesh images with dropout layers. Training procedure took longer time (since there was early stopping procedure and it did not stop the network during training), and model learnt more features without over-fitting the training data.The model is more robust compared to previous model (without dropout layers) as expected because the gap between training loss and validation loss is smaller in the second model. The inference from this plots is that, dropout layers worked well in terms of regularization which is also visible on the results table \ref{table_2}.

    
    
    In conclusion, this binary semantic segmentation experiment with dropout layers employed showed that dropout layers improved model performance with data augmentation applied data-set. This improvement is obviously seen in mesh image samples and the evaluation results are better than previous experiment. In the figure \ref{mehs_dropout_img} image b seems reasonably accurate for mesh sample, as well as figure \ref{wire_dropout_img} image b for wire sample. As it is intended, the noise in the image prediction is reduced. Thus, dropout layer helped the model to be less complex and model could learn better.







     
\clearpage






\section{Binary Semantic Segmentation with U-Net++ }
    U-Net++ model consists of an encoder and decoder path. The encoder path involves a series of convolutional blocks with max pooling layers (that is resulting in a down-sampled output) and that down-samples the input image. On the other hand the decoder path involves a series of upsampling layers that concatenates the corresponding encoder layer (concatenation layer merges multiple input tensors along a specified axis) followed by transpose convolutional block (used for upsampling the input data that has 2x2 filters with stride 2 and same padding), and the final output layer has a sigmoid activation function.

    U-Net++ have a deep supervision flag, if it set to True, it adds additional output layers with sigmoid activation for each skip connection, allowing intermediate outputs at different scales. However in this study Deep supervision is set as disables and consider U-Net++ model as default setting. In this architecture a Batch Normalization layer is used that normalizes the output of previous layer by subtracting the mean and dividing it by the standard deviation of the output and so making the model more robust and stable during training. As activation function ReLU (Rectified Linear Unit) is used, which sets all negative values to zero and leaves positive values unchanged.

    The proposed UNet++ model starts with filter size 32 that is used as standard U-Net++ model application in practice by the U-Net++ paper\cite{zhou2018unet++}. This model increases the filter size up to 512 filters in each encoder block. It is different that the standard UNet model which starts with 64 filters and goes up to 1024. (This thesis focuses on comparing standard models.)


\subsection{U-Net++ model performance results}

    The data-set that is exploited in this experiment consists of images with a resolution 512x512. The data-set is divided into three subset in pre-processing step as it was in U-Net experiment;
    For no data augmentation techniques employed data-set, for mesh: training set contains 854 images, validation set contains 163 images, and test set contains 120 images. For wire set: training set contains 772 images, validation set has 152 and test set has 112 images. 
    
    For data augmentation applied set, for mesh: the data-set is divided into three sub-sets: a training set consisting of 1708 images, a validation set consisting of 326 images, and a test set consisting of 120 images. For wire, data-set split is: Training 1544, validation 304, test 112.

    
    The U-Net model is trained for 250 epochs and Adam optimizer is used, with a batch size of 32 and a binary cross-entropy loss function. The early stopping process is monitored by calculating the mean Intersection over Union (IoU) on the validation set, and the patience was set to 50.

    The obtained results on test set of U-Net++ model trained on mesh and wire images with and without data augmentation employed approaches are reported in Table \ref{unetPP_reusults_test_set}.\\  


    \begin{table}[H]
        \centering
        \resizebox{\textwidth}{!}{%
        \begin{tabular}{
        >{\columncolor[HTML]{DAE8FC}}l 
        >{\columncolor[HTML]{FFFFFF}}l 
        >{\columncolor[HTML]{FFFFFF}}l 
        >{\columncolor[HTML]{FFFFFF}}l 
        >{\columncolor[HTML]{FFFFFF}}l }
        \color[HTML]{FE0000}\textbf{U-Net ++} &
        \cellcolor[HTML]{DAE8FC}Wire  &
        \cellcolor[HTML]{DAE8FC}Wire with augmentation &
        \cellcolor[HTML]{DAE8FC}Mesh  &
        \cellcolor[HTML]{DAE8FC}Mesh with augmentation \\
        Mean IoU                           & 54.5\% &\textbf{56.1\%} & 88.8\% &\textbf{89.3\%} \\
        Dice coeff.                        & 70.6\% & 71.8\% & 94.1\% & 94.4\% \\
        Pixel accuracy                     & 97.2\% & 97.2\% & 93.0\% & 93.2\% \\
        Pixel accuracy wire or mesh only & 63.0\% & 66.6\% & 93.5\% & 93.6\%
        \end{tabular}%
        }
        \caption{Report of U-Net++ results on test set on wire and mesh images with and without data augmentation }
        \label{unetPP_reusults_test_set}
    \end{table}
    
    A sample of each wire and mesh prediction that is trained by separate U-Net++ model in the scope of Binary Semantic Segmentation is shown in the figure \ref{sample6} 

    \begin{figure}[H]
        \includegraphics[width=140mm,scale=0.99]{images_thesis/22.jpg} 
        \centering
        \caption{A sample of \textbf{mesh} prediction that is trained by U-Net++ model on \textbf{no data augmentation} employed mesh images. Input image in the left, prediction in the center and target mask in the right.}
        \label{sample6}
    \end{figure}


    A sample of mesh prediction that is trained by U-Net++ model on no data augmentation employed mesh images is shown in the figure \ref{sample6}.
   

    \begin{figure}[H]
        \includegraphics[width=140mm,scale=0.99]{images_thesis/65.jpg} 
        \centering
        \caption{A sample of \textbf{wire} prediction that is trained by U-Net++ model on \textbf{no data augmentation} employed wire images. Input image in the left, prediction in the center and target mask in the right.}
        \label{sample7}
    \end{figure}


     A sample of wire prediction that is trained by U-Net++ model on no data augmentation employed wire images, is shown in figure \ref{sample7} \\
     

    \begin{figure}[H]
        \includegraphics[width=140mm,scale=0.99]{images_thesis/unetPP_mesh.jpg} 
        \centering
        \caption{A sample of \textbf{mesh} prediction that is trained by U-Net++ model on input images that is utilized \textbf{with augmentation} techniques. Input image in the left, prediction in the center and target mask in the right. }
        \label{sample8}
    \end{figure}

    A sample of mesh prediction that is trained by U-Net++ model on input images that is utilized with augmentation techniques, is shown in figure \ref{sample8} \\

  


    \begin{figure}[H]
        \includegraphics[width=140mm,scale=0.99]{images_thesis/unetPP_wire_auhm.jpg} 
        \centering
        \caption{A sample of \textbf{wire} prediction that is trained by U-Net++ model on input images that is utilized \textbf{with augmentation} techniques. Input image in the left, prediction in the center and target mask in the right. }
        \label{sample9}
    \end{figure}
      A sample of wire prediction that is trained by U-Net++ model on input images that is utilized with augmentation techniques, is shown in figure \ref{sample9}

       \begin{table}[H]
        \centering
        \resizebox{\textwidth}{!}{%
        \begin{tabular}{
        >{\columncolor[HTML]{DAE8FC}}l 
        >{\columncolor[HTML]{FFFFFF}}l 
        >{\columncolor[HTML]{FFFFFF}}l 
        >{\columncolor[HTML]{FFFFFF}}l 
        >{\columncolor[HTML]{FFFFFF}}l }
        \multicolumn{1}{c}{\cellcolor[HTML]{DAE8FC}\textbf{U-Net vs U-Net++}} &
          \cellcolor[HTML]{DAE8FC}U-Net wire &
        \cellcolor[HTML]{DAE8FC}U-Net++ wire &
          \cellcolor[HTML]{DAE8FC}U-Net mesh &
          \cellcolor[HTML]{DAE8FC}U-Net++ mesh \\
        Mean IoU                           & 52.1\% & \textbf{56.1\%} & 87.8\% & \textbf{89.3\%} \\
        Dice coeff.                        & 68.5\% & 71.8\% & 93.5\% & 94.4\% \\
        Pixel accuracy                     & 96.9\% & 97.2\% & 92.1\% & 93.2\% \\
        Pixel accuracy wire or mesh   only & 63.3\% & 66.6\% & 94.4\% & 93.6\%
        \end{tabular}%
        }
        \caption{comparison of U-Net and U-Net++ models trained on data augmentation applied data.}
        \label{comp_unets}
    \end{table}

    To conclude this experiment, it is worth to say that the U-net++ model gave good results, even better than U-Net model. However, there is no big difference between two model performances. Since the standard U-Net model is a lighter weight and less complex network compared to U-Net++, and the performance of both models is good enough, it is better to investigate the U-net model rather than UNet++ to search for the optimal parameter selection phase in the following chapter.
    Comparison of these two models on data without data augmentation employed is reported in table \ref{comp_unets} to better compare the results. 

   

\clearpage
\section{Multi-class Semantic Segmentation with U-Net }
    Following experiment aims to determine if multi-class semantic segmentation approach leads to improve the detection of mesh and wires. 
    Annotated data was pre-processed by labeling each pixel with a value of 0 indicating the background, 1 for the mesh, and 2 for the wire. \\ This application is shown in Figure \ref{multiclass_labeling}
    This annotated data is then converted into a numerical format by one-hot encoding.

    One-hot encoding is used to represent categorical data in a format that can be interpretable by algorithms. In multi-class semantic segmentation, it is used for converting pixel labels into a format that can be used to feed the deep learning model during training and evaluation processes. In  multi-class semantic segmentation, there are several labels that are assigned to each pixel in an image. One-hot encoding plays a role to represent these labels as a binary vector that c corresponds to a particular class. For instance, considering a multi-class semantic segmentation with three classes: wire, mesh, and background, each pixel is assigned to one of these labels. In this experiment, if a pixel is labeled as a wire, one-hot encoded representation would be [1, 0, 0], whereas if it's labeled as a mesh, it is [0, 1, 0] and lastly, if a pixel is labeled as background, its representation is [0, 0, 1]. In summary, this technique represents categorical labels as binary vector, where each element of the vector is corresponding to a possible label. The element is set as 1 if the pixel belongs to that class, or 0 otherwise. This allows the model to easily process each label as a separate class. For evaluation, metrics are adjusted for multi-class semantic segmentation with a pixel-by-pixel comparison used to identify intersecting pixels.

    \begin{figure}[H]
        \includegraphics[width=150mm,scale=0.99]{images_thesis/93.jpg} 
        \centering
        \caption{Pixel labeling is demonstrated; Original image in the left, ground truth mask in the center and corresponding pixel label in the right.}
        \label{multiclass_labeling}
    \end{figure}

    U-Net model without data augmentation is trained on data-set composed of 858 training images 167 validation images and 118 test images. Data preparation and splitting is shown in Figure \ref{multiclass-split-data}

     \begin{figure}[H]
        \includegraphics[width=150mm,scale=0.99]{images_thesis/multiclass-split-data.png} 
        \centering
        \caption{Data-set creation and splitting for multi-class semantic segmentation experiments is shown step by step.}
        \label{multiclass-split-data}
    \end{figure}


    
    In Figure \ref{multiclass_sampe}, multi-class semantic segmentation mask is shown with the original image. Green pixels are the demonstration of wire wheres red pixels refers to mesh and black is for background.


    \begin{figure}[H]
        \includegraphics[width=140mm,scale=0.99]{images_thesis/0.jpg} 
        \centering
        \caption{Original image in the left and multi-class mask in the right. }
        \label{multiclass_sampe}
    \end{figure}



    The data-set used in this experiment consists of images with 512x512 pixel resolution. The U-Net model is trained for 250 epochs using the Adam optimizer, with a batch size of 32 and a Sparse Categorical Crossentropy loss. The reason for using sparse categorical entropy loss in this experiment is that each image has more pixels in the mesh category than in the wire category. To deal with the imbalance between mesh and wire pixels, the sparse categorical entropy loss function is employed which can penalize the model more for incorrectly classification for the pixels of the minority class (or classes). By taking into account the frequency of each class in the training data and assigning higher weights to the minority class, loss function encourages the model focusing on accurately classifying the pixels in the less common classes. Thus, this is leading to potentially better performance in overall. It is worth to mention that in order to avoid over-fitting the early stopping strategy was monitored by calculating the mean Intersection over Union (IoU) on the validation set, and the patience value is set to 50.



\subsubsection{U-Net applied on images and multi-class masks}

    The obtained results on test set of U-Net model trained on images are reported in Table \ref{multi_adam1} and a sample composition of original image, prediction and ground truth mask is shown in the Figure \ref{90_multi_adam1}



    \begin{table}[H]
        \centering
        \resizebox{\textwidth}{!}{%
        \begin{tabular}{lllll}
        \rowcolor[HTML]{DAE8FC} 
        \textbf{\begin{tabular}[c]{@{}l@{}}\color[HTML]{303498}Multiclass\end{tabular}} & \textbf{Average} & \textbf{Background} & \textbf{Mesh} & \textbf{Wire} \\
        \rowcolor[HTML]{FFFFFF} 
        \cellcolor[HTML]{DAE8FC}\textbf{Mean IoU}       & {\color[HTML]{303498} \textbf{63.9\%}} & 79.9\% & 80.8\% & 30.8\% \\
        \rowcolor[HTML]{FFFFFF} 
        \cellcolor[HTML]{DAE8FC}\textbf{Dice coeff.}    & {\color[HTML]{303498} \textbf{75.1\%}} & 88.8\% & 89.4\% & 47.1\% \\
        \rowcolor[HTML]{FFFFFF} 
        \cellcolor[HTML]{DAE8FC}\textbf{Pixel accuracy} & {\color[HTML]{303498} \textbf{72.0\%}} & 90.8\% & 89.9\% & 35.3\%
        \end{tabular}%
    }
    \caption{U-Net results on test set with average evaluation value in the firs column, background only in the second, mesh only in the third and wire only in the last column. Optimizer employed: Adam (lr=0.0001))}
    \label{multi_adam1}
    \end{table}

    \begin{figure}[H]
        \includegraphics[width=140mm,scale=0.99]{images_thesis/90_adam2.jpg} 
        \centering
        \caption{A sample of prediction that is trained by U-Net model on input images for multi-class semantic segmentation. Input image in the left, prediction in the center and target mask in the right. Model trained with Adam (lr=0.0001) }
        \label{90_multi_adam1}
    \end{figure}

    \begin{figure}[H]
        \includegraphics[width=140mm,scale=0.99]{images_thesis/multiclass_std_unet.png} 
        \centering
        \caption{Training and validation loss behavior of U-Net model for multi-class semantic segmentation during the training}
        \label{multi_train_val_loss}
    \end{figure}

    The validation loss - training loss plot is shown in the figure \ref{multi_train_val_loss} and it shows that model is trained until 200th epoch even though it was set until 250 epochs. For obtaining less complex and more robust network dropout layers will be employed in the following.     
\section{Multi-class Semantic Segmentation with U-Net, with dropout layers }

    To improve the performance different parameters are employed and this is started with different learning rate for Adam optimizer. Therefore, same configuration is tested with default values of Adam is reported in table \ref{multi_adam2} and a sample composition of original image, prediction and ground truth mask is shown in the Figure \ref{90_multi_adam22}
 
    \begin{table}[H]
        \centering
        \resizebox{\textwidth}{!}{%
        \begin{tabular}{lllll}
        \rowcolor[HTML]{DAE8FC} 
        \textbf{\begin{tabular}[c]{@{}l@{}}\color[HTML]{303498}Multiclass \end{tabular}} & \textbf{Average} & \textbf{Background} & \textbf{Mesh} & \textbf{Wire} \\
        \rowcolor[HTML]{FFFFFF} 
        \cellcolor[HTML]{DAE8FC}\textbf{Mean IoU}       & {\color[HTML]{303498} \textbf{69.1\%}} & 80.6\% & 82.6\% & 44.2\% \\
        \rowcolor[HTML]{FFFFFF} 
        \cellcolor[HTML]{DAE8FC}\textbf{Dice coeff.}    & {\color[HTML]{303498} \textbf{80.3\%}} & 89.3\% & 90.5\% & 61.3\% \\
        \rowcolor[HTML]{FFFFFF} 
        \cellcolor[HTML]{DAE8FC}\textbf{Pixel accuracy} & {\color[HTML]{303498} \textbf{77.1\%}} & 88.8\% & 92.1\% & 50.5\%
    \end{tabular}%
    }
    \caption{U-Net results on test set with avarege evaluation value in the firs column, background only in the second, mesh only in the third and wire only in the last column. Optimizer employed: Adam (lr=0.001) }
    \label{multi_adam2}
    \end{table}




    \begin{figure}[H]
        \includegraphics[width=140mm,scale=0.99]{images_thesis/90_multi_adam1.jpg} 
        \centering
        \caption{A sample of prediction that is trained by U-Net model on input images for multi-class semantic segmentation. Input image in the left, prediction in the center and target mask in the right. Model trained with Adam (lr=0.001) }
        \label{90_multi_adam22}
    \end{figure}

    \begin{figure}[H]
        \includegraphics[width=140mm,scale=0.99]{images_thesis/multiclass_std_unet_dropout.png} 
        \centering
        \caption{Training loss - Validation loss plot that shows training and validation loss behavior thorough the increasing epoch numbers of U-Net model with dropout layers for multi-class semantic segmentation during the training process.}
        \label{multi_train_val_loss_dropout}
    \end{figure}

    Figure \ref{multi_train_val_loss_dropout} shows the validation loss - training loss plot for U-Net model with dropout layers. It shows that training took longer time compared to the model without dropout layers that gave result in the table \ref{multi_adam1} (since there was early stopping procedure and it did not stop the network during the training), and model learnt more features without over-fitting the training data. Model is more robust with dropout layers compared to previous model as expected.\\ 
    The gap between training loss and validation loss is not higher compared to the graph \ref{multi_train_val_loss}. Thus, as expected, dropout layers worked well with the effect of regularization which is also reported at the results table \ref{multi_adam2}. \\

    Lastly, data augmentation applied data-set which involves 1716 training, 334 validation and 118 test images is used (data augmentation techniques are only applied on training and validation sets). How the data is prepared and split is shown in the figure \ref{multiclass_split_aug_1}. It is expected to obtain better results since data augmentation employed data-set gave better results so far in the previous experiments.

    \begin{figure}[H]
        \includegraphics[width=140mm,scale=0.99]{images_thesis/multiclass_split_aug_1.png} 
        \centering
        \caption{ Data-set creation with data augmentation step by step and indication of the number of images of each data-set}
        \label{multiclass_split_aug_1}
    \end{figure}


\begin{table}[H]
\centering
\resizebox{\textwidth}{!}{%
\begin{tabular}{lllll}
\rowcolor[HTML]{DAE8FC} 
{\color[HTML]{303498} \textbf{Multiclass, Data Augmentation}} & Average                                 & Background only & Mesh only & Wire only \\
\cellcolor[HTML]{DAE8FC}Mean IoU & \cellcolor[HTML]{FFFFFF}{\color[HTML]{303498} \textbf{70.7\%}} & 81.1\% & 82.7\% & 48.3\% \\
\cellcolor[HTML]{DAE8FC}Dice coeff.                           & {\color[HTML]{303498} \textbf{81.7\%}} & 89.5\%         & 90.6\%   & 65.2\%   \\
\cellcolor[HTML]{DAE8FC}Pixel accuracy                        & {\color[HTML]{303498} \textbf{79.6\%}} & 90.6\%         & 90.7\%   & 57.5\%  
\end{tabular}%
}
\caption{Multiclass-Semantic Segmentation experiment with data augmentation employed dataset (loss: sparse categorical cross entropy, Optimizer:  Adam(lr=0.001)}
\label{multiclass_data_aug}
\end{table}


    Table \ref{multiclass_data_aug} reports the results of multi-class U-Net model on data-set that is obtained with data augmentation. It is obvious that the performance of the model is increased. However still not as good as binary semantic segmentation for U-Net.

    Figure \ref{figure_multiclass_data_aug} shows the prediction that is made by multiclass U-Net on data augmentation employed data.

    
        \begin{figure}[H]
        \includegraphics[width=140mm,scale=0.99]{images_thesis/figure_multiclass_data_aug.jpg} 
        \centering
        \caption{ A sample of prediction that is trained by U-Net model on input images extended by data augmentation for multi-class semantic segmentation}
        \label{figure_multiclass_data_aug}
    \end{figure}

    In this multi-class semantic segmentation experiment, the goal was detecting both mesh and wire in a single model implementation. However, the performance of the model decreased significantly (compared to binary semantic segmentation application), especially in the detection of wire. Therefore, there is a compromise between employing a single model to detect both objects at pixel-level and accurately obtaining the segmentation results. As this thesis aims to identify an optimal model with high evaluation scores on test set, it will continue to focus on the binary semantic segmentation approach.

\clearpage



