\section{Grid Search}

    This section aims to find an optimal combination of parameters such as optimization and loss function for the previously used U-Net model by exploring different options. This investigation involves training the standard U-Net model and finding the best combination for both mesh and wire separately in terms of binary semantic segmentation. Once the reasonably good combination is found among the chosen parameters, the focus shifts to determine the best filter size and number of convolution blocks for the up-sampling and down-sampling blocks.

\subsubsection{Trial of different loss functions and optimizers}
    Different combination of loss functions and optimizers are investigated in this section. For binary semantic segmentation most widely used functions are taken into consideration. 
    For optimizer Adam and RMSprop are going to be considered with their default values and for loss functions Binary focal loss and Binary Cross Entropy loss will be taken into account with their default parameters. 
    
    
    Results on test set of U-Net model trained on wire images with combinations of Adam and RMSprop optimizers, and Binary-focal Loss and Binary Cross Entropy loss are shown in the table \ref{tab:my-table} for wire and for mesh it is shown in table \ref{tab:my-table-mesh}. \\


    

\begin{table}[h!]
\centering
\resizebox{\textwidth}{!}{%
\begin{tabular}{|l|l|l|l|l|}
\textbf{Wire} & 
  \cellcolor[HTML]{DAE8FC}Adam(lr=0.0001) + BCE &
  \cellcolor[HTML]{DAE8FC}Adam(lr=0.0001) + BFL &
  \cellcolor[HTML]{DAE8FC}RMSprop + BCE &
  \cellcolor[HTML]{DAE8FC}RMSprop + BFL \\
\cellcolor[HTML]{DAE8FC}Mean IoU                 & 54.1\% & 55.3\% & 56.5\% & \textbf{56.8\%} \\
\cellcolor[HTML]{DAE8FC}Dice coeff.              & 70.2\% & 71.2\% & 72.2\% & 72.4\% \\
\cellcolor[HTML]{DAE8FC}Pixel accuracy           & 97.1\% & 97.2\% & 97.2\% & 97.3\% \\
\cellcolor[HTML]{DAE8FC}Pixel accuracy wire-only & 63.1\% & 65.2\% & 67.8\% & 65.8\%
\end{tabular}%
}
\caption{Results on test set of U-Net model trained on wire images with combinations of Adam and RMSprop optimizers, and Binary-focal Loss and Binary Cross Entropy loss.}
\label{tab:my-table}
\end{table}

    In the combination of Adam with both Binary Cross-Entropy and Binary Focal Loss, the U-Net model was fitted to the local minimum and thus the learning rate was adjusted to obtain the results. (In default learning rate = 0.001, adjusted form has learning rate = 0.0001)



\begin{table}[h!]
\centering
\resizebox{\textwidth}{!}{%
\begin{tabular}{|l|l|l|l|l|}
 \textbf{Mesh}&
  \cellcolor[HTML]{DAE8FC}Adam + BCE(lr=0.0001) &
  \cellcolor[HTML]{DAE8FC}Adam(lr=0.0001) + BFL &
  \cellcolor[HTML]{DAE8FC}RMSprop + BCE &
  \cellcolor[HTML]{DAE8FC}RMSprop + BFL \\
\cellcolor[HTML]{DAE8FC}Mean IoU                 & \textbf{88.4\%} & 88.3\% & 87.6\% & 86.6\% \\
\cellcolor[HTML]{DAE8FC}Dice coeff.              & 93.8\% & 93.8\% & 93.4\% & 92.8\% \\
\cellcolor[HTML]{DAE8FC}Pixel accuracy           & 92.5\% & 92.6\% & 92.0\% & 91.1\% \\
\cellcolor[HTML]{DAE8FC}Pixel accuracy mesh-only & 95.1\% & 94.1\% & 95.1\% & 95.5\%
\end{tabular}%
}
\caption{Results on test set of U-Net model trained on mesh images with combinations of Adam and RMSprop optimizers, and Binary-focal Loss and Binary Cross Entropy loss.}
\label{tab:my-table-mesh}
\end{table}


    In conclusion, this grid search to find optimal parameters among the ones that is chosen, has shown that the RMSprop + Binary Focal Loss gives best results for wire among all others. 
    On the other hand, for mesh the best results are obtained with combination of Adam and Binary Cross entropy (with learning rate 0.0001).\newline


    Once the optimal optimizer and loss function have been selected for the model and data, the next step is to determine the best filter sizes. While the initial experiments with standard U-Net model were conducted using a starting filter size of 64, we are now examining the impact of filter size on model performance by testing with starting filter smaller filter size as base filter size is 32.  This search will provide if the optimal filter size for the model is smaller.

\subsubsection{Trial of different filter sizes for wire}


    \par Starting with the best parameters that are obtained by the previous experiments, the focus is turned into changing the number of filters. The obtained results on test set of U-Net model trained on wire images with RMSprop optimizer and Binary Focal Loss with base filter size 32 and the model that gave the best result so far for wire ımages during grid search previously(with base filter size 64) are reported in Table \ref{tab:my-table_small_fs_wire} 


    \begin{table}[h!]
        \centering
        \resizebox{\textwidth}{!}{%
        \begin{tabular}{|l|l|l|}
        \textbf{Wire}
         & \cellcolor[HTML]{DAE8FC}RMSprop + BFL
         (beginning fs:32) & \cellcolor[HTML]{DAE8FC}RMSprop + BFL(beginning fs:64) \\
        \cellcolor[HTML]{DAE8FC}Mean IoU                 & {\color[HTML]{000000} 54.5\%} &\textbf{56.8\%} \\
        \cellcolor[HTML]{DAE8FC}Dice coeff.              & 70.6\%                        & 72.4\% \\
        \cellcolor[HTML]{DAE8FC}Pixel accuracy           & 97.1\%                        & 97.3\% \\
        \cellcolor[HTML]{DAE8FC}Pixel accuracy wire-only & 63.7\%                        & 65.8\%
    \end{tabular}%
}
\caption{Different filter sizes (fs) comparison on wire images for U-Net model with base filter size 32 and 64.}
\label{tab:my-table_small_fs_wire}
\end{table}

    This search shows that bigger filter sizes are more suitable to obtain better results on test sets for wire images. 


    \par The obtained results on test set of U-Net model trained on mesh images with Adam optimizer and Binary Cross Entropy (best optimizer loss function combination on mesh images during grid search so far) with base filter size 32 and the previous model with base filter size 64 are reported in Table \ref{table_smaller_fs_mesh} 


        \begin{table}[H]
            \centering
            \resizebox{\textwidth}{!}{%
            \begin{tabular}{|l|l|l|}
            \textbf{Mesh}
             &
             \cellcolor[HTML]{DAE8FC}\begin{tabular}[c]{@{}l@{}}Adam + BCE(lr=0.0001)\\ (filter size starts with 32)\end{tabular} &
              \cellcolor[HTML]{DAE8FC}\begin{tabular}[c]{@{}l@{}}Adam + BCE(lr=0.0001)\\(filter size starts with 64)\end{tabular} \\
            \cellcolor[HTML]{DAE8FC}Mean IoU                 & 87.7\% & \textbf{88.4\%} \\
            \cellcolor[HTML]{DAE8FC}Dice coeff.              & 93.4\% & 93.8\% \\
            \cellcolor[HTML]{DAE8FC}Pixel accuracy           & 92.1\% & 92.5\% \\
            \cellcolor[HTML]{DAE8FC}Pixel accuracy mesh-only & 94.3\% & 95.0\%
        \end{tabular}%
        }
        \caption{Different filter sizes comparison on mesh images for U-Net model with base filter size 32 and 64.}
        \label{table_smaller_fs_mesh}
    \end{table}

    This search shows that bigger filter sizes are more suitable also for obtaining better results on mesh images.




\subsubsection{Experimenting with varying numbers of up-sampling and down-sampling layers.}

    After having identified the best optimizer, the best loss function, and the base filter size among the chosen parameters, the focus now turns on examining how the performance is being affected by different number of the up-sampling and down-sampling blocks in U-Net model.

    In this experiment original U-Net architecture is compares with smaller one with one encoder and one decoder block eliminated from both sides of the network. 
    For the parameter selection the best results given parameters are selected and indicated in the Table \ref{smaller_enc_Dec_wire} with the all results for wire and results for mesh images are indicated in table  \ref{smaller_enc_Dec_mesh}
    
    

    
    \begin{table}[h!]
        \centering
        \resizebox{\textwidth}{!}{%
        \begin{tabular}{|l|l|l|}
        \textbf{Wire}
            & \cellcolor[HTML]{DAE8FC}RMSprop + BFL \textbf{smaller} & \cellcolor[HTML]{DAE8FC}RMSprop + BFL \textbf{standard} \\
            \cellcolor[HTML]{DAE8FC}Mean IoU                 & 54.3\% & \textbf{56.8\%} \\
            \cellcolor[HTML]{DAE8FC}Dice coeff.              & 70.4\% & 72.4\% \\
            \cellcolor[HTML]{DAE8FC}Pixel accuracy           & 97.2\% & 97.3\% \\
            \cellcolor[HTML]{DAE8FC}Pixel accuracy wire only & 61.0\% & 65.8\%
        \end{tabular}%
        }
        \caption{Different size of encoder decoder blocks are compared on wire images for U-Net model. 3 encoder blocks - 3 decoder blocks are reported in the left and standard U-Net model with 4 encoder - 4 decoder in the right.
        }
     \label{smaller_enc_Dec_wire}
    \end{table}


    
    
    \begin{table}[h!]
        \centering
        \resizebox{\textwidth}{!}{%
        \begin{tabular}{|l|l|l|}
        \textbf{Mesh}
         &
             \cellcolor[HTML]{DAE8FC}\begin{tabular}[c]{@{}l@{}}Adam + BCE \textbf{smaller}\end{tabular} &
             \cellcolor[HTML]{DAE8FC}\begin{tabular}[c]{@{}l@{}}Adam + BCE \textbf{standard}\end{tabular} \\
            \cellcolor[HTML]{DAE8FC}Mean IoU                 & \textbf{89.1\%} & 88.4\% \\
            \cellcolor[HTML]{DAE8FC}Dice coeff.              & 94.3\% & 93.8\% \\
            \cellcolor[HTML]{DAE8FC}Pixel accuracy           & 93.0\% & 92.5\% \\
            \cellcolor[HTML]{DAE8FC}Pixel accuracy mesh only & 95.6\% & 95.1\% \\
        \end{tabular}%
        }
        \caption{Different size of encoder decoder blocks are compared on mesh images with the same configuration. Adam optimizer and Binary cross entropy loss with learning rate=0.0001 is used for both settings.}
        \label{smaller_enc_Dec_mesh}
    \end{table}
    
    
    
    
    The experiment shows that smaller encoder-decoder blocks results in less accurate solution for wire images. However for mesh images the result is the opposite. 
    After obtaining all the results, the best model for wire is U-Net model with dropout layers, Adam optimizer and binary cross entropy loss. For mesh the best result that is achieved is the U-Net model with smaller encoder decoder blocks, Adam optimizer Binary cross entropy loss with learning rate 0.0001. \\ \\ 
    Last experiment is employing data augmentation applied data-set on mesh images for best model that is obtained so far for mesh since it is certain that  data augmentation always improved the performance. The results are reported in the table \ref{bestmesh_results}

   
    \begin{table}[H]
    \centering
    \resizebox{\textwidth}{!}{%
    \begin{tabular}{
    >{\columncolor[HTML]{DAE8FC}}l lll}
    \begin{tabular}[c]{@{}l@{}}\color[HTML]{FE0000}Mesh \\    Adam + BCE (lr=0.0001)\end{tabular} &
      \cellcolor[HTML]{DAE8FC}smaller enc-dec &
      \cellcolor[HTML]{DAE8FC}smaller enc-dec + dropout &
      \cellcolor[HTML]{DAE8FC}{\textbf{smaller enc-dec + dropout + augmentation}} \\
    Mean IoU                 & 89.1\% & 89.6\% & \textbf{90.2\%} \\
    Dice coeff.              & 94.3\% & 94.5\% & 94.8\% \\
    Pixel accuracy           & 93.0\% & 93.4\% & 93.8\% \\
    Pixel accuracy mesh-only & 95.6\% & 95.3\% & 95.1\%
    \end{tabular}%
    }
    \caption{Best mesh results trained on data augmentation employed mesh images by U-Net model with dropout layers, smaller encoder decoder blocks }
    \label{bestmesh_results}
    \end{table}
    
    Finally, the best models for mesh and wire images that is obtained after all these experiments are reported in the table \ref{comp_models_best} and model selection phase is concluded with these results.


    \begin{table}[h!]
        \centering
        \resizebox{\textwidth}{!}{%
        \begin{tabular}{
        >{\columncolor[HTML]{DAE8FC}}l 
        >{\columncolor[HTML]{FFFFFF}}l 
        >{\columncolor[HTML]{FFFFFF}}l }
        \multicolumn{1}{c}{\cellcolor[HTML]{DAE8FC}\textbf{Best Model }} & \cellcolor[HTML]{DAE8FC}\textbf{Best model for mesh} & \cellcolor[HTML]{DAE8FC}\textbf{Best model for wire} \\
        Mean IoU                           & \textbf{90.2\%} & \textbf{58.1\%} \\
        Dice coeff.                        & 94.8\%                        & 73.5\%                        \\
        Pixel accuracy                     & 93.8\%                        & 97.3\%                        \\
        Pixel accuracy wire or mesh   only & 95.1\%                        & 68.2\%                       
        \end{tabular}%
        }
        \caption{Best model results for mesh and wire are reported.\\ \textbf{Best model for mesh:} smaller network with dropout layers on data augmentation employed data-set, optimizer: Adam, Loss: Binary Cross Entropy (learning rate = 0.0001). \\ 
        \textbf{Best model for wire:} standard model with dropout layer on data augmentation employed data-set. Adam, Loss: Binary Cross Entropy (learning rate = 0.001).}
        \label{comp_models_best}
    \end{table}

    To conclude this grid search section, it is worth to mention that 90.17\% mIoU result on test set is an outstanding performance rate for a data that is manually labeled by humans because manually labeled data can vary depending on human annotators expertise, complexity of objects in an image, etc. This situation is valid also for wire images. Almost 60\% of mIoU ratio is very good for smaller objects in an image (in this case wire objects in the input images have smaller pixel values compared to meshes). Smaller objects are more challenging to detect or segment accurately due to the fact that smaller objects may involve less distinct boundaries and/or may be overlapped by other objects in an image and this may lead unexpected predictions. It is harder to detect and segment the smaller object pixels for computer vision models. Therefore it negatively affects the performance. 
    The best models predictions on mesh and wire images are shown in the figure \ref{lastlast}

    

    \begin{figure}[H]
        \includegraphics[width=145mm,scale=0.99]{images_thesis/lastlast.png} 
        \centering
        \caption{A sample of prediction that is trained by \textbf{best} U-Net models on input images for mesh and wire. Input image in the left, prediction in the center and target mask in the right. }
        \label{lastlast}
    \end{figure}

