Adversarial certificate-based testing of mTLS (mutual TLS) handshakes for test-driven development

TLS handshake implementations are often difficult to comprehensively test. This is in part due to the complexity of the X.509 certificate specification and its influence on relationships between certificates (for example, tests should check for “cA” in Basic Constraints before verifying relevant certificate authority signatures, and should also test that all certificates are currently valid). Even when the TLS protocol seems to be implemented correctly, subtle vulnerabilities in certificate validation can still occur and may only appear in edge cases. Contributing factors include incomplete implementation of the X.509 specification, relative weakness of cybersecurity as a part of the software development lifecycle, and the traditional software development perspective that certificate generation is an uncommon procedure (in contrast to DevOps, microservices, or certificate-rotation based strategies). mTLS (mutual TLS) is used by applications to create a zero-trust architecture: the mTLS protocol proposes that both the server and client (or two nodes more generally) are untrusted and that each must authenticate the other. However, mTLS handshakes are even more difficult to implement and test compared to ordinary TLS, requiring a correct implementation of server authentication and client authentication while simultaneously meeting the X.509 specification requirements. The choice to adopt a TDD (test-driven development) strategy for the creation of mTLS-based protocols may resolve some challenges relating to gaps in protocol implementation and may help integrate effective mTLS cybersecurity testing into the software development process. There is currently no commercial or academic testing suite designed to specifically test mTLS cybersecurity. This thesis builds on the existing cybersecurity method of adversarial testing by adapting it to the mTLS protocol and creating a programming-language-agnostic mTLS test suite. The certificate-based test suite is applied to a public code base to demonstrate the suitability of the tests and their ability to reveal subtle bugs in a mTLS handshake implementation. The test suite is further optimized for test-driven development, with a particular focus on explanation of vulnerabilities. Future work could include expanding the testing scope by analyzing mTLS revocation procedures or randomizing test inputs; further testing the use of the tool as a part of the SDLC; and expanding real-world evaluations to other mTLS endpoints, mTLS mock servers, or mTLS software implementations.