To all developers, DevOps, testers, and those interested in testing: Do you know this?

Source: The Square Hole by Brock

Every time I test something I am immediately reminded of the video. So also last time while manually testing a REST API where I needed to create an offer and therefore, posted personal data of a fictional person as policy holder. The policy holder needs to be an adult, which is checked by the provided birthdate in the response body as string. The tests passed with different valid birth dates as strings. But an unexpected behavior occured when inserting an integer like 2021 instead of a string. It was assumed that the request couldn’t be processed due to invalid input types, but an offer was successfully created with the birthdate of 1975-07-15. Meaning that a badly formatted input for the birthdate can be abused for creating offers. This was only a small example of manually mutating an input for fuzz testing, but there are more sophisticated solutions.

Read the glass box

Testing an application in an operational state is referred to as Dynamic Application Security Testing (abbr. DAST) and is also called behavioral testing, in contrast to the source code focused Static Application Security Testing (abbr. SAST). Web application vulnerability scanners are tools for DAST and are automated tools for security vulnerability detections like Cross-site scripting, SQL injection, Command Injection, and Path Traversal. These tools are also helpful for penetration and fuzz testing. Penetration testing is an authorized and simulated vulnerability scan under real conditions where the system is attacked to gain access to features or data. Fuzz testing is more known for black-box testing of the inputs and outputs, the protocols, and file formats of applications. By fuzz testing, invalid or unexpected input is sent or submitted to a target, commonly a system under test (abbr. SUT).

There are different types of fuzz testing and its fuzzers: mutation-based, generation-based, and protocol-based fuzzers. Mutation-based fuzzers create the input data by mutating existing data samples. Generation-based fuzzers generate input from scratch and protocol-based fuzzers use the knowledge from protocol formats being tested and are also known as syntax testing, grammar testing, robustness testing.

(Non-Covid) Mutations

Thanks to my colleague, Paloma and her DevJF talk on March 25, 2020 we’ve already gained insights on mutation testing. In short, slightly altered code in our application, the so-called mutants are used to let our tests fail and conclusively, let us know what we haven’t considered or tested yet. This result is described as the mutation score in %, which is the ratio between the number of killed mutants and total mutants. The altered code is adapted by using an adequate library like PITest and is a replacement for logical or arithmetic operators, mutated in- or decrements, assignments, inverted negatives, conditions and many more.

Hot Fuzz

A similar procedure can be applied to fuzz and mutation testing of REST APIs. By utilizing the vulnerability scanning tool Open Web Application Security Project Zed Attack Proxy (abbr. OWASP ZAP) and its tools for OpenAPI and fuzzing support, an example OpenAPI specification is loaded into OWASP ZAP.

By that, all endpoints including their request and response bodies are loaded and can be browsed. As a guinea pig, the Swagger Petstore specification is imported from its URL https://petstore.swagger.io/v2/swagger.json and was successfully parsed.

Even after loading the specification, OWASP ZAP can already provide alerts. One of the medium alerts is Cross-Domain Misconfiguration, which is indicated by Access-Control-Allow-Origin: * in the DELETE/pet/{petId} header response. Meaning that the domain allows access to resources from all sites and web browser data can be loaded and abused.

Let’s have a look at the DELETE/pet/{petId} endpoint and its request header. The petId is a required parameter and needs to be an integer. By this provided petId, the pet is looked up and deleted from the database. Beneath 200 – Ok, 400 – Invalid ID supplied and 404 – Pet not found can be responded.

Marking the part of the header that is fuzzed, right-clicking on it and selecting Fuzz… the following Fuzzer pop-up appears. For the before marked part a payload can be added. The possible types of payloads can be manually defined by typing or selecting File Fuzzers, which have a predefined set of payloads.

The available fuzzers are FuzzDB, JBroFuzz, and DirBuster. The available files contain payloads as used for Cross-Site Scripting, Buffer and Integer Overflows, No-SQL and SQL Injections, and many more.

After adding the fuzzer files, the fuzzer can be started. Finally, the output needs to be overlooked for suspicious responses and error messages.

CI/CD Integration

A step further, would be to integrate the above mentioned DAST vulnerability scanning tools like the above mentioned and open-source OWASP ZAP in the project’s pipeline. The scanning is initiated by a merge request and runs as a CI/CD pipeline job. The relevant findings are noted in the merge request, when the OWASP ZAP report is created and the vulnerabilities are evaluated by GitLab between the source and target branches.

Note that this feature is only available in GitLab Ultimate and when the GitLab Runner runs with the docker executor.

DAST can be included by Auto DAST or manually including the DAST template. For an example of the DAST template, please refer to the DAST.gitlab-ci.yml.

Summary

Fuzzing, as part of DAST, is a powerful tool for finding bugs, unexpected behavior, memory leaks, assertion failures, unhandled exceptions, crashes, and invalid inputs of applications. However, fuzzing isn’t capable of dealing with security threats like viruses, worms or Trojans. Another disadvantage is that the setup will require significant time and knowledge. On the other hand, due to the possibility to use automated scans and tools and even CI/CD integration, the limitation of time and resources in software testing can be softened.

Sources

• What is Software Testing | Everything You Should Know
• Dynamische Software-Tests: Wie Sie mit Fuzzing mehr Bugs aufspüren
• Fuzz Testing(Fuzzing) Tutorial: What is, Types, Tools & Example
• Dynamic Application Security Testing
• Vulnerability Scanning Tools
• OWASP Zed Attack Proxy
• Cross-Domain Misconfiguration
• 3 ways you can exploit CORS Misconfigurations
• DAST in GitLab