We started writing more software with AI tools. AI is just another tool for writing software, it is unlikely that we will stop using AI to write software.
Software isn’t just getting bigger; it’s evolving fast. Today code is written by tools or even AI with humans in the loop. More and more code will be generated by tools or AI, which somehow changes how we need to test software. It seems that the testing software of your implementation is more important than your actual domain implementation to have confidence in changing and upgrading software.
Testing has always been about proving the software works as expected. You write a unit test, give function-specific inputs, and check the output. That’s fine, but what happens with inputs you didn’t think of?
Fuzzing is not new. It was used earlier, but it was slow because of slow computers at that time. But now we have multi-core computers that are capable of doing fuzzing in more practical ways, so it’s worth looking into it again.
Fuzzing creates randomly simulated inputs and remembers them in the snapshot as an input. Then it re-runs with the same inputs to expect the same outputs.
Think about your APIs. They’re the gates between your software and the rest of the world. If those gates aren’t solid, no amount of internal testing will protect you.
Instead of trying to imagine every possible input, fuzzing throws everything at your code. The correct randomness uncovers bugs you’d never anticipate.
There’s a challenge, though: the input search space is huge. Take a simple function like
sum(int a, int b)—32-bit integers alone giving you billions of combinations. Testing all of
them isn’t practical.
So the question is how to reduce fuzzing search space for even more complicated functions. There are tricks, like symbolic execution and formal methods but they only work on pure math functions or simple cases. Real-world software is messy. The problem is figuring out how to test enough without wasting time on everything. One possible solution is to use LLMs to extend the simple fuzzer-generated test suite with smart string generation. Exploit symmetries in certain domains, such as REST APIs: For example POST, and DELETE are the opposite of actions. That’s why I started building fuse-drill a fuzzing tool for REST APIs.
Inspired by Haskell QuickCheck this is where FuseDrill comes in. It makes fuzzing practical and useful, especially for APIs. Here’s how:
Automatic Input Generation: FuseDrill creates all kinds of random input combinations for API requests, so you don’t have to write it manually yourself.
Snapshot Comparisons: Saves API responses to source control. This gives you a baseline to catch changes later.
CI/CD Integration: FuseDrill runs as part of your CI/CD pipeline, so certain types of bugs are caught early—before they hit production.
Feeding snapshots to LLMs: As software eats itself, we feed dynamic fuzzing reports to reasoning LLMs to extract defects that we did from the previous version to the current one.
Here are some ideas to take it further:
AI-Driven Fuzzing: Use large language models to improve Fuzzer generated inputs, to reduce fuzzing search space.
HPC distributed Fuzzing: Run fuzzing on many machines with thousands of containers by splitting search space in parallel to cover more cases in isolation.
Deep System Testing / Vertical slices testing: Extend fuzzing beyond APIs to test how internal components interact, using tools like OpenTelemetry.
Synthetic data: Fuzzers create synthetic high-quality data that extracts the essence of domain knowledge that no amount of static analyzers can do.
Want to learn more or try it yourself? Or if you like to improve the tool make an issue or even better a pull request. Check out the FuseDrill GitHub repository.