<p>I have developed a framework that is used by several teams in our organisation. Those "modules", developed on top of this framework, can behave quite differently but they are all pretty resources consuming even though some are more than others. They all receive data in input, analyse and/or transform it, and send it further. </p>

<p>We planned to buy new hardware and my boss asked me to define and implement a benchmark based on the modules in order to compare the different offers we have got. </p>

<p>My idea is to simply start sequentially each module with a well chosen bunch of data as input. </p>

<p>Do you have any advice? Any remarks on this simple procedure? </p>

The most meaningful benchmark is to measure how your code performs under everyday usage.  That will obviously provide you with the most realistic numbers.
Choose several real-life data sets and put them through the same processes your org uses every day.  For extra credit, talk with the people that use your framework and ask them to provide some "best-case", "normal", and "worst-case" data.  Anonymize the data if there are privacy concerns, but try not to change anything that could affect performance.
Remember that you are benchmarking and comparing two sets of hardware, not your framework.  Treat all of the software as a black box and simply measure the hardware performance.
Lastly, consider saving the data sets and using them to similarly evaluate any later changes you make to the software.