Benchmark synthesis

[mcopik]

We need the following:

• Python
  • computation in flosp/instructions
  • memory allocation
  • storage read/write
  • disk read/write
• NodeJS
  • computation in flosp/instructions
  • memory allocation
  • storage read/write
  • disk read/write

[mcopik]

We made progress on this issue on branch meta-benchmarks and in PR #59. However, there is still work to be done - any input and help towards synthesizing benchmarks are welcome!

[veenaamb]

I will research on this and update here shortly

[AtulRajput01]

i am working on it.

[octonawish-akcodes]

@mcopik Can i get some guidance to this issue?

[mcopik]

@octonawish-akcodes Hi! The overall idea is to synthetically create Python/JS functions that perform CPU computations, memory accesses, and I/O accesses. Given a simple configuration, it should generate a function that performs selected actions with a specified frequency and intensity, e.g., calling some well-established CPU benchmark (like matrix-matrix multiplication), using our interface to make storage calls, etc.

The next step will be to make these functions more varied, e.g., with different loop complexity.

[octonawish-akcodes]

Can you also provide me with some resources and target files for the start?

[mcopik]

@octonawish-akcodes I'd look in what can be reused from the prior PR: https://github.com/spcl/serverless-benchmarks/pull/59/files

I wouldn't try to merge new updates into it as it's quite difficult. Instead, I'd cherry-pick some of the files you find useful.

[MinhThieu145]

Hello @mcopik,

Thank you for outlining the specific benchmarks you're interested in: computation in FLOPS/instructions, memory allocation, storage read/write, and disk read/write. I've reviewed our current benchmark suite, and here is what I found:

• Computation: Our workload/python/function.py benchmark measures computational performance by executing arithmetic operations on numpy arrays.

• Memory Allocation: The memory/python/function.py benchmark is designed to evaluate memory allocation performance by timing numpy array allocations.

• Storage Read/Write: The storage/python/function.py benchmark assesses storage operation speeds, focusing on read/write performance.

• Disk Read/Write: While we don't have a direct benchmark for disk I/O, the disc/python/function.py script performs read/write operations with numpy arrays to and from disk, which might be useful for your disk I/O performance analysis.

Could you please provide more details on the specific improvements or additional metrics you're looking to incorporate? Currently I have a few ideas but I would be appreciate if you have any additional sources I can look into