New benchmarks and applications

[mcopik]

In SeBS, we provide a representative set of functions and have developed a set of serverless workflows that will be included in the upcoming release. However, the serverless field is constantly changing, and new types of applications are being "FaaS-ified". SeBS will benefit from new type of functions, new workflows, and from new applications - the last type has not been considered for SeBS yet.

Functions

The current list of functions is available in the documentation.

New ideas (all should be rather simple to implement thanks to a large number of open-source implementations):

• ML - More ML examples (different Resnet versions, BERT, etc.) - we can use Ml-Perf as a reference; we can use VGG, AlexNet, BERT - import code from MIGnificient benchmarks.
• Utilities - PDF generation and conversion seems to be a common - headless Chrome to generate PDF of a webpage (maybe with Browsershot?), or convert PDFs to different formats (Calibre? pdfkit for Node.js?). Example article with browsershot.. Also possible to generate screenshot of a website.
• Utilities - scanning object storage for viruses; another example.
• Webpage/Utilities - QR code generator.
• URL shortener - example 1, example 2.
• Web scraping - example
• Processing Excel/CSV files - excel file, csv files, DuckDB integration - very interesting; another example and one more.
• Visualization and rendering - Blender example and Mermaid example.
• Two collections of serverless snippets - there might some interesting applications there. ServerlessLand and FunctionResources.
• New scientific applications - examples from Lithops, sentiment analysis and mandelbrot.
• Database queries - deploy SQL database?
• LLM inference - one motivation case, C++ version.

Workflows

The current list of workflows is in the PR #88 and in the related thesis. In the PR, we have workflows for MapReduce, video analysis, ExCamera, and ML fitting. The thesis also documents the abstract language we use to specify each workflow.

To extend SeBS, we want to cover new application types and rich workflows with new computational patterns.

Potential new ideas:

• ServerlessLand snippets
• Simple webapp application - todos.
• Another webapp - airline flight booking
• Pywren has several linear algebra applications. It is not written as a workflow, but turning a Cholesky or matrix-matrix multiplication into a workflow with pre-defined schedule would be very interesting.
• AFCL has two interesting benchmarks - we should analyze if they can be expressed properly in our system to run on AWS Step Functions or Azure Durable.
• cbl-translate is a very interesting, rich and complex ML inference workflow that utilize several different models. The workflow uses external APIs such as DeepL, but we should be able to not rely on them - either deploy an existing open-source model for this task instead of an external API or remove that feature entirely.
• maskopy - another interesting utility application that runs on AWS Step Functions.
• Compilation benchmark - examples are in the gg paper. We do not need generality, but an example of offloading compilation steps.

Applications

Our benchmark suite contains functions and workflows, but it does not contain full applications not written as workflows. This can be standalone applications offloading certain tasks to serverless, and fully serverless applications.

• Simple web applications - one motivation example, another one, another one.
• Black-Scholes from PARSEC benchmark suite - we already have code for a serverless invocation that should be integrated soon.
• Microservices are an extremely important workload to the cloud, and several authors have attempted to port them to serverless. This usually requires adding database for state, and replacing RPC with queues and triggers. DeathStarBench is a great source of microservices - SocialNetwork might be a great candidate and it's in C++ for which we have provisional support. Another paper describes porting several microservices - the paper does not come with a code, but some of these application should be open source, e.g., Overleaf.
• Another work is focused on benchmarking applications built on top of serverless triggers - one of the examples might also be new to SeBS and interesting. So far, we have not explored asynchronous triggers in SeBS.
• Serverless Google Maps is open source.
• Simple webapp application - todos.
• Another webapp - airline flight booking
• Pywren has several linear algebra applications.
• cbl-translate is a very interesting, rich and complex ML inference workflow that utilize several different models. We can represent this as an application and remove the dependency on an external ML API.
• Hello-retail is an interesting one. The original repo is gone, but forks are available one and [another](https://github.com/SimonEismann/hello-retail]. Papers using it: 1 and 2

[Rajiv2605]

@mcopik Thanks for creating this issue. I have some doubts:

1. What are the expected number of benchmarks you are looking for in the GSoC period?
2. At what detail should a GSoC proposal address the adding of new benchmarks?

[mcopik]

@Rajiv2605 It depends on their type - functions should be fairly straightforward, while applications can take a lot of time to port and test for correctness. I think that once you create a schedule with milestones, it will be much more clear.

When it comes to the second question, it should be clear from the proposal how you are planning to approach the transition to SeBS - is their open-source implementation with an appropriate license, do you plan to implement it from scratch, how much work will be involved, do you foresee any potential issues, etc. In the proposal, there's no need for deep technical details about each benchmark, but an assessment that the application is interesting as a benchmark, novel for SeBS, viable technically to be used as benchmark, and an estimation of the difficulty and time commitment.

[Rajiv2605]

@mcopik Will we be working on our deliverables during the community bonding period or does the coding start only after it? I am confused if I can include the community bonding period to the coding period while deciding upon the milestones and the amount of work in the proposal.

[mcopik]

@Rajiv2605 Working on deliverables during the community bonding period is not required, but it's a great time to do research, work with other libraries/projects and ensure they work as expected.

[mcopik]

Adding one more interesting application - serverless maps from HackerNews.

[lawrence910426]

I think Black-Scholes from PARSEC benchmark suite could be very interesting for me. I have some experience in Dertivative Pricing & HPC. QuantLib also implemented BS-Model (and similar models).

[mcopik]

@lawrence910426 As I said above, we already have code for that :-) However, other Monte Carlo simulations might be a great addition! If you know some other examples from QuantLib, particularly those with different I/O and computational intensity, and the library's code allows us to integrate example to SebS, then I think this might be a great and interesting addition.

[mcopik]

Recent PRs added support for new benchmarks; we need to test them and merge.

• Utility: OCR (PR Add benchmark - ( 312.OCR ) #219)
• Utility: PDF generation (PR Add 321.pdf-generator benchmark and its data #217)
• ML: image caption generation (Add 421.image caption generator benchmark and added its data in bench… #218)