Significant RAM and CPU impact

[bengotow]

Hey folks! I installed Scout APM on our service last night (deployment 6dca in the graphs below) and have noticed some really significant RAM and CPU impacts that will force us to disable it again for now. I'm sure this is NOT typical but our application isn't super unusual so I wanted to bring it to your attention. Most interestingly it does not seem to be correlated with volume of traffic to the service - this is actually nice because I think it narrows down the problem quite a bit.

Service details:

• Express application running NodeJS 12 using TypeORM and Postgres (pg)
• Fairly low traffic app - no more than 10 requests / second typically.
• Also using Sentry for error reporting and Sqreen (https://www.sqreen.com/) a security agent our client requires.
  + I think this Sqreen service could be interacting with ScoutAPM, both do sketchy observational stuff. ;-)

When the app launches it calls setInterval to schedule a few recurring tasks, and it also calls an async function that sits in a loop processing data from another system forever. I unfortunately can't share the code, but it more or less looks like this. I suspect that somehow this is causing Scout to keep data/context around forever - there's nothing else I can think of that would increase RAM usage by 500MB.

If I get a chance I can try to make a sample project to repro but I don't think I'll have time in the next week.

# main.ts

async function pollQueue() {
  while (true) {
     // use `request` module to hit an internal API
     // use TypeORM to make several pg queries and process results, queue more jobs
     await sleep(500ms)
  }
}
pollQueue()

[bengotow]

Hey folks, just to be sure this wasn't a bizarre interaction between Sqreen (company mandated security monitoring service) and ScoutAPM I turned off Sqreen and did another deployment. Unfortunately it doesn't seem to have made a difference, RAM usage is still spiking to more than double our previous RAM usage.

Hey @bengotow thanks for reporting this -- a few questions:

• What version of Scout are you using?
• How are you instrumenting the tasks you're keeping around? Are you using scout.transaction(...) manually?
• As for the data processing bit -- are you doing any database calls? Also while the main app doesn't receive many requests per second, how many times would you say the data processing runs?

I share your suspicion that it might be traces staying around, we just released a new version with some memory fixes so I'm curious to see if this is a separate issue.

[bengotow]

Hey! My lock file is showing it's version 0.2.0. I was just getting the integration going so I hadn't done any manual instrumentation yet. Just running the scout install command and then adding the middleware to the express app.

I think the queue processing the app was doing in the background only performed about a dozen queries a minute, but every minute or so it would write about 200kb of updated JSON into the Postgres database so it's plausible that holding on to data could have achieved the memory usage.

I queried the logs just now and it turns out the spikiness of the graph (when memory usage goes down) was just caused by one of the dynos running out of RAM and crashing with Allocation failed - JavaScript heap out of memory so I expect there isn't any garbage collection or cleanup happening within the processes. I can't seem to get single-dyno graphs from Heroku to verify.

Thanks for the clarifications! Would you mind sharing what size dyno this is? Judging by the ~.5GB memory available I assume this is a Standard-1X? The agent does actually run a light weight subprocess by default (the core-agent) I wonder if the increase in usage is related -- If your app was idling at around 300MB and the agent takes anything more than 200MB (which it very likely doesn't) then we've blown the budget.

Were you able to get any metrics? At present the integrations that could possibly be related are express and pg, I wonder if there are any others that I should take a closer look at.

@bengotow could you share the structure of the code for the async tasks? I'd like to write a small application to simulate what you're doing -- is it something like:

import scout from "@scout_apm/scout-apm";
import express from "express";
// other imports/requires

const app = express();

app.get('/regular-endpoint', ...);
app.post('/regular-endpoint', ...);
// ... other endpoints ... 

// Set up the database then run recurring work and start express application
setupDB()
  .then(db => {
    setInterval(() => {
    // .... do periodic listening ...
    }, intervalMs);

    app.listen();
});