[QUESTION] Share object between threads ¿Transfer?

[manujoz]

I have an application that works with a really large object on some occasions. Currently I am using native nodejs worker threads to perform tasks with a lot of CPU load, but also, these tasks are performed every 70ms since this data is extracted from a game and in the game it is updated every 1ms.

When the game sometimes works with really large objects, I run into a bottleneck when I send the object via postMessage from a thread to the main and vice versa. It should be noted that I cannot send portions of the object since to perform the tasks in the worker thread I need all the data.

I'm looking for a way to work with the object in a buffer so that when I make a modification in the main thread this modification is in the worker thread and the worker can perform the calculations. I have tried to use packages like "@bnaya/objectbuffer", but for some reason it fails when I try to access for example a non-existent index of an array inside some object property.

I put a simple example of what the application does to see if you can give me a hand to achieve optimal performance. I have been looking at the Transfer property but I cannot understand it very well since I have never worked with buffers and I do not achieve the result I am looking for.

// Main Thread

const myObj = {
  players: [],
  weather: {
    sky: "",
    airTemp: 0,
    winSpeed: 0,
    raining: false,
  },
  dataReduced: null,
}

function dataRecivedFromGame(data) {
  myObj.weather.sky = data.enviroment.sky;
  myObj.weather.airTemp = data.enviroment.air_temp;
  myObj.weather.winSpeed = data.enviroment.win_speed;
  myObj.weather.raining = data.enviroment.is_raining;

  data.players.foreach(player => {
    if(!myObj.players[player.idx]) {
      myObj.players.push({
        name: player.name,
        type: player.type,
        weapons: player.weapons,
        weaponsTrajectory: {},
      });
    }
  });

  myWorker.postMessage(myObj);
}

myWorker.on("message", res => {
  myObj.players = res.players;
  myObj.dataReduced = res.dataReduced;
});

// Worker

let myObj = null;

function calculateWeaponTrajectory() {
  for(let i = 0; i < myObj.players.length; i++) {
    const player = myObj,players[i];

    if(!player) {
      continue;
    }

    // Calculate weapon trajectory...
    for(let w = 0; w < player.weapons.length; w++) {
      const weapon = player.weapons[i];

      if(player.weaponsTrajectory[weapon.name]) {
       continue;
      }

      // cons trajectory = {...calculatedProperties};
      player.weaponsTrajectory[weapon.name] = trayectory;
    };
  };
}

function getDataReduced() {
  // Do thigs to get important data....
  // const dataReduced = {...};
  myObj.dataReduced = dataReduced;
}

parentPort.on("message", response => {
  myObj = response;

  calculateWeaponTrajectory();
  getDataReduced();

  parentPort.postMessage(myObj);
});

As you can see in the example, I have a data object that is filled with the data that the game gives me, but the game gives raw data that I need to process. In order not to block the main thread since they are calculations with a high CPU consumption I derive them from the main thread to the worker. Once the worker has done all the heavy calculations, I return the data to the main thread to continue working with them.

This is just an example of what my application does, but in reality the game and the object that I have can be converted according to the number of players online into a really large object that can weigh 80Mb.

Obviously such a large object added to a fast refresh rate creates that bottleneck by serializing the object to pass it to the worker and vice versa.

Looking for solutions I found the package @bnaya/objectbuffer which is the perfect solution if it was not because it fails as I said before, looking further I found this library. (I take this opportunity to congratulate you on the work since I am sure that I use it a lot from now on), but I don't know if it can be adapted to my application. I've seen the Transfer property, but like I said I'm dumb working with buffers and I've done a couple of tests and it obviously doesn't allow me to transfer the object directly.

Could someone give me a nudge on how it could be used as I am sure there is a solution here that eludes me?

[andywer]

Hey @manujoz! Thanks for the kind words.

This issue is not as easy to solve as one might think.
Things to try:

• do the network communication in a worker, so to avoid needing to share the received data between threads
• not parse the HTTP data in the communication thread, but only get the raw data as an ArrayBuffer which is transferable using Transfer()