Object detection for dummies

[bartbutenaers]

Dear @yhwang,

Thanks for sharing this interesting node!

I would like to detect objects (e.g. human body or a car or ...) in images from my IP cameras. I assume that is possible somehow with your node?

But unfortunately I have no experience with machine learning... And I have no clue how to implement object detection on images in Node-RED. Where should I find the model (in which format), do I have to train anything, how do I supply the image, and so on ...

Would be very nice if you could add somewhere (e.g. on the wiki of this repository) some kind of step-by-step tutorial, for dummies and idiots like me :-)

Thanks a lot for your time!!
Bart

[yhwang]

Sure! Actually, I just finished my speech to text flow and object detection would be my next goal. I will work on it and share it here soon.

For object dection, Coco SSD would be a good one. And I would use it in my flow. If you are good at JavaScript programming, you can check the documentation in that Coco SSD link. If you prefer the Node-RED and would like to use drag and drop style for composing the data flow, I would provide a detailed step-by-step tutorial later.

[bartbutenaers]

Hi @yhwang,
Thanks for the pointer. Will have a look at it.
Kind regards,
Bart

[yhwang]

I added an example flow here: https://github.com/yhwang/node-red-contrib-tf-model/tree/master/examples/object-detection
and it's for object detection. Hope this may help you.

[johnwalicki]

@ bartbutenaers I've successfully implemented a few flows using the node-red-contrib-tf-model - If you want some assistance, I'd be happy to help.

[bartbutenaers]

Hi @yhwang , @johnwalicki,
Thanks a lot for the great support!!
But I had a surgery this week. Will contact you guys as soon as I'm back into business :-)
Kind regards,
Bart

[viacheslav-dobrynin]

Hello @johnwalicki !
I would like to get a simple linear regression in my flow.
But when I load my model in JSON format, the node throws an error: "Cannot read property 'producer' of undefined".
I searched the internet. As I understand it, the problem is loading the model, but here the node controls it.
I am not a JavaScript developer.
Could you tell me what the problem is?
Thanks in advance.

Best Regards!
Dobrynin Vyacheslav.

[bartbutenaers]

@yhwang , @johnwalicki,
I had completely lost this issue out of sight. My sincere apologies for that!!!
I'm too busy with other Node-RED stuff at the moment, but I will surely follow this discussion!
Thanks!
Bart

[Viggen-l]

Hi @yhwang and @bartbutenaers
I have followed the node-red object detection example and everything is running with the cocco ssd model. But i need to use an other special model, So i trained one at https://teachablemachine.withgoogle.com with the 4 new classes needed.
When i try to load this i get the same error as @johnwalicki: "Cannot read property 'producer' of undefined"
Same error both with link to download page and from file.

Can any one of you explaine what's wrong?

[yhwang]

@Viggen-l when loading different model, you need to know what inputs are needed for the model you are using. The main purpose of tf-model node is to load tf.js models. You have to handle the data pre-processing and post-processing separately. After you specify the URL/file path in the tf-model node and deploy the flow, you should be able to see a log message from the console of Node-RED process telling you what inputs the model needs. Then you have to provide the corresponding named object for each input your model needs in the pre-processing node and pass the data to tf-model node.

Could you share detailed message you see from the console and also the model? Then maybe I can help you to figure out what the problem is.

[Viggen-l]

Hi
Yes all help is appreciated, I stumbled into AI for picture handling a few weeks ago and made some test with python (and it's working with the trained model). But realised that Node-red would be a better solution for maintainace and monitoring in a live system. So know i'am lerning javascript as well :), A lot of new things for the last weeks.

I made a quick sampel model that can be found here https://teachablemachine.withgoogle.com/models/KhW7t95Hm/model.json
There are 4 classes (Class 1, Class 2, Class 3, Class 4) And it's trained on loaded pallets, 1 and 2 are bad pallets, 3 are empty woodpallets, and 4 are correct load (also dogs and other stuff ends up here if you try).

Setup running on a Pi 4 8gb ram.
so first run with the model in the example flow to verify setup and it's working, then changed model.

From debug window in Node-Red
2020-08-13 09:20:43node: COCO SSD lite
msg : string[44]
"Cannot read property 'producer' of undefined"

In console
13 Aug 09:20:15 - [info] Starting flows
13 Aug 09:20:16 - [info] Started flows
13 Aug 09:20:17 - [info] [tf-model:COCO SSD lite] loading model from: https://teachablemachine.withgoogle.com/models/KhW7t95Hm/model.json
13 Aug 09:20:17 - [error] [tf-model:COCO SSD lite] Cannot read property 'producer' of undefined

I think it's beacuse the base models are different. tf-model (coco) are used for objectdetection and teachablemachine produce an object cassification model.

i looked into the example node-red-contrib-teachable-machine (only working on windows) what was setup, but as i can't find the real node-red parts for tf-model code i cant compare.

I hope you can find something. Teachablemachin offers a very easy way of training a model so to be able to run it in Node-Red would be great.

[Viggen-l]

So i actually made some progress today, I found where the node-red code was hiding (index.js dep down).
And the problem "Cannot read property 'producer' of undefined" looks like it's solved, The T.machine model say it's a 'layers-model' type and that's not handel so it was loaded with loadGraphModel.
Added check for that so it's loaded with loadLayersModel and it's happy so far.

Know there is a new problem this model gives an empty input array: [info] [tf-model:COCO SSD lite] input(s) for the model: []
and when and prediction are made it's crashing.
I have no clue where the code for the prediction are made so my journey ends here.

Log from console.
13 Aug 14:15:38 - [info] Starting flows
13 Aug 14:15:38 - [info] Started flows
13 Aug 14:15:38 - [info] [tf-model:COCO SSD lite] loading model from: https://teachablemachine.withgoogle.com/models/KhW7t95Hm/model.json
13 Aug 14:15:39 - [info] [tf-model:COCO SSD lite] loaded model type: layers-model
13 Aug 14:15:39 - [info] [tf-model:COCO SSD lite] Loading as: layers-model
13 Aug 14:15:39 - [info] [tf-model:COCO SSD lite] model loaded
13 Aug 14:15:39 - [info] [tf-model:COCO SSD lite] input(s) for the model: []
13 Aug 14:15:39 - [info] [tf-model:COCO SSD lite] model name: "sequential_4"

Here a prediction are started
13 Aug 14:16:03 - [info] [tf-model:COCO SSD lite] Run as LayerModel:
13 Aug 14:16:03 - [error] [tf-model:COCO SSD lite] Error: Error when checking model : the Array of Tensors that you are passing to your model is not the size the the model expected. Expected to see 1 Tensor(s), but instead got 0 Tensors(s).

[yhwang]

@Viggen-l the information you provided above are awesome. And you're right about the LayerGraph. The checking criteria is the value of format property needs to be 'layers' and apparently the value of the format property in your model is 'layers-model'. I can change the comparison from "strict equal" to "startsWith"

And the input for your model should be an array of tensor and size should be 1 and the image size needs to be [224, 224]. I tried the following pre-processing code in a tf-function node and linked it to the tf-model node:

const image = tf.tidy(() => {
  return tf.image.resizeBilinear(tf.node.decodeImage(msg.payload, 3), [224,224]).expandDims(0);
});

msg.image = msg.payload;
msg.payload = [ image ]; // tensor array with size 1
return msg;

I used bilinear sampling to resize the image to [224, 224]. You can use tf.image.resizeNearestNeighbor if you prefer nearest neighbor sampling.

[yhwang]

BTW, the inference result is the probabilities of your 4 classes, i.e.:

Tensor
     [[0.000408, 0.0019049, 0.0000467, 0.9976404]]

[Viggen-l]

Thank you very mutch for all your the help here.
I will be of for a week vacation so i can't try it ot right away.

If you ever think of dooing an other tutorial about AI and tensorflow, can it be about how to use preetrained models. How to find out what the input/output parameters are like you showed above. That would be greate because most of the information focus on training a model.

again thank's so far. i will notify when it's tested

[Viggen-l]

So back in business, The preprocess are working and are receiving predictions as
Tensor
[[0.005894, 0.0066852, 0.0068979, 0.9805231],],
So far so good,
But the result are always sorted, what ever class picture is sent in the predictions are returned with lowest in [0] and highest in [3].
what do I misunderstand about this result?