Understanding "Data Traffic Quality" and possibly decoding unknown telegrams

[ralfbergs]

My "Data Traffic" quality is almost perfect (not surprising as the bus length is just a bit more than 1 m), according to the stats:

I would first like to further understand the exact meaning of the above values.

Is it true that the EMS Telegrams Received (Rx) tells how many telegrams have been received in total, with their telegram "integrity" intact, i.e. no transmission error?

And is it true that the number of EMS Reads (Tx) are part of the number of EMS Telegrams Received (Rx)?

What does it then mean if the number of EMS Reads (Tx) is much higher than the number of EMS Telegrams Received (Rx)? Does it mean that those "failed" EMS Reads (Tx) were received "intact" (flawless reception of "raw telegrams"), but could not be interpreted/decoded properly? Or is it due to some corruption that was not detected by the underlying transport?

Here's a log of dozens of such failures. If I look at it (and I admit I have zero clue about EMS datagrams!) one could think that it's always the same "entity" or "opcode" (no clue if this is the proper language) with slightly changing values, and that EMS-ESP may be unable to decode the entity/opcode and its changing values.

Is this a correct guess? Or is it something completely different?

My goal is not to get rid of the tiny fraction of "failures" (100% quality is good enough for me 😎), but my goal is rather to improve the firmware, if possible. I would love to help, if I can.

Is the above log snippet sufficient, or do we need some more low-level log/network trace?

[MichaelDvP]

I think you know the structure of ems telegrams (https://docs.emsesp.org/EMS-bus/)
Each telegram ends with a CRC and when receiving a telegram and the CRC check fails it is ignored and marked as rx error.
The busmaster sends, and the clients receive, telegrams in voltage modulation (12V+-2,5V) on the unshielded 2-wire bus. Voltage modulation can be easiely disturbed by EM events like switchnig inductive load (motors/transformators) nearby. So some rx errors are normal, depening on cable length and if there are power lines nearby.

If a client sends a telegram, this can be a read, write or broadcast. Ems-esp does not have own values and do not broadcast anything.

When sending a read request (dest with 8'th bit set), the dest-device should answer with the requested value.
E.g. ems-esp sends 0B 88 33 02 01 (+crc) requesting the boiler dhw seltemp, the boiler should answer 08 0B 33 02 3C (+crc) for 60° dhw seltemp. If there is no answer, ems-esp try again and mark this read request as read(tx) error after 3 tries. If the boiler answers, but the answer has a crc error, it is also a rx error. But mostly it is missing answer and gives no rx errors.

If a client sends a write command, e.g. 0B 08 33 02 3C to set dhw to 60°C, the dest should answer with a ack or nack byte. If there is no answer, or nack, the write (tx) error is count up. The ack/nack is not a complete telegram, has no crc and will never count as rx error.

[ralfbergs]

Maybe I was too quick?

The ratio is increasing... Let's see where we end up when we have around 190K sucessful EMS Reads (Tx)... If the number of fails is then indeed considerably lower than 171, you might want to include your revised code...

[VlastiBroucek]

@ralfbergs, are you sure it's not your cable? It's not always about length but quality too. Mine is significantly longer (at least 5m) and I am getting

and to be honest, I cannot remember ever seeing any fails.

I vaguely remember that changing the polarity of the cable sometimes worked???

[ralfbergs]

@ralfbergs, are you sure it's not your cable?

Yes. If you recap what we discussed further above, then the "line noise errors" would show up under EMS Telegrams Received (Rx), and that metric is almost perfect.

The "large" number for EMS Reads (Tx) seems to be due to the fact that a bus device seems to be slow ("overloaded?") to respond.

I suppose your bus hardware (boiler etc.) may be simpler more powerful, more recent. What kind of equipment you got, how old? Mine is about 14 years old...

[VlastiBroucek]

The "large" number for EMS Reads (Tx) seems to be due to the fact that a bus device seems to be slow ("overloaded?") to respond.

The words "seems" are the key here... You cannot prove it, it's just speculation.

Did you put digital analyser or at least an oscilloscope on the line to prove that?

I suppose your bus hardware (boiler etc.) may be simpler more powerful, more recent. What kind of equipment you got, how old? Mine is about 14 years old...

Yes, mine are more recent, but also communicating lot more than older devices...

So, I still think it might be worth of testing polarity switch of the EMS cable. Worst case scenario, it will stop working completely, hence sort of confirming the speculation above, or best-case scenario, will eliminate the errors.

I don't think it is the best idea for @MichaelDvP changing what has been working for so many people for so long just because of one case, yours...

[ralfbergs]

The "large" number for EMS Reads (Tx) seems to be due to the fact that a bus device seems to be slow ("overloaded?") to respond.

The words "seems" are the key here... You cannot prove it, it's just speculation.

Well, "is" would have been more appropriate. We had discussed this here in this thread. Please look it up.

So, I still think it might be worth of testing polarity switch of the EMS cable.

There's no point. The KPI for the physical transmission quality (EMS Telegrams Received (Rx)) is close to "perfect", there's no point in trying to reverse the cable.

I don't think it is the best idea for @MichaelDvP changing what has been working for so many people for so long just because of one case, yours...

I cannot see how making the detection logic more "tolerant" in the way @MichaelDvP described can degrade things, but I'm definitely not the expert here, but @MichaelDvP is...

[ralfbergs]

I've now approximate the same number of EMS Reads (Tx) as in my original message here in this thread, but only a fraction of fails (less than 25%):

@MichaelDvP So I would indeed say the code change to make the code more tolerant is doing its job. Feel free to roll it out to production...