Extended Metadata

[mmaehren]

Currently, the metadata only accepts one RFC reference but sometimes different RFCs requirements result in the same test.
i.e
If a DTLS implementation chooses to generate an alert when it receives a message with an invalid MAC, it MUST generate a bad_record_mac alert with level fatal and terminate its connection state. from Sect 4.1.2.1. MAC in RFC 6347 results in the same test as If the MAC validation fails, the receiver MUST discard the received record as invalid. from Sect 4.1.2.6. Anti-Replay of the same RFC.

Currently, we are using the "description" field for the RFC quote. We should change it so that each RFC entry has a "requirement" field. The description fields should actually describe what the test does. We should also add an "expected behavior", "limitations", and "impact" field and display them in the report analyzer. The latter should give an idea about the gravity of failing the test (beyond the serverity levels).

[Conradowatz]

Suggestion for a metadata format:

"6347-rMf9lpA6G3": {
  "rfc": [ 
    {
      "number": 6347,
      "section": "4.1.2.1. MAC",
      "requirement": "If a DTLS implementation chooses to generate an alert when it receives a message with an invalid MAC, it MUST generate a bad_record_mac alert with level fatal and terminate its connection state."
    },
    {
      "number": 6347,
      "section": "4.1.2.6. Anti-Replay",
      "requirement": "If the MAC validation fails, the receiver MUST  discard the received record as invalid."
    },
  ]
  "description": "In this test, the behavior with an invalid MAC is tested. A message is sent with an invalid MAC.",
  "expected_behaviour": "The receiver should either send a alert with level fatal or ignore the message and send no response."
  "limitations": "If the recipient chooses to ignore the message, we can not guarantee, that is is not passed through to the application layer."
  "impact": "If messages with a broken MAC is passed though to application layer, or is processed by the TLS implementation, the integrity and secrecy of the connection can be broken.",
  "suggestion": "Validate the MAC of all messages correctly and discard the message without processing if a validation error occurs.",
  "severityLevels": {
    "Handshake": 100,
    "Interoperability": 100
  },
  "tags": [
    "both",
    "dtls12",
    "rfc6347",
    "AntiReplay",
    "invalidMAC"
  ]
}

[mmaehren]

I think we should have two descriptions and expected outcomes: one that presents the intuition and one that is on a technical level. This would address both users and developers. Maybe we could even refactor tests to build the technical description from the test template automatically. For example, extract messages to be sent and received from the workflow trace. For long term development, my biggest worry is that metadata and test won't match anymore at some point.

[ic0ns]

What about other data - for example the parameters that are modified for the test, what values they can take and what they mean? For some its rather obvious - but I assume if you have like specific bit masks or something it can get quite unintuitive.

[mmaehren]

Good point. We can add the parameters automatically. Adding which limitations are applied to the parameters is more challenging due to our DynamicValueConstraints.

[Conradowatz]

We are already writing out the parameter combnations. Or do you want further explanation for every parameter? An example would be nice

[ic0ns]

I am thinking of something like: CipherSuite -> The cipher suite TLS-Anvil will enforce for the connection to the target.

[ic0ns]

A lot of these parameters can probably be collected as "common parameters" - then you just need to list them here and have the description elsewhere - but there are also test specific parameters which would need to be collected here.

[Conradowatz]

Okay that can be done but does not have to be in the metadata of every test