I can't figure out how to model this partition mapping the best way.

[isaacsanders]

A vendor has an SFTP server. The files on the server all contain the same type of data. However, the files are split up in different ways.

The data is focused on a specific trait, called trait, there are about 50 different values for trait. The data happens all day, almost every day.

For data from 2010-01-01 to 2020-12-31, the files are split by trait and into two parts with 2015-01-01 being the date on which the split occurs. From 2021-01-01 forward, the files are split only by date, with all values of trait being in the same file, together.

Here is an example file listing:

FOO.part1.ndjson
FOO.part2.ndjson
BAR.part1.ndjson
BAZ.part2.ndjson
date-2021-01-01.ndjson
date-2021-01-02.ndjson
date-2021-01-03.ndjson
date-2021-01-04.ndjson

I currently use two assets to write data from both types of files into the same location. I want to use an external asset to represent this location. I want the partitions of both upstream assets to map to the observed partitions of the downstream external asset.