Multiple profiles files and TulipaClustering

[abelsiqueira]

Currently our representative periods profiles are separated into files/tables like profiles-rep-periods-<profile_type>.
Inside that file we have the profile_name to further separate the different profiles.

TulipaClustering does the clustering using all profiles. So, if we follow the same strategy as above, we would need to join the profiles into a single table, only for them to be split again after clustering.

It seems to me like the best solution is to change TEM to only use profile_name and incorporate the profile_type into that name. That way we only have a single table for full profiles and another single table for representative periods profiles.

The alternative would be that TulipaClustering handles separate profiles as input and generate separate profiles as output, all of this efficiently, which is harder.

Also, the motivation to split the profiles in the first place is not clear to me anymore (I didn't go looking).

[datejada]

Hi @abelsiqueira there is no particular reason for having them separated. So, I would say that we should aim to what makes the workflow more lean and straigtforward.

My only consideration, that might be more in the Tulipa Clustering package, is that the user should be able to select which profiles go to the clustering and which don't for the selection of the representatives. Although, all of the profiles will be part of the representative. For example, consider having three profiles: solar, demand, inflows. I want to select my representatives with only the solar and demand profiles. Once I get the representatives, that representatives will contain the solar, demand, and inflows profile of the days that have been selected as representatives. But, the user should be able to also to determine the representatives using the three profiles. Most probably they will get a different representative that also has its own solar, demand, and inflows profiles.

If this makes sense to @greg-neustroev too, I think we should go forward with having only one file to make it easy in both TulipaClustering and TulipaEnergyModel.

[abelsiqueira]

That does make sense to be an option of TulipaClustering because we want all the profiles in the input to be converted to representative periods profiles.
To work only on the solar and demand will require filtering of the table. We either:

• filter on the profile names (user calls cluster(table, [:Asgard_Solar, :Midgard_Demand]));
• create a new column for types and filter on the types (user calls cluster(table, [:Solar, :Demand]), but also has to inform this new column)
• filter on user given filtering function on whatever the user add (user calls cluster(table; filtering=...), and does whatever he wants with the table, including adding the new types column).

The profile types column mentioned above should not be a key though, I think.

[datejada]

Great! To move forward, we can continue with one table and create an issue in TulipaClustering to create the filter later. Unless @abelsiqueira think it is better to include it now.

@greg-neustroev, any extra thoughts/caveats?

[abelsiqueira]

After some work, it looks like keeping the profile_type and profile_name separate might be better, although we should still join all of these file in a single file to match with TulipaClustering. The reason for that is that we actually need the profile_type - which, in retrospect, is obvious - so making it easier to query that is better.

[clizbe]

Addressed by #668