I agree – it is over-complicated and confusing in my opinion.
How I think they should have done it: each individual punchcard should just be… whatever item level it is, and the effects on each punchcard scale to that item level.
And then the item level of the trinket itself is only of consequence for e.g. your average item level for queuing for dungeons – just set it to the highest ilvl punchcard in the trinket, done.