Hello everyone,

I've been reading about this duplicates issue so I tried to add my grain of sand to the problem. I ran some deduping of the full themes list and it seems that out of 6327 theme suggestions, only 927 are exact duplicates, which is below 15%.

The thing is removing duplicates is not an easy task, because you have to find: + a) Exact matches + b) Semi-Exact matches: themes that are the same one but misspelled or with a special symbol like a "." at the end + c) Themes that are extremely similar and can be clustered together

C is not as simple as removing exact matches. I think, since LD is Open Source, we can do something about it.

So, if we are going to add something for future jams, we should probably think about preprocessing the themes list.

IMO Preprocessing should:

• 1) Remove exact and "Semi-exact" matches but keep the number of appearances
• 2) Cluster similar themes under one unique entry +3) In the UI:
  • 3.1) Show themes along with the number of appearances
  • 3.1) Show clusters of themes that are very similar under one unique entry

This way no theme is removed and all similar ideas can be approved or denied all at once.

It's just an opinion, let me know what you think

Cheers!

Edit: If anyone wants the full list as an spreadsheet here it is

https://docs.google.com/spreadsheets/d/1xfigRLvo8Y97TI9oyteUyGiFoRAC7pQ88MLFftme44U/edit?usp=sharing