BRING BACK THE BELTS

[zardus]

BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS BELTS (on the scoreboard)

[zardus]

One caveat here: because this uses the live, latest requirements, a lot of people who have previously earned belts have been de-belted. Not super sure how to handle this without a separate belts table.

[zardus]

Shall we use ctfd awards to handle belts from prior requirements? We can grant BELT_YELLOW, BELT_BLUE, etc awards (and also use this for emoji). That'll also be easy to work into the various belt checker functions.

[adamdoupe]

Shouldn’t we have a belts table? What’s the criteria for having a belt then if it can be taken away when new content is added?

[ConnorNelson]

Yeah, I think we should use CTFd Awards (https://github.com/CTFd/CTFd/blob/master/CTFd/models/__init__.py#L213).

We can make Belts subclass it.

• name can be color ("Yellow Belt", "Blue Belt", etc)
• category can be the dojo's reference_id (e.g. "program-security")
• date allows us to understand when it was earned, loosely associate it with the version of the dojo from which it was earned.

We can best effort retroactively assign null category / best effort date for prior belts.

---

Also, make sure you don't make emails just suddenly go public.

[zardus]

for now, i'm going to approximate this by just having a cutoff date after which belts become cumulative (and before which they are not). Right now, that's going to be October 10, 2023. We can implement the awards-based model next :-)

[zardus]

OKAY! Provided tests pass, this is ready to go. In summary:

• https://pwn.college/belts is BACK (and links to static.pwn.college/belts for previously-belted people)
• the scoreboards now show the belt level of every person
• belts are cumulative after 2024.10.01

@robwaz one caveat is that I moved your get_user_belts function to dojo_plugin.utils.belts.get_user_belts out of dojo_plugin.utils.dojos. I updated where I saw it being referenced, but could you double-check that I'm not about to break the role boat?

[zardus]

Okay, we also fetch belts out of the awards table now. @ConnorNelson @robwaz could you guys create relevantly-dated awards for people that need them?

Right now, Award-derived belts are explicitly non-cumulative, but we might just want to handle them via the normal date cutoff for cumulativeness or something. Anyways, this code will allow us to make the belts page accurate (just someone has to do the DB footwork).

[zardus]

Also, I'm all subclassing Awards and such (as long as someone else does it); just didn't feel like figuring it out with sqlalchemy

[zardus]

To create the belt entries for all the users who meet the live dojo requirements:

docker exec -it dojo dojo flask

from .. import utils
belt_info = utils.belts.get_belts()
for color,color_data in belt_info['dates'].items():
    for user_id,date in color_data.items():
        db.session.add(Belts(user=Users.query.filter_by(id=user_id).one(), date=date, name=color))
db.session.commit()

[zardus]

The deployment process should be:

1. merge and deploy this PR
2. run the belt award creation code above
3. change get_belts() to only use the Belts table (rather than live-computing belted people)

[zardus]

Other related TODOs:

• add access control to orange belt dojo
• move assembly crash course to orange belt dojo

[zardus]

TIME TO YOLO

[zardus]

For the record, the script to create belt roles from the discord data:

import datetime
import csv

roles = csv.DictReader(open("/tmp/belt_roles.csv"))
for entry in roles:
    discord_id = entry["member.id"]
    role = entry["role.name"].split()[0]
    if role not in [ "Orange", "Yellow", "Green", "Blue" ]:
        continue
    discord_user = DiscordUsers.query.filter_by(discord_id=discord_id).first()
    if not discord_user:
        continue
    user = discord_user.user
    date = datetime.datetime.fromisoformat(entry["role.created_at"])
    print(user, role, date)
    db.session.add(Belts(user=user, name=role.lower(), date=date, description="derived from discord role logs"))

db.session.commit()

[zardus]

And the script to create belt roles from challenge completion data:

from .. import utils
belt_info = utils.belts.get_belts()
for color,color_data in belt_info['dates'].items():
    for user_id,date in color_data.items():
        user = Users.query.filter_by(id=user_id).one()
        if Belts.query.filter_by(user=user, name=color).first() is None:
            print("CREATING BELT!",user,color)
            db.session.add(Belts(user=Users.query.filter_by(id=user_id).one(), date=date, name=color))
db.session.commit()

[zardus]

Wiping the bad discord dates:

import datetime
bad_dates = [
    datetime.datetime.fromisoformat("2020-12-05 05:24:48.458000"), #blue
    datetime.datetime.fromisoformat("2020-12-05 05:58:25.987000"), # yellow
    datetime.datetime.fromisoformat("2020-12-05 05:58:25.987000"), # orange
    datetime.datetime.fromisoformat("2024-01-08 22:10:54.756000"), #green
]
for date in bad_dates:
    for award in Awards.query.filter_by(date=date):
        award.date = None
db.session.commit()

[zardus]

Importing belt data (including dates) from the legacy belts page:

import yaml
import datetime
for color in [ "yellow", "blue" ]:
    belts = yaml.load(open(f"/tmp/{color}.yml"))
    for b in belts:
        if b["version"] != "v2":
            continue
        u = Users.query.filter_by(id=b["id"]).first()
        d = datetime.datetime.fromisoformat(b["date"])
        award = Awards.query.filter_by(user=u, name=color).first()
        if award is not None and award.date is not None:
            continue
        print(f"{color} {u.name=} {b['handle']=} {bool(award)=}")
        if award:
            print("... updated date")
            award.date = d
        else:
            print("... added missing")
            b = Belts(user=u, date=d, name=color, description="importing from old belts page")
            db.session.add(b)
db.session.commit()

[zardus]

Using last-solve time for still-missing Green and Yellow belts (blue is harder because of how it's changed):

missing_dates = Belts.query.filter_by(date=None, name="yellow")
dojo = Dojos.from_id("program-security").first()
for award in missing_dates:
    last_solve = dojo.solves().filter(Solves.user==award.user).order_by(db.desc(Solves.date)).first()
    if not last_solve:
        continue
    award.date = last_solve.date
db.session.commit()

missing_dates = Belts.query.filter_by(date=None, name="green")
dojo = Dojos.from_id("system-security").first()
for award in missing_dates:
    last_solve = dojo.solves().filter(Solves.user==award.user).order_by(db.desc(Solves.date)).first()
    if not last_solve:
        continue
    award.date = last_solve.date
db.session.commit()

[zardus]

The last recovered dates, from manual sleuthing (mostly discord).

discord_blue_dates = {
    "m0nst3r": datetime.datetime.fromisoformat("2023-07-31T08:09:00"),
    "zammo": datetime.datetime.fromisoformat("2023-05-05T18:00:00"),
    "sh0n00b": datetime.datetime.strptime("September 23, 2023 4:12 AM", '%B %d, %Y %I:%M %p'),
    "pipironi": datetime.datetime.strptime("August 19, 2023 11:10 AM", '%B %d, %Y %I:%M %p'),
    "NullPoExc": datetime.datetime.strptime("November 7, 2023 12:11 AM", '%B %d, %Y %I:%M %p'),
    "Bena": datetime.datetime.strptime("November 21, 2023 5:27 AM", '%B %d, %Y %I:%M %p'),
    "Canlex": datetime.datetime(2024, 1, 3, 11, 15, 34, 258442),
    "Gromji": datetime.datetime.strptime("August 26, 2023 10:43 PM", '%B %d, %Y %I:%M %p'),
    "whoamiamleo": datetime.datetime.strptime("May 17, 2023 5:59 PM", '%B %d, %Y %I:%M %p'),
    "KatsuragiCSL": datetime.datetime(2023, 12, 2, 7, 48, 9, 887747),
    "X3ero0": datetime.datetime.fromisoformat("2020-12-08T12:05:15"),
    "dorcelessness": datetime.datetime.strptime("April 22, 2023 4:30 AM", '%B %d, %Y %I:%M %p'),
    "DeathByGunboat": datetime.datetime.strptime("May 2, 2023 11:10 PM", '%B %d, %Y %I:%M %p'),
    "0xADE1": datetime.datetime.strptime("October 19, 2023 6:44 AM", '%B %d, %Y %I:%M %p'),
    "Elvis": datetime.datetime.strptime("April 27, 2023 10:55 PM", '%B %d, %Y %I:%M %p'),
}
for name,date in discord_blue_dates.items():
    u = Users.query.filter_by(name=name).first()
    assert u is not None
    belt = Belts.query.filter_by(user=u, name="blue").first()
    assert belt is not None
    assert belt.date is None
    print(f"Updating {u.name=} {date=}")
    belt.date = date
db.session.commit()

Currently trying to get this to actually apply, but it's hanging on the first update for some reason.

[zardus]

Okay, we have two yellow belt awards without dates remaining:

award.user.name='KatsuragiCSL'
award.user.name='X3eRo0'

Both of these hackers have blue belts now, so I'm not going to worry about it.

[zardus]

And some code to port over belts for users that match from the V1 database:

import csv
import yaml
import datetime

v1_data = csv.DictReader(open("/tmp/users_v1.csv"))
email_matches = { }
name_matches = { }
unmatched = [ ]
v2_users = Users.query.all()
v2_emails = { u.email: u.id for u in v2_users }
v2_names = { u.name: u.id for u in v2_users }
for e in v1_data:
    v1_email = e['email'].strip("'")
    v1_name = e['name'].strip("'")
    v1_id = int(e['id'])
    if v1_email in v2_emails:
        email_matches[v1_id] = v2_emails[v1_email]
    elif v1_name in v2_names:
        name_matches[v1_id] = v2_names[v1_name]
    else:
        unmatched.append(v1_id)



unmatched = 0
matched = 0
for color in [ "yellow", "blue" ]:
    belts = yaml.load(open(f"/tmp/{color}.yml"))
    for b in belts:
        if b["version"] != "v1":
            continue
        if b["id"] in email_matches:
            u = Users.query.filter_by(id=email_matches[b["id"]]).first()
            matched += 1
        elif "mail" in b and Users.query.filter_by(email=b["mail"]).first():
            u = Users.query.filter_by(email=b["mail"]).first()
            matched += 1
        else:
            print(f"NOMATCH: Failed to find a user for {b=}")
            unmatched += 1
            continue
        d = datetime.datetime.fromisoformat(b["date"])
        award = Awards.query.filter_by(user=u, name=color).first()
        if award:
            print(f"SKIPPING: {color} {u.name=} {b['handle']=} {award.date=}")
        else:
            print(f"MATCH: {color} {u.name=} {b['handle']=} {bool(award)=}")
            db.session.add(Belts(user=u, name=color, date=d, description="user-matched to v1 database"))
db.session.commit()