Scraping BattleTech

We’re mostly board gamers in this house. Sure, we play and enjoy a lot of video games. But board games are a passion. I occasionally blog about what we’re up to, and I should do more of that.

But what none of you know, probably, is that we would very much like to be into the tabletop miniatures games. These are games like Warhammer 40K, Star Fleet Battles and other games of similar ilk where you bring your armies of miniatures and set them across the table from someone else’s army.

Part of my Frostgrave army, AKA “the good guys”

The first game of this sort we played was Frostgrave, for which I printed — and painted — a metric crap tonne of terrain. I used mostly the minis I’d printed after playing Octopath Traveler for my army, with a special custom character for my main wizard. Not sure what units my boyfriend played with. I may have had too much terrain printed for our small kitchen table, actually.

The scenario was fun, don’t even remember who “won” but it wasn’t really important. I was disappointed that we only played that one time.

Time passed and we slowly started accumulating Warhammer 40K, and I set out to learn the rules and enough about the lore to decide upon a faction to play. I finally decided upon the warrior nuns, the Adepta Sororitas. I haven’t really finished or started to assemble or paint an army. It’s such a huge task, that I was waiting for my partner to start with his, but recently, his attention has turned to BattleTech.

He’s making quite an army. At least one night a week, he puts on his magnifying glasses, fills his airbrush, and starts painting. So I think this is going to be the real thing. (I’ll post his army when it’s done, as he stopped blogging years ago).

He showed me, last night, an article he’d found on Reddit about a site that had information about all the 9,000 or so units that have made an appearance in BattleTech’s 30 year history. It’s… it’s a lot. Most of them are variations of some other unit, but still, they are distinct enough that they have their own base stat sheets.

My BF wanted to grab all the information about all these units from that site and drop them into a CSV file, so that he can do some data analysis and figure out what his optimal army should look like. The article included Python code, but it was missing some key elements — like, how to get the list of all the units.

I took a look and figured I could do the whole thing on my own, and so I did. I used one Python library grab the HTML page for a unit, another to parse the HTML, and the CSV package to write it all out. It’s a brute force approach, just generating unit numbers sequentially and seeing if they exist. If not, try the next one.

It took a little debugging, but it’s been churning for the past couple hours while I played Palia.

Anyway, tl;dr: coding, it’s fun.

The scraping wasn’t complete when I wrote this. Now it is; here’s the GitHub repo with both this Python source code and a CSV of the BattleTech stats.

4 thoughts on “Scraping BattleTech”

  1. been there done that.

    Save the networking problem (unless that’s the exercise) look at megamek, and/or mekhq. Java simulator for the tt game. That being said it has unit files for almost all the units, buried in side a zip of a zip. Granted .mtf files are old school \r\n seperated files but the data’s there.

    github/MegaMek/megamek/tree/master/megamek/data/mechfiles/mechs

    • Oh, wow. Okay definitely taking a look at that. I imagine there’s no AI in the simulator?

      But cool, would definitely have been easier to just download the stats. It wasn’t tough to get the data from that website; we were just concerned we’d be stressing out their resources by making thousands of calls. Especially since we noticed the site is not secured, which means it is vulnerable to actual attacks.

      Just took a look at the repo — wow, yeah, I’m going to have to show this to my partner.

      • nope, there’s an AI… it’s pretty good but has no object permenence. (will take the best shot every time, doesn’t stay fixated on a single target) and doesn’t react unless it can see an enemy unit.

        the megamek suite is pretty insane. Game simulator with network capabilities, Mech designer, and campaign manager (the campaign manager can, but by default doens’t, use the published rules, but has it’s own home grown system.)

        You can even print out mech record sheets.

        It’s a 20 year old repo ;), but the megamek devs, and the battletech game developers work hand in hand, testing rules edge cases, etc.

        The UI is bad, and is designed so that you can play if you already know the rules, but not learn them, and the devs keep it that way so it’s a “game aid” instead of a game, that would require licensing from Microsoft.

        Tutorials are on youtube if you need them, but i suggest
        youtube/watch?v=TFENDsevbL0&list=PLHXHV2JDcebr4J3VyJlmU-Yp3cZp_OhWV&ab_channel=StrategosLevel3

        bg.battletech.com/forums/index.php/topic,72636.0.html

        If Megamek doesn’t work for you, for printing sheets there’s SolarisSkunkWerks.
        It’s dev community isn’t as active as Megamek, but it’s a much better UI.
        Specifically designed for building armies, and printing record sheets.
        http://solarisskunkwerks.com/
        it went dormant for a while, but is active again, and it too has files for most units.

        • We just had to do another scraping run last night because my scraper wasn’t interpreting checkboxes correctly…

          Okay, I am definitely going to try and get this running. I think it could be really fun to try it out. Me, I know so little about BattleTech; it’s mostly Kasul’s thing, and I think he’s just preparing to go play up in Enfield or Springfield in a few months. But I’d love to play, too — this could help me get into the game.

Comments are closed.