The past few days have been very productive for me on Spawning Tool on several fronts. Not only did I work through several features, I made some important backend changes. Consequently, I have also thought on future development, product, and business concerns that I think might be of interest to the community.

Let’s start with the backend changes. First, Graylin updated sc2reader to 0.6.0, which brought 2 important changes for Spawning Tool: support for patch 2.0.10 replays and support for GameHeart replays. See my last post for what GameHeart parsing entailed. Importantly, this brought in ASUS ROG Summer 2013 replays and corrected data from various other replays. Second, I migrated from a mysql to a postgres database tonight. This change won’t impact end-users much (other than some load times), but it makes me feel better about the data integrity.

Next up are the new features. First, you can now view counts on the abilities casted in a game so you can see that Protoss players typically use Psionic Storm about 2.67 times a game, whereas they can expect 1.72 EMPs and 1.37 Snipes against them. The data is pretty rough when summarized from so many replays like this, but hopefully more specific queries will yield more interesting results.

You can also specify times to capture particular counts, so you can see that on average, 12.32 SCVs are built in the first 5 minutes of a game. You can now also filter replays by the date played so you can be sure to only consider the latest and greatest replays from your favorite players.

So all of these changes have me looking forward at the direction of Spawning Tool. Over the past few months, I have been talking a lot with ChanmanV, who has provided product direction for Spawning Tool. The current feature set represents what we consider a Minimum Viable Product (MVP) for showing what basic statistics we can pull out of replays. It’s been a strange road, but Spawning Tool today is far more developed than I imagined when I first sat down pulling an all-nighter to just build something. I originally envisioned it as a proof of concept and technical achievement for the community to get excited about. Today, it’s a product when potential value, and I need to re-evaluate it in that context.

The primary change is that the Spawning Tool site is no longer open source. It was originally open source because it was supposed to be a demonstration for the community, and I hoped to pick up collaborators along the way. Well, I didn’t find many collaborators (though I’m still open to it; email me if you’re interested!), and between forums, code contributions, and the still open source spawningtool parser, I hope I’m doing enough. There are lots of great reasons to open source things, but today, the site doesn’t quite fit.

The other aspect is that I need to start looking forward towards a business plan. Before I scare you off, know this: I intend for Spawning Tool to always be free for public, individual use. The big picture goal of the site is to make quantitative thinking pervasive in the StarCraft community, and I don’t want to change that. I thankfully have a day job that I really like, and this is something that I’m doing for fun and for the community on the side. Given that, there are server and development costs to cover. That’s currently trivial (~$30/month), but those will increase with scale. I’m thinking on a lot of different ideas, so let me know if you have any thoughts on it. Hopefully we can get bigger, commercial entities interested and have them throw us some change, but nothing is locked down.

As usual, leave a comment or email me (kevin@kevinleung.com) if you have any thoughts. And if you’re here for my StarCraft strategy content, I’m working on updating my Protoss strategy guide right now. I had forgotten how much cheese you get from Zerg in real laddering, like this 8 Pool. I’ll try cater the content to this type of play.

3 big updates for Spawning Tool this week.

First, you can see units killed and units lost in the research tab. For example, you might be curious to know how many Marines a certain player killed during Dreamhack Valencia or how badly their workers are harassed on average. These stats, of course, are very dependent on how long a game goes and what units composition are, but they’re in there and maybe fun to consider.

Second, I split out the win rates data onto its own page from research, so for any of your favorite filters, you can see how the games go. The most obvious use is to see how a player does in different matchups. It may be unsurprising how good JvZ is (the other numbers may be a bit off because of GameHeart replays; more on that next paragraph), but you might be surprised to see that TLO wins 80% of his games from 21 to 25 minutes long, but only wins 16.7% (1-5) of games longer than that. How strange.

Third, GameHeart support is coming soon. GameHeart games are unusual for a few reasons. First, the players and teams aren’t reflective of the actual matchup because the actual players are only picked in the in-game lobby, so the observers are mixed into the data and races may not be accurate. Second, all events are mistimed because the actual game doesn’t start until the in-game lobby is set. Finally, because the teams are misaligned, the winner may not be marked correctly.

So the fix is coming soon, which I can explain the process of. I mentioned before that Spawning Tool is actually built on top of an open source replay parser. Well, Spawning Tool is actually built on top of sc2reader, an amazing open source library that literally turns 1s and 0s into comprehensible data structures. Since many people may be interested in having GameHeart replays normalized to work like standard replays, this functionality should be added to sc2reader (not spawningtool) since it will have better reach that way. sc2reader was cleverly built to allow for plugins for specific functionality. Since sc2reader is an open source project, I wrote up a plugin to normalize GameHeart replays and submitted it to the maintainers of sc2reader to be incorporated into the primary project. If they like it, they can accept the request, and it will be available for everyone (including me) to use. Anyways, get excited about that because there are a lot of Spawning Tool replays that aren’t quite working correctly.

And on that note, if you’re interested in working with StarCraft data (whether replays, profiles, or anything else), I recommend you check out the new API and Data Analysis Forum that Blizzard made. As you might have guessed, the StarCraft developer community has some amazing people at the forefront, and I think it’s a supportive and generous group of people. If you’re interested in contributing or have an idea, chime in there or check out and contact someone else with a project in this thread. No matter where you’re coming from, we could always use a helping hand.

Dreamhack Valencia is coming up this weekend, and in preparation for that, I fired up Spawning Tool for some analysis. Check out a few observations I made in a TL post http://www.teamliquid.net/forum/viewmessage.php?topic_id=421974

Because Dreamhack releases replays after their events, we actually have a ton of data available for each of these players. I went through the players seeded into Group Stage #2 and checked their performance at Dreamhack Summer. Here’s what I found:

1. Lucifron beat Sonder 2-0 using almost the exact same opening in both games. He opened with 2 Reapers into 3 CC and Hellions. The builds only start to diverge around 7:00 (ref)

2. Harstem went 4-4 (50%) in PvZ. He cannon rushed in 3 of his 4 wins, and in none of his losses (ref)

3. TheStC only played TvT in 1 series, going mech in both games against Morrow in by far his longest games (24 and 31 minutes) (ref)

4. Hyun went 7-1 (82.5%) in ZvT (ignore the issues parsing gameheart games). The games lasted between 9 and 19 minutes, with his 1 loss coming against TheStC in the longest game. His 3 fastest wins, unsurprisingly, came off of Roach-Zergling all-ins (ref)

5. StarNaN went 0-6 in PvZ. Granted, it was against Hyun, Life, and Pig, but YugiOh isn’t going to be much easier (ref)

6. Stephano went 6-2 (75%) in ZvZ, losing only to Hyun. He doesn’t build very many Roaches but does build a lot of Zerglings(ref)

7. Tefel went 8-1 (89%) in ZvP, losing only to Harstem’s cannon rush. He went Roach/Hydra in ALL of these games (ref)

8. NightEnD went 3-1 (75%) in PvZ, using a Gateway Expand build in all of them (ref)

I thought those were pretty interesting, but they also show off some of what you can do with the research tool. You might have noticed that this looks a lot different from the old research tool. After getting some feedback about the valuable parameters and developing an approachable interface, I scaled back the page a bit. The fancier bits about searching over the build orders are still around in advanced search, though I’ll need to revisit that code, too, probably.

Oddly, my work recently has been at 2 extremes. There was a lot of parsing stuff I needed to go through that basically just result in more accurate data and low-level extraction of data and tags. On the other extreme, I worked with Julie to get a splash of purple and revised interfaces. The result is that the actual core functionality of the site (somewhere between those 2 ends) didn’t change much, so not too many new features to hawk. Despite that, the site is in much better shape than before.

Anyways, enjoy Dreamhack. I think we’re on the cusp of some really exciting developments with Spawning Tool as we get more replays and hit the core functionality.

For all Americans, I hope you had a good Memorial Day weekend. If you’re unfamiliar with it, Memorial Day is a federal holiday for honoring departed members of the US armed forces. Given the time of year, many people will go out to grill and enjoy the weather. I did some of that, but I also got started on the new season of Arrested Development and built a number of new features for the Spawning Tool.

First, I added basic authentication so you can login and upload replays. This mechanism isn’t particularly interesting, but it’s necessary to restrict who can edit the tags on a replay. I once thought it best if anyone could edit any tags, but I think that’s inviting vandalism. If anyone feels so strongly to add tags to other replays, we’ll come up with a mechanism to enable that.

Second, I improved the Browse Replays interface. You can now do text search and toggle a few options. The actual list of replays is now much more readable thanks to built-in bootstrap styles.

Third, I actually incorporated more parsed data from the replay into the actual database schema for the site itself. The full explanation requires some context.

The Spawning Tool is primarily composed of 2 parts. First, there’s the spawningtool tool, which actually goes through the replay data and pulls out the build order. The result of this tool is basically just a bunch of structured text. Second, there’s the spawningtool site, which gets this text and turns it into a presentable website.

Previously, I was just taking the raw build order data as a whole and re-rendering it from scratch every time. It was easy and was sufficient for displaying builds. The downside was that the spawningtool site was pretty blind to what the data was. I re-implemented it so that the build order is stored in the database itself: instead of a raw build order in text, the build order itself is stored with times, units, and supply counts in the database.

The biggest benefit is coming soon: more advanced filters and statistics from replays. With this data, we can make database queries that translate naturally into questions like, “If you build a Robotics Facility before 7:00 in PvT, how often will you win?” It’s just a matter of finding all replays with builds that fit the criteria, then counting the wins and losses.

I’m very excited about the next steps for the Spawning Tool, and I think that advanced filters is a big part of that. Additionally, I’m working with ChanmanV to make the Spawning Tool a replay archive for practice games. Tune in at http://youtu.be/vstWWo0Gmao?t=29m30s for more about that.

One last thing: I’m in the process of moving the Spawning Tool off of my personal server and onto Amazon Web Services. As such, there may be some downtime in the near future. Deployment might be rough, but I think this will ultimately lead to a much more stable environment.