Whenever you put your game out into the wild, you can never be quite sure what you’re going to learn from it.
The Jetsam v0.2 Beta was no exception.
The Beta Data
This time around, we were prepared to gather hard data through an in-app logging call that dumped delicious data into a Google Spreadsheet every time a player successfully completed a level – and every time they failed.
From this data, we were able to capture some interesting statistics. As of the time of writing, the Jetsam Level Editor Beta had:
48 Unique Players
219 Unique Levels Played
91 Hints Used
1573 Unique Play Sessions
4 distinct versions:
Identifying Difficulty Cliffs
Given the linear structure of our game, our expectation was that level completion numbers would start high at the beginning of each world and dwindle as the levels got progressively harder – but actual completion numbers told a different story.
We had ourselves a cliff – the dropoff from W1-D to W1-E was severe, over 33%. Were people getting bored with our game after four levels? Was the difficulty ramping too quickly? We asked these questions of our testers, many of whom corroborated the common complaint that “it got too hard, too fast.”
We dug into failure data to see if that complaint held water – and lo and behold, we found that 11 of 17 unique W1-E players failed at least once when playing it!
W1-E – does it look difficult to you?
Armed with quantitative confirmation of our qualitative feedback, we redesigned several levels from W1-E onward and launched a new version, which performed thusly:
This time around, the dropoff between 1-D and 1-E was far less dramatic. Failure data also bears this out, as 0 of 6 unique W1-E players failed it in this later trial. Chalk one up to the power of data to make our level design better!
While not really the objective of our Beta test, we tracked the number of people who used Hints to get past tough levels, and reached an interesting conclusion: almost no one was using them (only 3 unique users had even bothered). This made intuitive sense, as the Hint menu wasn’t an obvious part of the UI by any means.
As a result, in v0.2.3 we added a Hint Tutorial that draws attention to the Hint menu in the UI after a player retries a level for the first time. Subsequently, Hint usage went up 75%. That simple little tweak has opened our mind to future experiments we want to perform around Hints in the v0.3 Beta.
Since we’re launching our game on iOS, we figured we’d capture data about the devices our players were using.
Perhaps unsurprisingly, the iPhone X led the pack – but the iPhone 7 had a strong second-place showing.
Most surprisingly of all, no one played our game on an iPad, even though we have Universal iOS device targeting set up. Maybe that means we should spend less time making our game iPad-friendly?
Running an open-ish Beta for nearly 50 people was a lot of fun, and taught us tons about our game’s weak points. Being able to comb through the level completion and failure data really helped us tune the difficulty curve better, hopefully leading to stronger player retention. Identifying our Hint issue certainly saved us headaches later down the road with monetization. Realizing that there aren’t as many iPad users also helped us better prioritize our development backlog.
Designing a game is great fun. As the designer, you get to curate the player’s experience from the moment they start their adventure. You can weave an intricate story around them, set them up for teaching moments, or put them in situations where their skills are tested. That ability to meticulously plan every detail – every high and every low – allows a well-designed game to become a beautiful, flowing conversation between a game’s designers and its players.
But when you’re designing a sandbox game, it can feel a lot more like handing your players loaded guns and hoping like hell they don’t shoot themselves in the foot.
As such, we’ve developed a comprehensive plan to prevent our obnoxiously feature-rich content creation system from sucking the fun out of our game. Our “sandbox disaster mitigation strategy” boils down to the following three tactics: Player Education, Helicopter Parenting, and Protocol Robustness.
This is the obvious one – of course you should teach your player how the game works, duh! But Player Education transcends simple design principles like Learner Levels. The entire structure of Jetsam’s main story has been built around the principle that players need to be educated really thoroughly.
Jetsam is a puzzle game based around a simple mechanic – you can only move in one direction at a time, and you slide indefinitely until you collide with something that brings you to a stop. In each level, you must collect a Relic and escape to the Exit. All the game’s puzzles are based around the awkwardness of movement in this environment.
To complicate matters further, other obstacles show up in the game’s various worlds. Fuel batteries give you extra moves, gravity wells stop you cold, boxes shatter on contact, wormholes preserve your momentum and warp you across the level, and deadly buzzsaws destroy you. The game’s main story thus begins with six distinct “worlds,” each of which emphasizes a certain mechanic and teaches the player all kinds of dirty tricks for dealing with it.
That’s a lot of information for a player to retain though – so occasionally, a game mechanic from a previous world is re-introduced alongside the current world’s primary mechanic to keep the player’s memory fresh. Only after the player has demonstrated competence by completing all fifteen levels in each of the six introductory worlds are they allowed to progress to world seven and beyond, where the real challenge starts.
“Are you sure you want to do that?”
It’s amazing how effective a simple confirmation popup guarding the “delete all my saved levels” option can be. In fact, it’s our opinion that most of the really bad things that players can do in a sandbox game are identifiable in advance, to a certain extent. As the game designer, it’s your job to identify them and take defensive action.
Obvious things: “deletion” actions should be guarded with a confirmation, possible runtime exceptions should be caught, game-breaking exploits should be patched if found, and Hidden Rails should attempt to prevent players from frustration. Fuzzing can really help when testing your functions for strange behaviors.
Non-obvious things: players shouldn’t be able to share unwinnable levels, players shouldn’t be able to load unwinnable levels, players don’t want to overwrite their save data by accident, and players will probably try to make levels that look like penises.
Not all non-obvious things can be solved outright (too bad “Not Hotdog” from Silicon Valley doesn’t work for identifying penis-shaped Jetsam levels), but a lot of them can be made more difficult or made less of an issue through clever design. The Nintendo classic Mario Maker forces would-be level creators to beat their own levels before they share them, for example.
In Jetsam, every level can be expressed with an ASCII code – for example, take a peek at the following level (which just so happens to be the first one in the game).
The universality of Level Codes is extremely useful for sharing and saving levels – all people need to do is transmit text between each other. However, it introduces the problem of protocol robustness – whereby we need to make damn sure that our Level Codes are backwards-compatible, and that we have flexibility available for the future should we need it.
As such, a lot of thought has gone into our encoding scheme. We settled on ASCII characters because of fears that some UTF-8 characters might be reinterpreted by overzealous text manipulation and rendering programs. We had to purposely avoid the control-character subset of ASCII as well, in order to make sure that no meaning was lost by text message apps that strip such characters from inputs.
We also left ourselves some breathing room – certain characters are invalid in our encoding scheme for now, but we might make them valid in the future to represent things like additional level types.
The moment we let Jetsam out into the wild, its success and staying power will depend greatly on a growing network effect from a massive library of player-created levels. Our Level Code protocol has to stand the test of time – we can never deprecate Level Codes once they’ve been deemed valid, or else we will have wasted our players’ hard work (and built up a library of Level Codes that don’t work!). The Level Code rules will thus be at their narrowest at launch, and will progressively loosen if we continue adding features.
Sandboxes are hard to design, but we’re confident that we’ve anticipated a lot of the headaches players might run into, and that we have designed Jetsam to mitigate them where possible. Players will always surprise us though – so we’re going to be launching a public Beta test of the Level Editor soon (on iOS). We’ll be adding 15 of our Beta testers’ best level creations to the shipping game as an added incentive to participate, so if that piques your interest, stay tuned!
Every now and then, in a game’s development lifecycle, you get the chance to pause and reflect. With the recent conclusion of our Jetsam Beta Test, we’ve been given such an opportunity.
While we’ve used the feedback to recalibrate our approach to level design, it’s also made a great stopping point to reflect on the bane of all developers’ existence: tech debt.
Jetsam has been in off-and-on development since 2015, when I was in grad school and working on it as an independent study. My programming prowess has improved noticeably since then – so looking through the codebase shocks me at times.
This week, I identified a number of anti-patterns in the code, and I wanted to walk through each of them to hopefully help newer developers recognize the traps they can lead to and how to avoid them. But first, I’d like to answer a question about the whole process.
Why refactor your code in the first place?
Refactoring can be really controversial among developers, especially in the game development community. It can be said that shipping something, even something that isn’t perfect, is more important than worrying about code quality – and rightly so! If no one actually gets a chance to play your game, it doesn’t matter how pretty your code is.
You’re not going to win awards for writing beautiful code in your game. You’re not going to be showered with acclaim for making a key algorithm perform slightly more efficiently. In fact – it’s pretty likely none of your players are ever going to see your game’s source code. Ever.
Then why refactor at all?
I think refactoring is important as a matter of hygiene, not as a goal in and of itself. I posit that refactoring is worth it if and only if it will make feature development faster in the future, and you have more features to develop. Refactoring greases the wheels to make development go faster, and it can do so in all kinds of situations:
Working in a team?
Refactoring can make it easier for other developers to understand what everyone else has built, meaning that people can “learn” other areas of the codebase faster and thus get work done quicker.
Refactoring can help you quickly re-learn areas of the code that you’ve long forgotten about, and increase your ramp-up speed the next time you need to come back to that part of the codebase.
Do you want to add new features to your game, ever?
Refactoring can make connecting new components to old code much easier.
After our beta test of Jetsam, it was clear that a few new features would really improve the game’s design. Those new features will change the way Jetsam moves – and the movement code in the game was a tangled hornet’s nest of repetition, magic numbers, and closed design. So it became a prime target for refactoring.
Don’t Repeat Yourself
This one’s a classic – the DRY (Don’t Repeat Yourself) principle. In Jetsam’s movement code, I had four separate methods bound to swipe event handlers – movePlayerLeft, movePlayerRight, movePlayerUp, and movePlayerDown. Each method was itself a rough copy-paste job of the others, with only X and Y values changed to reflect different iterator patterns for each of the cardinal directions. Most of the code in these methods was identical!
The solution: creating a movePlayer method that has an enum of the four cardinal directions as a parameter. That way, other code can call movePlayer(MoveDirection.kUp) to create an Upward motion; or movePlayer(MoveDirection.kLeft) to create a Leftward motion. The X and Y iterators are set in a switch statement at the beginning of the method, so that the looping logic works identically for all four directions.
As a result, Jetsam’s movement code went down in lines of code by a factor of four.
Minimize Magic Numbers
Numbers are extremely important as configuration parameters – especially in games. That shrinking animation won’t look quite right at a scale of 0.25x or 0.5x, but 0.378x seems just right, you know?
All too often though, these “magic numbers” get into our code in raw number form – we create a label with font size 30, or instantiate a sprite with dimensions 64px by 32px directly.
When I went to tweak the default menu button size in the game, I found, to my horror, that every single menu button throughout the entire game had its size set directly. Rather than changing the value in 39 separate locations, I declared a static variable on a Utilities class called defaultMenuButtonSize, and changed all 39 separate instantiations of menu buttons to utilize Utilities.defaultMenuButtonSize for their size.
As a result, I was able to tweak and re-tweak the button size until I got it juuuust right. And I’ll be able to easily tweak it again in the future if I change something about the UI’s layout.
This one is a little more hand-wavey than the rest, but one of the joys of game development is that things always change. If you’re just starting to develop a game, it’s pretty likely that it will go through several revisions of the core concept – whether they be tweaks, full-on redos, etc.
With that in mind, it is extremely important to write your code in a way that makes individual components easy to tweak, revise, and redo – without breaking the rest of your game in the process.
When I initially built Jetsam’s level editor, one of the core ideas was that levels themselves could be completely represented with ASCII strings. For example, ))/”()%%”!&”&#&$&%&&$&%&& or )) “(%((“!!!”!(!)”!”E”)$B%B&B(!(E())!)”)()). This more or less required that each bit in each character be assigned a meaning – and that any changes to the meanings of each bit would require all of the previous levels to be re-encoded.
So, when building the encoding scheme, I intentionally left some “spare bits” in these ASCII strings that I defaulted to zeroes. I didn’t have any plans to use them at the time, but I figured that in the future I might come up with a reason to use them, and I didn’t want to have to re-encode all the levels if I ever wanted to change something. And lo and behold, thanks to a beta tester’s great feedback, an entirely new way to play the puzzles was invented – and I had the spare bits to encode its activation into the level strings, without needing to change any of the existing 50+ levels.
Refactoring was the right call for Jetsam at this point – we significantly reduced the complexity of the movement code, making way for some exciting new features. We made our UI much easier to tweak with some magic number removal, and we also benefited from earlier open-ended thinking in that we were able to quickly add hooks in for a new level style to be announced in the near future.
While we don’t have anything tangible to show for it (other than some gory git diffs), we’ll definitely have lots to show for it in the near future as we build exciting new things.
Do you have a refactoring story to share? Post it in the comments!
We came. We saw. We beta’d. Jetsam has been in the hands of real players for the first time!
The Jetsam Beta was a carefully-selected sampling of stuff that will be in the final game. We were intentionally cagey about further details – we wanted players to get into the game and explore, thereby testing how good our design was.
We collected feedback and all kinds of metrics from our Beta – including how people navigated the basic UI, what they enjoyed/hated, and how difficult they thought the levels were.
This post will go into detail about some of the lessons we learned – what worked, what didn’t, and how we plan to address the great feedback we received from finally putting our game out into the wild.
Jetsam Beta – By the Numbers
12 levels and 1 unlockable “secret” level
3 testable, level-altering mechanics
27 unique beta testers
146 unique play sessions
6.2/10 average fun rating for fuel cells
6.8/10 average fun rating for crates
7.5/10 average fun rating for wormholes
~1/2 of players say that wormholes were their favorite mechanic
~1/3 of players say that wormholes were their least favorite mechanic
We asked a lot of questions about our UI to our testers – here are some lessons learned.
Expectation: Main Menu UI is easy to navigate. Reality: True! All testers reported that navigating the menus was easy.
Expectation: Level Select UI clearly described how levels were unlocked and whether or not you had completed a given level. Reality: False! One third of testers reported some degree of confusion about which levels were completed.
What we’ll be changing: We’ll be making the level selection UI prettier and abandoning the “badge” approach of putting an icon on the level button if you’ve completed or perfected a level. There was clearly too much confusion.
Expectation: All three tested mechanics would be roughly equally liked. Reality: False! Testers seemed to really engage with the wormhole mechanic, with half citing it as their favorite, and a third citing it as their least favorite. Testers liked all of the mechanics, but cited fuel cells as their least favorite, generally because grabbing extra fuel always seemed like the best option and didn’t present much strategic challenge.
Expectation: The three tested mechanics would be relatively easy to understand and learn without a tutorial. Reality: True! Most players cited that it was easy or very easy to learn how the three mechanics worked.
What we’ll be changing: We’ll be adding more wormhole levels since players liked them. We will be more purposeful about our fuel cell level design though, so several of those levels are going to be redesigned to make their fuel component more enjoyable.
Expectation: The levels we selected were a good representation of the difficulty spectrum across the whole game. Reality: False! Several players cited “extreme difficulty jumps” across our levels.
Expectation: We selected an easy (A), medium (B), and hard (C) level from each mechanic’s pool of existing levels, so we believed that each letter’s corresponding levels should receive similar difficulty scores. Reality: False! The 1X series of levels were too easy. The 4X series of levels were way too hard. The 2X and 3X series of levels hit our difficulty targets right about on the money.
Expectation: Roughly half of players would unlock the secret bonus level. Reality: False! Only three of our twenty-seven testers unlocked it.
What we’ll be changing: Our levels were too damn hard! We need to refocus our level design on creating more “medium” difficulty levels.
That about wraps it up – we learned a ton from the Jetsam Beta, and look forward to improving things across the board as we approach our holiday season launch window! There are lots of new tickets on the backlog and new ideas swimming through our heads.
We view Jetsam, progression-wise, almost like two different games. Some players will enjoy one section more than the other, but we want to make sure our appeal is as broad as possible by giving both our best attention.
The “campaign” in Jetsam (consisting of the main puzzle missions) can rightly be seen as a game in and of itself. There are progression aspects, the player learns new mechanics, secrets await discovery, and more. Plus – all of that is totally free to play, no strings attached. No ads, no annoyances – just a great offline puzzle game.
We could’ve stopped there, but decided not to. A lot of games these days attempt to bring players to an “endgame” state – the point where most players perceive that they have completed the content available in the game, but discover new, more advanced forms of gameplay they can repeat. This is a valuable goal for us, because we want to give something really rewarding to our most dedicated players, while also giving our more casual players something that keeps them coming back.
To do endgame “right” requires that we design a whole new set of mechanics that veteran players can keep doing forever.
It’s hard to articulate how intimidating “forever” is in game design – it requires thinking about a game in sandbox terms instead of completion terms, which is a pretty radical development mindset shift.
So what do other games do?
Endgames in first-person shooters are usually centered around competitive PvP play: ranking systems, matchmaking, etc. Endgames in MMOs usually focus on collaborative PvE experiences: dungeons, raids, and clans come to mind.
But what the hell does an endgame for a puzzle game look like?
After looking at endgame experiences across a lot of titles and genres, we found that the successful ones usually emphasize some form of player-to-player socialization aspect. We thus concluded that a level editor with sharing tools would be a great place to start. We have so far structured our endgame around building and sharing levels, enabling that socialization in our specific genre’s context. The idea is that an otherwise solo experience becomes dramatically enriched by others once you reach “endgame.”
Making levels buildable was exciting – we designed a new interface for our game that lets players “stamp” the tiles they want to create onto the play space. Given that we’re making a mobile game, the touch interface is perfect for something like that. Obviously lots more polish is needed, but the basic idea was effective enough that we have been able to develop the main game’s levels using the editor!
But making levels shareable was trickier – essentially we had to compress level data down to the most succinct format possible with some bit-level hacking, which resulted in what we call “level codes” – shareable strings of ASCII text that fully encode levels created by players in the editor. They’re so compressed, in fact, that you could conceivably tweet them…
The way we see it, the main game trains players to understand the sandbox, and the endgame frees them to go wild with it.That’s the philosophy underpinning our endgame experience. We want to empower players to build their own worlds in our game, and we hope that gives players of all stripes reasons to stick around long after they start playing Jetsam.
We’re serving up a delicious taster of Jetsam soon, so if you have an iOS device, sign up for Jetsam’s Beta here.
It will be free to download, and will have a main “campaign” of no less than 50 levels, ranging in difficulty from easy learner levels to downright sadistic brain-benders.
It will also ship with the ability to purchase a full-fledged in-game level editor that allows the player to craft their own levels, using all of the game mechanics we designed and more. Level sharing mechanisms will allow people to show off their levels far and wide, and the best of the best will be collected into level packs that will give players plenty of awesome new content to keep coming back to after launch.
So – how on Earth do you beta test all that?
We concluded that we want to start small. Therefore, our beta test will start off with a small handful of levels – enough to give a taste of what our main game has to offer, but not so much that it spoils the main course when the time comes. We’ll talk plenty about the level editor in the coming months, but mostly through dev blogs and not hands-on testing.
We really want to nail down the level design and make sure we’re properly calibrating the difficulty as the game progresses. We also want to introduce players to a few of the mechanics that we’ve thus far kept under wraps, but do so in an open-ended way that leaves us with the ability to surprise players when they download the finished game.
So, we’ve settled upon a 12-level design and a really basic (not final!) UI to express it.
Each row represents a different mechanic – the top row consists of standard levels, the second row consists of levels where managing your fuel count is important, the third row consists of levels with breakable crates, and the fourth row consists of… a mysterious kind of level you’ll just have to find out about yourself! Difficulty escalates across the columns – so 1A is easier than 1B, which is easier than 1C. That same progression should be true for the other rows as well.
Because the final game will have a lot of secrets and unlockables, we figured we ought to reflect some of that in the beta. As such, the last row’s levels can only be unlocked by beating the levels in the first three rows.
We expect that a typical player will get somewhere between 30 minutes and 1 hour worth of gameplay from our beta test – enough for them to formulate opinions about our game and give us some useful feedback that we can incorporate into the game before we launch it later this year.
The game, which revolves around the titular robot collecting pieces of space junk in a zero-g environment, could ruin the player’s experience in a number of ways.
Destructible environments meant that players could trap themselves in situations they couldn’t escape from.
A malicious sequence of wormholes could get players caught in an infinite loop.
Given that we wanted the game to have a full-fledged level editor, this caused problems. We couldn’t just hand-wave away these scenarios and exclusively design levels where these situations weren’t possible – the community was eventually going to get ahold of the level editor, and they surely wouldn’t be so merciful with it.
So we came up with a game mechanic that solved all of these issues: fuel.
How does fuel work? Well, it’s pretty simple really. Every move costs the player one fuel point. Run out of fuel and you explode. Explode, and the level restarts.
Fuel is what we call a “Hidden Rail” mechanic. Even though it appears benign, its intent is to prevent the player from getting too frustrated with him or herself. Believe it or not, this happens a lot – especially in puzzle games. Our game is pretty openly inspired by the Pokémon Ice Cave puzzles of yore – so posts like this one on Reddit really resonated with us and underscored the need to keep players from getting frustrated with Jetsam.
Running out of fuel lets us loudly tell the player “TRY AGAIN,” which prevents them from reaching a moment of frustration on their own after many failed attempts at a situation that might actually be impossible. We don’t want to waste our players’ limited time by letting them struggle indefinitely against impossible odds. Fuel also constrains the total number of moves players can make on a given level, in effect making levels slightly easier and giving us another difficulty lever to play with. The same level can be radically more or less difficult if we tweak the starting amount of fuel.
Hidden Rails (like Jetsam’s fuel mechanic) can help you as the game’s designer shepherd your player into enjoying your game in the way you intended. By identifying undesirable behavior patterns, you can use the invisible hand of design to break players of those behaviors – whether it’s through the sweet embrace of explody death as in Jetsam, or otherwise.
Constraints, counterintuitively, can often bring more creativity to bear. Adding fuel in as a Hidden Rail actually gave us more ideas, which led to further developments in the game, including collectible fuel packs, fuel maximization challenges, and a whole host of other things beyond the scope of this dev blog post.
We can’t wait to tell you about those soon!
If you liked this blog post, follow us on Twitter and check back regularly here for updates. If you have an iOS device, sign up for our iOS TestFlight Beta here. We would love for you to be part of our development journey.