What do I think about Lua after shipping a project with 60,000 lines of code?
Hi there! This is Oleg from Luden.io. We decided to have a deep and meaningful conversation about Lua programming language with Ivan Trusov, lead programmer of the video game Craftomation 101. It contains ~60,000 lines of Lua code and is made with Defold game engine.
I asked Ivan to talk about the real issues and show real code, not the “hypothetical code, carefully prepared for the public to illustrate not how we do it but how we think it's supposed to be done.”
Craftomation 101 is a game about self-replicating robots, instructed with visual programming to craft (and eat) resources to terraform a frozen planet. The game was recently released to Early Access for Windows, macOS, and Linux on Steam, itch.io, and GOG. Also, there is a free demo playable in the web browser.
The game was well received by players and currently holds “Very Positive” user rating on Steam. Additionally, 2 months after the launch, a few educational organizations are using it for computer science classes and coding camps (which is so exciting and inspiring for us!)
Oleg Chumakov: Ivan, first of all, why Lua? Why have we not considered using TypeScript, Haxe, plain C++, or something else at the beginning of development?
Defold engine provides C++ and Lua support but there is an open-source TypeScript + Defold toolkit https://ts-defold.dev/ and Haxe support library for Defold https://github.com/hxdefold/hxdefold
The Defold engine was chosen for the project because it provides super lightweight web builds that are Chromebook-compatible and native builds for PCs, consoles, and mobile platforms. It’s important because our games are used in educational organizations full of Chromebooks.
Ivan Trusov: I asked Dmitry…
Dmitry — a colleague of ours, and a veteran of Defold engine — told me that everything is cool with Lua. I asked him if Lua is the language we use to write config files for Redis. He assured me that everything is going to be great because he had already shipped a bunch of projects with it.
So I trusted Dmitry…
My previous experience was with C++ (I thought I knew C++, but working with ClickHouse showed me otherwise), Python (though I was aware that Python wasn’t a big player in game development), and Java. So my stack was outside of major game development engines. Additionally, Defold provided native support for Lua and C++, making it an easy decision.
I heard that Lua is also known among game developers as an easily integrable, tiny language for any homebrew engine, especially when you want to provide more flexibility for your game designers in implementing complex in-game systems like abilities or spell effects.
On the flip side, you might occasionally find performance spikes in the profiler when game designers use some “dark magic” inside their Lua scripts.
But this integration as an additional language for game designers wasn’t our case; we intended to code all the game logic in Lua.
O.C.: So you started developing in Lua. Do you remember if anything surprised you about Lua in the beginning?
I.T.: Sure, the lack of an increment operation, no “continue” instruction, and array indices starting from 1 instead of 0. These differences can be jarring, especially when switching between two languages for different projects/scripts. I still sometimes write [array.length-1] to get the index of the last element; you just can’t get used to that after years of working with C++ and Python.
Also, back in my university years, I took two courses on Haskell. Surprisingly, I see some benefits in applying approaches from functional languages to Lua. For example, creating an uncurried function with 4 steps, then folding and decorating it.
What I’m trying to say is that Lua was probably inspired by some functional elements along with simple scripting languages. This is why some functional patterns can bring new ideas to a Lua developer’s life.
Those functional vibes were quite surprising to me. I can illustrate it with something like this:
Here is also a bit more complex illustration of “Haskell vibes” in Lua with currying a function and an alternative solution for comparison.
O.C.: And what about the lack of classes in Lua?
I.T.: Actually, I always loved simple solutions and was never a fan of OOP. To create a good class, one needs to understand a lot about its future usage and OOP principles in general. I still cringe at memories of past programming job interviews that included questions like, “Tell me about the three principles of OOP”. Maybe some companies still ask about that during interviews. We don’t judge here.
The organization of code in Lua was quite intuitive to me. It was even clearer because of its' simple “everything is a table” approach. But this fundamental principle wasn’t clearly communicated, I just came to it when I got a “a table in the table is not a table but a reference to the table” bug and was trying to fix it. I just set some value using “=” and after that, my table was referenced and not copied, so two parts of the game started editing the same table, and I got some “interesting and unexpected” outcomes.
Let’s illustrate it with some code. I had the function shown below, and I wanted to create a table for a queue in which I would write and then sort robots (I don’t remember what the queue was for). So, I created it with a size of 100*1, giving it a table like {x=0, y=0} as default value.
However, as you can see below, the default value is assigned via “=”, without copying it to each element. As a result, for a very long time, I couldn’t understand what was happening and why my robot sorting always produced some kind of nonsense.
It turned out that the table contained 100 references to the same coordinates. Each time I rewrote x and y in an element of this table, I was actually rewriting them in one table, and the reference to this table was assigned to all table values. So, I ended up sorting 100 identical elements.
In the end, I fixed it like this, but I learned an important lesson about Lua in the process.
O.C.: Okay, and what do you think about…
I.T.: One more thing about the tables, please!
At the end of the day, I think writing game logic (or business logic) is much more convenient with all these tables in Lua than with C++. When compared with Python on a basic level (I mean the pure language, not all the amazing packages like PyTorch, NumPy, and thousands of others), I suppose it’s fair to regard Lua as a simplified version of Python.
Almost anything that is easy to implement in Python is also easy to implement in Lua. However, when it comes to separating code into modules or packages, Python is more convenient.
O.C.: Alright, I got it about tables, I want to ask about…
I.T.: The very last thing about the tables, please!
There is another side to “everything is a table” — you don’t need to make hard decisions when choosing the structure since no other options are provided!
Now I’m done with talking tables, thank you.
O.C.: Have you consulted about this “tables” approach with other Lua developers?
I.T.: After that, I went back to Dmitry and asked him if my understanding of “everything is a table” was correct and, if so, why Lua was designed this way. Dmitry told me that Lua was created at the Pontifical Catholic University of Rio de Janeiro and that it was acceptable for Pontifical Catholic Universities to design programming languages this way.
O.C: What do you think about the performance?
I.T.: Just by perception, developing in Lua makes you feel like it’s a “feels like fast” language. In Craftomation 101, there are very few places where we actually face performance issues due to the speed of the language, so our case may not be entirely relevant. That’s why I refer to it as “feels like fast” rather than just “fast”. There's a bunch of benchmarks that shows how fast it is in numbers.
There are options to write C++ code in Defold, such as creating a module or modifying the engine itself (code available on GitHub). For me, it was a potential workaround for future moments when I might need to tackle computationally complex tasks.
I always start by implementing a module in Lua, expecting to rewrite it in C++ if the performance profiler shows significant CPU problems. However, the problems were not so huge, and applying a bit of algorithm knowledge to Lua code often resolved or minimized them.
The game is relatively simple from a technical standpoint compared to projects like Warnament (another Defold project of ours, that heavily relies on C++ modules). Additionally, there are natively implemented C/C++ modules available in the Defold ecosystem, such as A* pathfinding.
O.C.: With your experience in C++ and Java, did you ever feel like “I don’t know about any issues until they happen in runtime”, because there is no compilation step and you don’t have the confidence provided by strictly typed compiled languages?
I.T.: Yes, sure, but it wasn’t a big deal since I've been working with Python a lot at that time. It wasn’t just business logic written in Python. It was about three years of machine learning with Python. Issues hidden in the huge bulk of data (and all the code written to handle the “dirty” data) are even less controllable than issues hidden in interpreted code.
O.C.: Have you used a Linter or other code analysis tools?
I.T.: Actually, I am a big fan of using the default “stock” setup until I find something that really excites me. For the first six months, I even wrote code using Defold’s embedded code editor because Dmitry told me it was a good way to work with code (it even has syntax highlighting!).
After that, I switched to VS Code (the author of this article is a big fan of Sublime and can’t resist adding this sentence saying that Sublime is the best!). However, I have to mention that I wasn’t really experienced with using Linter, which is why simply googling “VS Code Lua Linter” and installing whatever I found by following the top link didn’t make a huge difference in my programming experience.
But the basic VS Code Lua package provides a type checker that helpfully highlights some issues in the code, which is quite useful.
O.C.: I remember a standup meeting when you told me that you hated Lua and suggested switching to TypeScript. At that moment, there were about 60,000 lines of code in the project. Isn’t it a good idea to add some static analysis tools when you end up with this much code?
I.T.: Static analysis tools or TypeScript were always on my mind, but in the background, as a “to-do for the future.” During periods of development when we lacked QA and I wasn’t sure if the game was stable, it was more on top of my mind. After a big round of QA which happened closer to one of the milestones, I fixed a number of issues of the “it’s a vector but for some reason, it was assigned a string” kind. During the QA process, the game turned out to be in a much more stable shape than I had expected.
O.C.: For the next project, which could be just as large or even bigger, do you want to start with TypeScript?
I.T.: Actually, as of now, I don’t think so. TypeScript support level in Defold might be one of the bigger issues for me — it’s unofficial, and even with the best community support, you can’t be sure that all the engine updates will be implemented, or you’ll have the time to implement them yourself.
Two main points I would like to have in any “alternative to Lua” are well-known:
- I want to be aware of my code’s issues during compilation, not during execution.
- I want to organize my codebase in a convenient and simple way. In dynamically typed languages like Lua, there is so much flexibility in organizing your codebase and designing architecture that it can sometimes be less convenient than systems that enforce a particular style.
O.C.: There are some rumors that Luau is under consideration for integration into Defold. Code that is valid in Lua 5.1 is also valid in Luau. What do you think about that?
Luau is a gradually typed language derived from Lua, made by Roblox.
I.T.: Actually, I’ve never tried Luau, and as far as I know, there is an issue with updating the Lua version to 5.2 in Defold. Lua 5.2 includes an increment operation, but what I really want is Lua 5.4 (dear Defold team, please!).
But we are talking about Luau here. I think Luau can make life easier but not dramatically so, because it is something in between Lua and strictly typed languages. A similar situation happened with TypeScript, which was born between JavaScript and strictly typed languages.
The perfect scenario is to have TypeScript officially implemented in Defold or, if we can dream big, another language designed from day one as a strictly typed.
O.C.: Which one?
I.T.: Once, my university friends asked me why it’s not possible to write code in Go for Defold. I asked, why should we ever consider this? They answered that this is because Go is such a cool language, why not! That was, of course, a hypothetical crazy question related to some hypothetical future, where we have language-agnostic systems like WebAssembly, which is already integrated into Defold.
Thanks to Unity, C# is now one of the leaders in our industry. However, I am not sure if it’s possible to integrate it into Defold without sacrificing the lightweight feel of the engine.
O.C.: What things in Craftomation 101 could be made better with a strictly typed language?
I.T.: For example, we have a huge map with many interactive objects. The player can interact with them, and our robots, the CraftoMates, can also interact with objects. Additionally, they can search for particular objects to mine resources from them.
All objects on the map are parameterized objects stored in an array, similar to entities in ECS. There are many types of objects, like fire, wood, fire on wood, fireplace, etc. To orchestrate all of this in a strictly typed language, I could intuitively use OOP or ECS and handle particular types with specific managers, controllers, or systems. Alternatively, I could create a grand manager-handler with a series of IF statements.
In Lua, all of this is also possible, but since it’s not as straightforward, a developer might start missing the ease of implementation found in languages like C#.
Here is the code for the “if selector,” which is the absolute champion of causing bugs in the project.
There are metatables in Lua which, in some ways, can serve as an alternative to classes. However, from my perspective, it’s better to either start with metatables from the beginning or ignore them, rather than trying to introduce a metatable approach to an existing codebase.
There is a huge UI library for Defold called Druid, which is implemented with metatables and has some OOP vibes. It seems that using OOP in Lua can quickly and surprisingly lead developers into a situation where adding new features becomes really hard.
O.C.: Is there something special on the CI side of using Defold and Lua?
I.T: I was so surprised by the speed of building my project with Defold. I remember hearing stories from friends who are making games with Unreal Engine about the long build process. We also have other projects in our studio made with Unity, which is quite good, but the build time in Defold feels like pure magic.
Even now, with the project being huge, the average build time is 4 minutes. We use GitHub actions and a local runner to build the game. Using a Mac Mini with M1 Apple silicon as our local runner we can build all the platforms using only one machine.
O.C.: Any thoughts on the next project?
I.T.: First of all, the full release of Craftomation 101 is still ahead; the roadmap is still oh so big. For the next project on Lua and Defold, I would use a much more module-based code organization. Because of my current code organization, I was using a lot of “messages” (a way of communication between game objects provided by Defold). I think I shouldn’t rely on them so much in the future.
Another thing I want to do is create more atomic game object managers because they have grown so huge, and make them more independent.
Unit tests and other automated testing sound great, but as is often the case with making video games, their usage could be very limited. However, I really like the idea of automated tests to check user interaction scenarios, and I would definitely give that a try.
O.C.: Do you have some materials to recommend?
I.T.:
- There are some official books about Lua on the official site.
- Also, I think it’s beneficial to read something about Haskell, Clojure, or even OCaml.
- And of course, while not specifically about Lua, the classic “Game Programming Patterns” is a gold mine.
O.C.: Tabs or spaces?
I.T.: Tabs!
O.C. What keyboard do you prefer for work?
I.T.: I use the Dark Project One keyboard. I also want to mention my mouse because it has been working perfectly since high school: the Bloody TL70. Recently, I bought the Logi MX Master 3S.
And also at my work desk, I do have a very helpful cube.
Do you have a secret?
Ivan revealed a secret, but it’s a bit shocking. Please think before scrolling down to discover what it is. For your safety, I am putting a long image of the game’s roadmap below, so you can’t accidentally scroll to the secret revelation. You have been warned!
The secret: I prefer this color scheme for VS Code and switched to the dark one just to make screenshots for this post.
P.S. I can’t end this article without expressing our love and a huge thanks to all our players — you are such amazing folks, and we are so inspired to keep going with our game projects thanks to all of you: coders, educators, parents, students, engineers, and all the curious minds from everywhere❤️
Special bonus part: we asked Defold's almighty Björn Ritzl to comment on this article, and he was very kind to agree:
“I suppose it might make sense to comment about other languages:
- We are no longer considering to use Luau (Roblox).
- We have instead decided to add generic transpiler support to our tools (editor and command line)
- This will allow developers to create their own extensions to transpile from other languages to Lua and run using the standard Lua VMs included with Defold.
- To show how this is done we have decided to provide official support for Teal, a typed Lua dialect which transpiles to standard Lua code. This will give you build time type errors when you use Teal files (files with the .tl extension). It is still early days and we have no syntax highlighting and no LSP support for Teal yet, but it is already useful.
Another thing worth pointing out is that we now also have built in Lua LSP support in the editor, using a well know Lua LSP together with LuaCheck. This will give you static code analysis of your code with hints while writing your code and build errors if the Lua syntax is wrong.
Finally, we are also rewriting the public portion of the C++ SDK that is currently used to write native extensions. We are now working on a C SDK and expanding the set of available SDK functions such that it will be possible to write entire games in native code.
This updated C SDK will be used to generate the C++ SDK used for native extensions today and it will also be used to generate C# bindings for those that wish to write game or extension code using C#. The use of C# will be completely optional and not in any way bloat the engine or degrade performance for those who chose to not use it.
C++ and Lua will still remain the main languages in Defold.
Transpiled languages such as Teal and compiled ones such as C# will be optional.”
