Today I started the guide on developing EVE Online bots, describing how to set up the programming tools to efficiently work on EVE Online bots. You can find it here:
With the new software, a generic interface between bot and host is introduced, which offers more flexibility in the functionality integrated into a bot.
One concrete example of how this flexible interface improves over the old Sanderling app: This allows coding custom logic to choose a windows process in case there are multiple instances of the game client process present.
Most developers won’t need to use this new interface directly but will use it indirectly over libraries encapsulating these low-level functionalities for easier use.
An example of such a library is the one for EVE Online, contained in the warp-to-0 auto pilot bot, in the file Sanderling.elm:
Recently there was a change in EVE Online which affected several users: After an update of the EVE Online client, some bots had issues undocking a ship from a station. @Aseratis and @Kaboonus fixed this problem as you can see in these posts:
I released version 2019-05-29 of the bot framework and the botengine console. The new version brings several improvements:
To improve the readability of the bot code, cleaned up the interface between bot and host (engine).
Added the bot side interface to set a configuration for a bot.
Better support for identifying bots: The bot ID is now displayed in the user interface. Also, a change in the packaging of a bot from local files fixes nondeterministic bot IDs. (The problem where the bot ID would change without changes in the files. (This problem was caused by timestamps being added at the time of loading the bot)).
Recently some people reported problems with downloading bots to load them into the engine. Today’s release of the new BotEngine Console adds a new feature to address this problem. Instead of downloading the bot from Github manually, you can now let the BotEngine Console do this for you. To use this new feature, specify the Github URL with the --bot-source parameter. The engine will then take care of the download:
This bot source looks like a URL. I try to load the bot from Github
I found 8 files in 'https://github.com/Viir/bots/tree/32559530694cc0523f77b7ea27c530ecaecd7d2f/implement/bot/eve-online/eve-online-warp-to-0-autopilot'.
I loaded bot 266EDDE2CCA2F71BC94DFD941F469E5F3DA20DACFD08A08E680B0C09646DF6C1.
Starting the bot....
I also updated the guide to explain the different kinds of bot sources and include examples:
Recently people reported issues slowing them down when using bots. This new release of the botengine console adds a feature to fix these bot configuration issues. The new feature allows you to:
Configure a bot using the command-line interface, by adding the new --bot-configuration parameter.
Configure a bot without changing the bot code, which in turn means without changing the bot ID. This means we can compare bots easier, by using the now more stable bot ID.
I updated the guide to illustrate how this new feature is used, including an example command line:
After the improvements on bot discovery and operation as reported in June, work continued on the bot development side in the last two weeks.
@TOBIAS95 had some questions, and while answering these I used the opportunity to write in more detail how the process of bot development works.
Some of the topics covered:
How do we describe what a bot should do, in a way that avoids misunderstandings between people?
How can I take and save the screenshots used to describe a bot?
How can I share the files I collected with other people?
I posted this guide here:
This week, I explored the bot implementation process further, based on the example screenshot given last week. This exploration answered several questions about developing bots:
How can we easily model example files to use in automated tests, for example when testing image file decoding implementations? The elm app implemented here displays a Base64 encoding of the file chosen by the user. The elm module containing the automated tests demonstrates how such a Base64 representation of a file is converted back to the original Bytes sequence representing the file.
How do we decode an image which was stored into a BMP file using Paint.NET? The automated tests added here model files and expected elm values resulting after decoding these image files.
How do we locate user interface elements in screenshots? The pattern model implemented here supports flexible configuration of a search which works with screenshots from video games like EVE Online.
How can we make it easy for developers to find the right search configuration for their use-cases? The graphical user interface implemented here supports quick configuration and testing of image search patterns.
I made progress with the guides for bot developers:
Last week, I started another exploration to learn how to further improve the development process for the image processing parts in our bots.
One result of this exploration is a guide which explains how to test candidate functions with example screenshots quickly. Using this approach, you can test the image processing parts of your bots without the need to start a game client. Instead, you point the test framework to an example screenshot and review the results.
I illustrated the approach to image processing and finding objects in screenshots with an example implementation and image.
A byproduct of this exploration is a demonstration of file loading (and parsing) in a bot. So if you have a use-case where you need this, the demo bot implemented here might be a good starting point.
I refined the interface between bot and host to integrate what I learned in the last seven weeks.
The bot interface version 2019-07-20 is introduced in the implementation with commit e998769bda1f47b65e2cbd5cee2cb0165222f9a5.
This refinement is also informed by the following two observations:
While working on the Sanderling framework for EVE Online bots, I saw this kind of interface already evolving (See the statusMessage record field in SimpleSanderling.elm)
More recently, I got confused about a bot status message displayed in the engine (Also see Avoid confusing the last status message from the bot · Issue #1 · Viir/bots · GitHub for the original report). In that case, the bot did not set the status message in each step, so the engine continued to display the message from an earlier bot step. I first thought this message came out of the processing of the last event (for which the time is displayed). But with the previous implementation, the engine kept the previous status message if the bot did not respond with a SetStatusMessage request.
The change in this commit adapts the bot interface to integrate these learnings. The new interface requires the bot developer to explicitly state the status message, so this can not be forgotten. At the same time, this interface avoids the confusion about cases where a bot would respond with multiple SetStatusMessage requests in the same step.
This commit also updates all example bots to the new interface.
Since we already kept the previous bot interface for seven weeks, I expect version 2019-07-20 to be sufficient for months to come.
Looking back on my guess about the longevity of interface version 2019-07-20, I found it almost comic today. Somehow I found it annoying to spend again so much time on refining this aspect of the platform. On the other hand, I feel relieved that we now have an even more solid foundation to build on. I hope perfectionism is satisfied enough so that I can build templates and example bots on this.
@csharper had some questions about bot development, so I expanded the development guide and example bots / templates to answer these:
These questions were driving a lot of progress and the whole process took more than two weeks. Following are some of the results of answering @csharpers questions:
Improvement of the file structure in the example bots and templates. Bot.elm as a common entry point.
Extension of the development guide to illustrate the overall bot architecture.
Simplified repository structure to help the code examples.
Today I explored memory reading EVE Online. This is for advanced users who want to learn how the memory reading for EVE Online works and maybe customize it. To make this easier, I took the reference implementation and improved it.
I used the implementation from there as a basis so the Python side of things might already look familiar to you.
In contrast to the older code, today’s version also supports reading from a file containing a sample of the client process. This new feature allows us to repeat the reading as often we want, without having to start an instance of the game client. In addition to that, another new feature lets you save the result of the memory reading to a JSON file for easy inspection and further processing. This JSON file contains the UI tree read from the EVE Online client. To use this feature, add the --output parameter when running the program.
Below is an example of a full command-line; in this case, it reads the memory contents from a file saved earlier:
dotnet run -- --source="C:\path-to-a-process-sample-file.zip" --output="C:\path\to\directory\with\write\access"
So the result of today’s exploration is a .NET console app which supports both from a live client process and a saved process. All the memory reading functions are inlined in the source code so that you don’t have to look up DLL files or other assemblies to see the implementation.
This week I refined the process to start a bot to reduce the effort required there. Before this week, the guide called for the installation of .NET frameworks as a prerequisite to running a bot. This weeks change integrates this third party software into the BotEngine application so that we do not anymore need the previous installation steps.
Besides, the setup of the BotEngine app was simplified. It is now bundled in a single file so that the installation now consists only of copying a single file. This bundling was made possible by the new Single-file executable feature recently released by Microsoft with .NET Core 3.0.
Reading the post from Microsoft, it seems simple: Just add this parameter when publishing the app and you get everything bundled into a single file. But … it did not work, and I filed this issue over at Microsoft to learn why: https://github.com/dotnet/cli/issues/12723
I am glad that the experts were so quick to help and we soon found a solution which works for us. The new version of the app was tested and worked on both Windows 10 and Windows 7, so I am happy with the results of this effort.
Building on the recent improvements of the BotEngine app, I simplified the guide on using the EVE Online warp-to-0 autopilot.
The number of steps to set up and run the bot is much smaller now:
The new app bundling achieved in September eliminated the need to install other software.
With the introduction of the new interface to run bots, the manual navigation steps in the Apps UI are not necessary anymore.
Another aspect contributing to easier use of the new guide is that it explains in more detail. In case of problems, the more specific error messages from the bot (for example, when no EVE Online client was started) help to figure out the cause. Compared to the older Sanderling App UI, the new, more sequential interface to run bots makes it more evident which information is relevant at the current stage.
The last three weeks have been busy. @MutantWizard joined us and explored the area of mining in EVE Online. He shared a lot of details about the game, and that made it possible to develop an EVE Online mining bot.
Starting with the simplest possible version, we refined this bot over several iterations to support a broader range of setups and make it easier to use. Since this bot has now matured into a robust solution, I integrated it into the general guide on EVE Online bots.
This bot is the most advanced bot based on the new development framework introduced this year. As such, this bot helps not only EVE Online users but also bot developers as an example of how to benefit from the most advanced development tools.
An update on the example EVE Online mining bot mentioned earlier:
Recently @Merizon discovered an issue with that bot. Jan found a setup not yet sufficiently represented in the training data used to develop the bot:
Using the screenshot of that setup, I expanded the training data set to cover it too. The guides now point to this new version of the mining bot.
In the last two weeks, I made several improvements to the botengine Windows app and the tools to run bots.
Online Bot Sessions
When you start running a bot, you now have the option to start an online bot session to help you keep track of your bots. Online bot sessions also allow you to check the status of your bots from other devices such as a smartphone. You can see an overview of your online bot sessions by logging in to your user account at https://app.botengine.org/
Bot Catalog Improvements
The bot catalog now got its own dedicated website. This week I improved the UI there to make it easier to read bot details and distinguish the entries. Head over to https://botcatalog.org/ to see the new, improved list of bots.
Easier Configuration
Version 2019-12-11 of the Windows console app comes with a new function to help you set up your environment: You can now choose to register the executable file so that you can use botengine in the Command Prompt instead of typing the complete path to the executable file. Use the sub-command install-botengine-command to perform this automatic configuration. (It will not affect the same Command Prompt window, start a new Command Prompt to try it)