BotLab Devlog

After the improvements on bot discovery and operation as reported in June, work continued on the bot development side in the last two weeks.

@TOBIAS95 had some questions, and while answering these I used the opportunity to write in more detail how the process of bot development works.
Some of the topics covered:

  • How do we describe what a bot should do, in a way that avoids misunderstandings between people?
  • How can I take and save the screenshots used to describe a bot?
  • How can I share the files I collected with other people?

I posted this guide here:

This week, I explored the bot implementation process further, based on the example screenshot given last week. This exploration answered several questions about developing bots:

  • How can we easily model example files to use in automated tests, for example when testing image file decoding implementations? The elm app implemented here displays a Base64 encoding of the file chosen by the user. The elm module containing the automated tests demonstrates how such a Base64 representation of a file is converted back to the original Bytes sequence representing the file.
  • How do we decode an image which was stored into a BMP file using Paint.NET? The automated tests added here model files and expected elm values resulting after decoding these image files.
  • How do we locate user interface elements in screenshots? The pattern model implemented here supports flexible configuration of a search which works with screenshots from video games like EVE Online.
  • How can we make it easy for developers to find the right search configuration for their use-cases? The graphical user interface implemented here supports quick configuration and testing of image search patterns.

I posted the complete exploration here:

I made progress with the guides for bot developers:

Last week, I started another exploration to learn how to further improve the development process for the image processing parts in our bots.
One result of this exploration is a guide which explains how to test candidate functions with example screenshots quickly. Using this approach, you can test the image processing parts of your bots without the need to start a game client. Instead, you point the test framework to an example screenshot and review the results.

I illustrated the approach to image processing and finding objects in screenshots with an example implementation and image.

A byproduct of this exploration is a demonstration of file loading (and parsing) in a bot. So if you have a use-case where you need this, the demo bot implemented here might be a good starting point.

I posted the completed exploration to the repository on Github here:
https://github.com/Viir/bots/commit/c36d58b7a68fa07ba5ca15668ef68f550778938f

For more details, see the overview document at bots/2019-07-10.locate-objects-in-screenshot.md at c36d58b7a68fa07ba5ca15668ef68f550778938f · Viir/bots · GitHub

I refined the interface between bot and host to integrate what I learned in the last seven weeks.
The bot interface version 2019-07-20 is introduced in the implementation with commit e998769bda1f47b65e2cbd5cee2cb0165222f9a5.
This refinement is also informed by the following two observations:

  • While working on the Sanderling framework for EVE Online bots, I saw this kind of interface already evolving (See the statusMessage record field in SimpleSanderling.elm)
  • More recently, I got confused about a bot status message displayed in the engine (Also see Avoid confusing the last status message from the bot · Issue #1 · Viir/bots · GitHub for the original report). In that case, the bot did not set the status message in each step, so the engine continued to display the message from an earlier bot step. I first thought this message came out of the processing of the last event (for which the time is displayed). But with the previous implementation, the engine kept the previous status message if the bot did not respond with a SetStatusMessage request.

The change in this commit adapts the bot interface to integrate these learnings. The new interface requires the bot developer to explicitly state the status message, so this can not be forgotten. At the same time, this interface avoids the confusion about cases where a bot would respond with multiple SetStatusMessage requests in the same step.

This commit also updates all example bots to the new interface.

Since we already kept the previous bot interface for seven weeks, I expect version 2019-07-20 to be sufficient for months to come.

To avoid confusion about the bot interface, I updated the guides in the bots repository to link to binaries built with the current interface:
https://github.com/Viir/bots/commit/7d1750e7e01fb006ac7aefa88c05da7a395615db

Looking at the interface between bot and host again, some questions came up. I found some things confusing. To avoid others being confused too, I refined the interface once again:
https://github.com/Viir/bots/commit/faf59c8d5f24a7648c2d009949c2b440c0c06eab

I already published the new build of the console app to run bots and updated the guides:
https://github.com/Viir/bots/commit/d423d2da7846465da476117d612058c4a9a7dae6
Version 2019-08-03 of the console app and the bot interface is up!

Looking back on my guess about the longevity of interface version 2019-07-20, I found it almost comic today. Somehow I found it annoying to spend again so much time on refining this aspect of the platform. On the other hand, I feel relieved that we now have an even more solid foundation to build on. I hope perfectionism is satisfied enough so that I can build templates and example bots on this. :pray:

@csharper had some questions about bot development, so I expanded the development guide and example bots / templates to answer these:

These questions were driving a lot of progress and the whole process took more than two weeks. Following are some of the results of answering @csharpers questions:

Today I explored memory reading EVE Online. This is for advanced users who want to learn how the memory reading for EVE Online works and maybe customize it. To make this easier, I took the reference implementation and improved it.

The original tutorial was published in 2016, and the memory reading was discussed in this thread: Advanced Do It Yourself Memory Reading In EVE Online

I used the implementation from there as a basis so the Python side of things might already look familiar to you.

In contrast to the older code, today’s version also supports reading from a file containing a sample of the client process. This new feature allows us to repeat the reading as often we want, without having to start an instance of the game client. In addition to that, another new feature lets you save the result of the memory reading to a JSON file for easy inspection and further processing. This JSON file contains the UI tree read from the EVE Online client. To use this feature, add the --output parameter when running the program.

Below is an example of a full command-line; in this case, it reads the memory contents from a file saved earlier:

dotnet run -- --source="C:\path-to-a-process-sample-file.zip" --output="C:\path\to\directory\with\write\access"

So the result of today’s exploration is a .NET console app which supports both from a live client process and a saved process. All the memory reading functions are inlined in the source code so that you don’t have to look up DLL files or other assemblies to see the implementation.

I added this program to the repository here: Support memory reading development for EVE Online · Viir/bots@92410ac · GitHub

This week I refined the process to start a bot to reduce the effort required there. Before this week, the guide called for the installation of .NET frameworks as a prerequisite to running a bot. This weeks change integrates this third party software into the BotEngine application so that we do not anymore need the previous installation steps.

As a result, the process to start a bot (for the first time on a machine) is now less than half as complex as before: Make it easier to use bots · Viir/bots@c166c13 · GitHub

Besides, the setup of the BotEngine app was simplified. It is now bundled in a single file so that the installation now consists only of copying a single file. This bundling was made possible by the new Single-file executable feature recently released by Microsoft with .NET Core 3.0.

Reading the post from Microsoft, it seems simple: Just add this parameter when publishing the app and you get everything bundled into a single file. But … it did not work, and I filed this issue over at Microsoft to learn why:
https://github.com/dotnet/cli/issues/12723

We found a way to work around this limitation and got the single-file bundle to work. Discussions are ongoing on how to improve the .NET Core feature in the future:
https://github.com/dotnet/sdk/issues/3685

I am glad that the experts were so quick to help and we soon found a solution which works for us. The new version of the app was tested and worked on both Windows 10 and Windows 7, so I am happy with the results of this effort.

Building on the recent improvements of the BotEngine app, I simplified the guide on using the EVE Online warp-to-0 autopilot.

The number of steps to set up and run the bot is much smaller now:

  • The new app bundling achieved in September eliminated the need to install other software.
  • With the introduction of the new interface to run bots, the manual navigation steps in the Apps UI are not necessary anymore.

Another aspect contributing to easier use of the new guide is that it explains in more detail. In case of problems, the more specific error messages from the bot (for example, when no EVE Online client was started) help to figure out the cause. Compared to the older Sanderling App UI, the new, more sequential interface to run bots makes it more evident which information is relevant at the current stage.

I uploaded the new guide at bots/how-to-automate-traveling-in-eve-online-using-a-warp-to-0-autopilot.md at 62863c120ee5d063cfc3490bbf6a73e432c70945 · Viir/bots · GitHub

The last three weeks have been busy.
@MutantWizard joined us and explored the area of mining in EVE Online. He shared a lot of details about the game, and that made it possible to develop an EVE Online mining bot.

Starting with the simplest possible version, we refined this bot over several iterations to support a broader range of setups and make it easier to use. Since this bot has now matured into a robust solution, I integrated it into the general guide on EVE Online bots.

This bot is the most advanced bot based on the new development framework introduced this year. As such, this bot helps not only EVE Online users but also bot developers as an example of how to benefit from the most advanced development tools.

An update on the example EVE Online mining bot mentioned earlier:
Recently @Merizon discovered an issue with that bot. Jan found a setup not yet sufficiently represented in the training data used to develop the bot:

Using the screenshot of that setup, I expanded the training data set to cover it too. The guides now point to this new version of the mining bot.

In the last two weeks, I made several improvements to the botengine Windows app and the tools to run bots.

Online Bot Sessions

When you start running a bot, you now have the option to start an online bot session to help you keep track of your bots. Online bot sessions also allow you to check the status of your bots from other devices such as a smartphone. You can see an overview of your online bot sessions by logging in to your user account at https://app.botengine.org/

monitor your bots using online bot sessions|

Bot Catalog Improvements

The bot catalog now got its own dedicated website. This week I improved the UI there to make it easier to read bot details and distinguish the entries. Head over to https://botcatalog.org/ to see the new, improved list of bots.

Easier Configuration

Version 2019-12-11 of the Windows console app comes with a new function to help you set up your environment: You can now choose to register the executable file so that you can use botengine in the Command Prompt instead of typing the complete path to the executable file. Use the sub-command install-botengine-command to perform this automatic configuration. (It will not affect the same Command Prompt window, start a new Command Prompt to try it)

Introducing the Alternate UI for EVE Online

In the last two weeks, I worked with Zachary Kline (@BLindGuyNW) to help blind people play the video game EVE Online and make the game client more accessible.

One of the results of this collaboration is an alternate UI for EVE Online. The alternate UI renders the contents of the regular game client into an HTML document. The HTML based representation enables better access to screen readers. The information there is on the same level of detail as available to bots, supporting in-game activities as diverse as ratting or mission running. Usually, this kind of structured information is not readily available from the game client in such detail; there is no supported API for this level of detail from the game developer CCP. By using memory-reading on the game client process, we get this information without depending on image processing.

You can read more about the evolution of the alternate EVE Online UI at Using Bots for Eve Accessibility

A guide on how to set up and use the new tool is located in the Sanderling repository at https://github.com/Arcitectus/Sanderling/tree/master/implement/alternate-ui

1 Like

Good news for people working with EVE Online!

I recently explored memory reading from the 64-bit variant of the EVE Online client. It turns out we can continue using the same approach as before, after adapting to the new 64-bit pointer size. I implemented the memory reading for 64-bit clients and added the program to the Sanderling repository at https://github.com/Arcitectus/Sanderling/tree/master/implement/read-memory-64-bit

The new memory reading program is also available packaged in a command-line application. You can use the --pid parameter to pick a live process to read from or use the --source-file parameter to read from a process sample file. Using the --output-file parameter, you can make it save the memory reading results into a file for further inspection with your favorite JSON viewer.

Since I was working on memory reading, I used the opportunity to make some improvements to the serialized representation used by other software to consume the results of the memory reading. The names in there now match the names found in the EVE Online clients’ memory where applicable. Also, the overall structure of the serialized UI tree nodes now matches the one found in memory more closely: The properties coming from the python dictionary object are grouped accordingly in the JSON representation.

Another result of this exploration was a new tool to save process samples to files. The older one did not work correctly with 64-bit processes, so we now have a new reference tool to collect samples for memory reading development. While we use process samples also for development for EVE Online, the sample format is more general and can be used with other game clients as well. I uploaded a guide on how to use the tool at https://github.com/Viir/bots/blob/master/guide/how-to-collect-samples-for-64-bit-memory-reading-development.md

1 Like

@frodd reminded me that I can expand a bit more on how to use the new memory reading for 64-bit EVE Online clients.

In the last post, I had covered the foundation, the core implementation that is implemented in C# and can be integrated into applications in the form of an .NET assembly, packaged in a DLL.
Using the DLL is a way for people who already have significant experience with a programming language that easily integrates .NET assemblies.
But most people using Sanderling don’t have this kind of prior experience. Also, dealing with the DLL directly is unnecessarily complicated for most applications. For developing intel tools or bots, there is an easier way to use memory reading.

After implementing the 64-bit memory reading and said DLL, I looked for a way to package this in a way that is both easier to use for beginners and also a good starting point for typical applications.
As a result of this exploration, we now have two complete examples of how to use the new memory reading for 64-bit clients, without worrying about such implementation details like the DLL. These examples are two bots:

Since these have the most usage and test coverage, these are the best starting points for developing an intel tool or bot that works with the 64-bit client.
A quick summary of what these examples get you:

  • No need to think about a DLL, just write your program.
  • No need to use a programming language as complex as C#. No runtime exceptions like in C#. Instead, static analysis checks your program to point out problems before you start your program.
  • Parsing of the memory reading, to transform it in a form that is easier to use. The parsing also includes parsing of strings into numbers with uniform units (E.g., distance texts in the overview window).
  • In the mining bot: A framework that makes it easy to build advanced EVE Online bots, based on the learnings from bot development projects like the mission running bot and the anomaly ratting bot.

As always, I am glad to answer questions in case you run into problems.

An update regarding EVE Online development:

@ORly recently helped improve the robustness of the parsing of the EVE Online clients user interface. (Mining bot "I cannot see the ore hold capacity gauge.")

To summarize the developments: @ORly discovered a scenario in which reading the capacity of the selected container in the inventory failed. They provided training data for this scenario, resulting in an update to the standard parsing library for EVE Online. I updated the example apps here to use the new version of the parsing library:
https://github.com/Viir/bots/commit/43e433706b3673ce7fa66b286978ef165cc4bfdc

By the way: In case you are looking for an easy way to keep track of updates to the EVE Online parsing library, you can use the history view at https://github.com/Viir/bots/commits/master/implement/applications/eve-online/eve-online-mining-bot/src/EveOnline/MemoryReading.elm

The last six weeks have been quite busy. Now I am trying to remember the most outstanding developments, to give an overview.

After three weeks of testing and iterating, near the end of January, the new Tribal Wars 2 Farmbot was mature enough to replace the old bot from 2018. As I now see on the catalog, the new farmbot quickly became popular, rivalling the diverse applications for EVE Online.

For EVE Online, I also saw some noteworthy developments: After the mining bot had been ported to the 64-bit client, it was time to tackle the next in-game activity calling for automation. So in February, the anomaly ratting bot was updated to support the 64-bit game client too, just in time to prevent interruptions by the phase-out of the older game client.

Diverse questions led to clarifying how to develop for EVE Online, for example, how to efficiently find the objects in the game to use when working on an application.

Besides improving the guides, the framework and tooling for EVE Online itself saw several improvements: For example, the shortcut to inventory items now also works for all view types. Before, that shortcut was limited to the List view, which sometimes caused confusion. The setup process for the tool to inspect the parsed user interface was simplified, reducing the number of steps. Also, the inspection views were expanded to offer a comprehensive tree view of the parsed user interface in the game client. In the views for the individual UI nodes, you now also find the new property getAllContainedDisplayTexts. This addition in the inspection view reflects the widespread use of this derivation in bots and other apps.

Thank you all for sharing videos, screenshots, process samples and other data about your scenarios and helping to improve the bots!

Windows App Version 2020-03-05

Version 2020-03-05 of the Windows app brings several improvements to the user interface.

Making it easier to use online sessions

Some people had reported problems with starting online sessions. It was not always clear how to input the personal key into the UI. The first step to improve on this was expanding the guides to cover this in more detail. Now I also found a way to make the user interface itself simpler to prevent those issues in the future:
Starting online sessions is now easier with the new --online-session option on the run-bot command. When using this option, you don’t anymore have to copy your key for each bot. Instead the engine uses the key you stored as the default on your system. You can set the default key using the new online-session-key store-default-key command. To make this easier too, the web UI at http://app.botengine.org/ shows you the complete command. Just copy and paste to run it in the command prompt.

As usual, you can explore the new commands in the botengine app using the --help option:

PS C:\Users\John> botengine  online-session-key  --help                                                                             Subcommands to manage online bot session keys. You can test keys and store a default key.

Usage: BotEngine online-session-key [command] [options]

Options:
  -?|-h|--help        Show help information

Commands:
  reveal-default-key  Display the stored default key. Other commands don't display the key to avoid accidentally sharing this
                      secret.
  store-default-key   Store the default key to use when attempting to start an online bot session. I will use this key when you add
                      the '--online-session' option to a 'run-bot' command. The key will be stored in a file scoped to your Windows
                      user account
  test-key            Test a key to see if it can be used to start an online bot session.

Run 'online-session-key [command] -?|-h|--help' for more information about a command.

Improving error messages and recommendations

I refined several error messages and recommendations in the user interface. For example, it now makes suggestions in case of misspelling commands or options:

Contributions From the Community

@Lanson and @TheRealManiac refined guides and bots:

There is a new guide for EVE Online developers to find things faster in the game client user interface: bots/parsed-user-interface-of-the-eve-online-game-client.md at a364d5d7c120450786ba05de2ca5ad5dd3931523 · Viir/bots · GitHub

Visual maps help you where to find things and how shortcuts to popular elements are named in the framework:

It also contains examples of how to use these shortcuts in your app.

1 Like