I few days ago i was very pleased to find this place (still am). About 8 years ago I used to script simple OCR bots which was very time consuming. This approach seems a lot more sophisticated.
I’ve been reading “guides” and watched a few of your videos, started with the 0-warp and instructions how to edit. But i can’t really figure out how or where to start from scratch and use memory information to create a basic script. That’d be a video i would love to see. To learn how to use the ui-tree instead of the already parsed interface, because i may want to use elements from the uiTree other than the presets.
How would i find a memory object? i’m aware of the memory reading framework and the alternate ui. What i don’t know how to find the object information with those tools and translate that to be used in a script. The examples I’ve seen so far are usually one of the presets or functions already available.
I haven’t found a guide with newbie-proof examples how to do this from start to finish.
Find the object information with a memory reading tool. i.e. Asteroid survey scan
Make a new (elm) script from scratch
use the script to parse the collected information from memory to the console-output and/or variable for processing
With the information i hope to learn the basics and expand upon it.
Nice questions! Indeed, there is no guide yet to focus on this particular sequence. That makes this thread the start of the development of this guide. Eventually, I will get to make a video based on your questions. For now, I have to refine the guide a little bit first.
For our scenario, an essential part of that guide is this:
Using the uiTree field, we can access the raw UI tree. All the other fields contain derivations of the UI tree for easier access.
Here we look at the scenario where all these other fields do not exist, so we want to get the objects of interest from scratch from the uiTree field. To put it in other words, we do not use 95% of the ParseUserInterface.elm module.
The Challenge
The tree we find in the uiTree field can contain thousands of nodes and tens of thousands of properties. Finding the object information means that we learn what features are unique to the object(s) we want to extract. It is finding a pattern that is common to all appearances of this object. This pattern has the job of filtering out all other nodes from the UI tree that we do not consider to represent the object. When we have found a working pattern, we can encode it in a function that takes the UI tree as input and returns our object(s).
Finding a Pattern
To find a pattern, we need examples. Usually, we save those examples to files so that we can return to the same samples when new questions come up. In the standard file format, the UI tree is represented as a JSON tree. For our challenge, we don’t need to know how the details of this representation. It is more important to know which tools produce the files in this format and which one can read it.
Collecting Examples
There are various ways to get samples of the UI tree in the right format:
One source for UI tree samples is to run an app with the botengine. When you run an EVE Online app, the engine automatically saves the readings to files. This source is interesting when the scenario of interest only appears sporadically, for example, once per ten hours on average.
If you can bring the object of interest on the screen quickly, you can use the route via a game client process sample. In this case, you initiate taking a reading manually, similar to taking a photo: You choose the right time when things on the screen appear in a representative way.
Another option is using the Alternate UI for EVE Online. It contains a button to save the last reading to a file.
Finding a Pattern in the Examples
When we have an example, we want to find out which subtree in the whole UI tree contains the object we are interested in. The most common way is to use the interactive explorer that is included in the Alternate UI for EVE Online because it contains some tools to help us find our object:
Contained display text. Often the object of interest contains some display text. We can expand the getAllContainedDisplayTexts node in the tree explorer to see the display texts contained in that node. This way, we can make sure we are following the right path when descending into the tree.
Total display region: The region in the field totalDisplayRegion uses coordinates relative to the upper left corner of the game client. This coordinate system makes it easy to compare with screenshots. We can open a screenshot of our scenario in MS paint and read the pixel coordinates of where our object appears.
When we found the object in the examples, we can start finding a common pattern that allows finding the object in new scenes. In contrast to the approach above, we cannot necessarily depend anymore on a particular location relative to the whole game client window. Some objects can be found in different places.
One often-used property to distinguish objects is the name of the python type of the node object. An extreme example of this is how the parsing framework find locked targets: It only uses this type name, nothing else, as we can see in the parseTargetsFromUITreeRoot function:
With this function, we see anything as a target, as long as it’s python type name is “TargetInBar”.
Sometimes, it is that simple. In other cases, we use a more sophisticated pattern, combining multiple filtering criteria. Spatial criteria can be a part of it, especially when used within a subtree. For example, we might find our object to be always located in the upper half of a particular window.
Thank you very much for taking the time to explain this. I believe i understand what you’re saying. I used the Alternate UI and kept expanding until i found what i was looking for. i.e:
I believe that the pythonObjectTypeName = “SurveyScanView” is unique but has a few children/containers perhaps are not so useful. If i dig down a little more i get to “pythonObjectTypeName = “SurveyScanEntry”” and see the entries like: “Concentrated Veldspar26.7272.672 m322 km”
I’ve started reading up on ELM and how the syntax works.
Perhaps a step-by-step explanation what the above code actually does? At the risk of making a fool of myself I’ll make an attempt to read it:
parseTargetsFromUITreeRoot : UITreeNodeWithDisplayRegion → List Target
: →
A function “parseTargetsFromUITreeRoot” is declared, that has an input value “UITreeNodeWithDisplayRegion” and will return a list as a variable named “Target”
I am not sure what the next few lines exactly do. List.filter seems a default function on the list-type that can filter a list based on an input value and returns one (or more) values that then get picked up by map that puts the results in a new list.
In general, the guide can be expanded much further. I just posted the first part today when taking the first break.
But I look at your current project first: Using the python type name “SurveyScanView” looks plausible. In your screenshot, I also see that node has 70 descendants and contains nine display texts. All in all, it looks like this is the whole “Survey Scan Results” window. You can check this by comparing the display region shown in the tree explorer with the coordinates from a screenshot of the same scenario.
The search depends on what you want. Your code above looks like you want just the individual scan results and not the whole window. In this case, the search would look like this:
The first line is the type annotation (Reading Types · An Introduction to Elm). The type annotation is optional; it does not affect how the program works. The type annotation constrains the types that can flow into and out of the function below, and this allows the development tool to give us more precise error messages.
The second line repeats the function name, lists all parameters that we want to name/bind here, and then concludes with the equals sign.
The third line uses the function listDescendantsWithDisplayRegion. As we can see in the type annotation of listDescendantsWithDisplayRegion, it takes a UITreeNodeWithDisplayRegion and returns a List UITreeNodeWithDisplayRegion. That list contains all descendants of that input node.
After we got all UI nodes, we filter them on the fourth line, using List.filter.
Let’s unpack what is going on in that line: The expression in parenthesis after List.filter combines three functions into one, using the >> operator two times between the individual functions.
The function .uiNode takes a record and returns the value of the record field named uiNode.
The function .pythonObjectTypeName takes a record and returns the value of the record field named pythonObjectTypeName.
The function (==) "SurveyScanEntry" returns a Bool value indicating if the arguments equals "SurveyScanEntry"
The three combined functions are all anonymous. They are not defined anywhere else, so we can determine the type without reading anything else about the project. We can use the Elm REPL tool to make the function type visible.
PS C:\Users\John> elm repl ---- Elm 0.19.1 ----------------------------------------------------------------
Say :help for help and :exit to exit! More at <https://elm-lang.org/0.19.1/repl>
--------------------------------------------------------------------------------
> (.uiNode >> .pythonObjectTypeName >> (==) "SurveyScanEntry")
<function>
: { b | uiNode : { a | pythonObjectTypeName : String } } -> Bool
>
So the tool tells us the type of this value is this:
{ b | uiNode : { a | pythonObjectTypeName : String } } -> Bool
It takes a record that has a field named uiNode which in turn has a field named pythonObjectTypeName which needs to be of type String. This constraint applies to the type UITreeNodeWithDisplayRegion in our project. So we can use this function with List.filter on a List UITreeNodeWithDisplayRegion to get the filtered list.
I see the expression (==) "SurveyScanEntry" can be explained in more detail. The expression (==) "SurveyScanEntry" takes the function == and applies the argument “SurveyScanEntry” to this function.
We can look at the function == in isolation by using the REPL again. When we enter (==) in the REPL, we get this:
> (==)
<function> : a -> a -> Bool
The function == itself takes two arguments and returns a Bool. By giving it only one argument of “SurveyScanEntry”, we get a new function which only takes one argument, which needs to be of type String.
Using the REPL again:
Looks like I ran into the post length limit with my post above, so I am leaving this here for now. This is an expansion on “Finding a Pattern in the Examples”:
listDescendantsWithDisplayRegion
In the example of getting the targets above, we can also see the function listDescendantsWithDisplayRegion which is one of the most often used when parsing the UI tree. This function returns all the descendants of a node, that is, all nodes in the subtree in a list. This transformation from the tree structure into a list is common because it enables us to use the core List functions down the line. In the existing parsing implementation, we can see the use of listDescendantsWithDisplayRegion often followed by the use of List.filter, which returns a sublist based on the given filter criteria.
The most important way to combine multiple search criteria is hierarchical composition: That means we search for a node using one filter. Then we search the subtree under that node using another filter. Here is a concrete example: We might want to find a button in a window. We can use the Python type name for the button and the display text contained in the button. But for some button labels, there could be multiple buttons in the game with the same label. To ensure we do not accidentally pick up some other button, we can insert an intermediate step to constrain the search to the specific window we want to search in. Many windows in the EVE Online UI tree have a very specific Python type name that helps us find them very easily, like “OverView”, “DroneView”, “ProbeScannerWindow”, etc.
