Memory objects getting lost

I just messaged you with another example where the “measurement?.Value?.WindowOverview?.FirstOrDefault()?.ListView?.ColumnHeader” element is missing even though it is shown in Eve. Again, restarting Sanderling, or using different versions, does not fix the issue. The Eve client must be restarted.

Good, having two examples might make it easier find the problem. I will take a look at the examples next week.

Just a friendly bump. Something like this happened twice today.

I am looking at sample 25B0FB541006BC6E96BE9016BE205260550B8FFE:

I used this sample with Demo_memory_reading_from_process_sample and a memory measurement was read from the sample. I see an object referenced by the local memoryMeasurement. Do you see a different result?

I am testing sample 05B7534F1DE715DB35F4E8EF5F37356C96218ABC with Demo_memory_reading_from_process_sample:
Translated to this environment, I guess the path you mentioned is memoryMeasurement?.WindowOverview?.FirstOrDefault()?.ListView?.ColumnHeader, is that correct?

Visual Studio shows me a Value of {Sanderling.Interface.MemoryStruct.IColumnHeader[3]} for the expression of that path. Not sure where Visual Studio takes Value from, my guess is ToString. Do you see a different result?

I’m not sure if you’re saying there’s a problem with the first example:
I was able to do it the way detailed in the How To Save A Memory Measurement To A File thread. The trick is to open Notepad as admin, and File->Open within it. Within the Open window drag the file/folder. The result seems to be about 270mb, so I will upload that soon with an explanation of what is missing from the snapshot. SanderlingGUIReadErrorMeasurement.zip

Even after restarting Sanderling the following element was not being found: ShipUI.SquadronsUI.SetSquadron[1].Squadron.LabelText
It should say “Ready” because it is not in space. And you can see where the others say “In Space” in .SetSquadron[0 and 2]. However, the GUI shows nothing even after restarting it. I have to restart the Eve client for it to start being found again. This occasionally happens to other elements.

In the second example (using the same method given in How To Save A Memory Measurement To A File) where the “measurement?.Value?.WindowOverview?.FirstOrDefault()?.ListView?.ColumnHeader” element is missing. Missing ColumnHeader Element. It should have multiple children, one for each header column.

Alright, I spent some more time on this and there is some progress: I see a new approach to adapt the code to help find the elements. Looks like there will soon be an app for you to test with.

2 Likes

Awesome!!! Can only click the love button once, sad.

I invested four hours into this today.
As a result, there is a new tool to find objects in memory measurements:
https://github.com/Arcitectus/Sanderling/commit/b0f73f76ae246b3322d10b7c3abbf35badb25c36

From the apps output, it seems like there is an improvement:

Welcome to the Sanderling process measurement explorer.
Available commands are:
----
Command 'load-process-measurement-from-file':
Load a process measurement from the given file path.
----
Command 'read-memory-measurement-from-process-measurement':
Read a sanderling memory measurement from the currently loaded process measurement.
----
Command 'explore-memory-measurement':
Explore the sanderling memory measurement at the given path.
----
>load-process-measurement-from-file "C:\Users\John\Desktop\demo-sanderling\2018-07-10.MeasurementNoColumnHeader.zip"
Loaded process measurement with SHA1 of 05B7534F1DE715DB35F4E8EF5F37356C96218ABC
>read-memory-measurement-from-process-measurement
memory measurement read succesfully
567 UIElements found.
>explore-memory-measurement .WindowOverview?.FirstOrDefault()?.ListView?.ColumnHeader
Object found at path '.WindowOverview?.FirstOrDefault()?.ListView?.ColumnHeader':

Sanderling.Interface.MemoryStruct.IColumnHeader[]
Type.FullName: Sanderling.Interface.MemoryStruct.IColumnHeader[]
----
Properties of this object:
-
Length: 3
LongLength: 3
Rank: 1
SyncRoot: Sanderling.Interface.MemoryStruct.IColumnHeader[]
IsReadOnly: False
IsFixedSize: True
IsSynchronized: False
>

I uploaded the executable binary at https://nofile.io/f/DnJDAtoKXbo/2018-07-25.Sanderling.ExploreProcessMeasurement.Debug.zip

1 Like

Thank you very much. I just want to make sure I understand.
This new tool looks like it allows for easier memory exploration from multiple input sources.

What was improved? I looked at the July Sanderling release and couldn’t tell if it was enhanced to be able to find elements when they are lost, like my examples?

I’m guessing that the bot could check “IsSynchronized” to see if there’s an error. But, can anything be done to synchronize the element after it returns false?

Improved was finding memory objects in the scenarios you described.

I uploaded the release for you to test with at https://nofile.io/f/DnJDAtoKXbo/2018-07-25.Sanderling.ExploreProcessMeasurement.Debug.zip

In this archive, there is a file called Sanderling.ExploreProcessMeasurement.exe. When you run this, you should get the prompt I demonstrated how to work with in my post at Memory objects getting lost - #39 by Viir

So after a while when I’m playing more with multiple sanderlings/eve processes on same machine I can say, that on single instance/bot there are basically no issues at all, it can run 24/7 without a single issue.

However, when there are two clients running and two bots running - of course binded to proper eveprocess, problems starts to appear after couple of hours.
I does look like some memory readings are lost, eg. modules are blank, some buttons are blank or even drone window is blank, not always and not on both sanderlings at same time. Usually one client is broken while second one is working normally (same build same configuration).
Weirdly enough bot is not really trying to do anything with it, it does look like stucked, but usually hovering above modules get him back on track.

So my guess is, both processes are somehow sharing readings (which is unfortunate) and then one of the client shit self and start behaving weirdly.

Strange thing is, that eve restart and bot restart will usually not correct the issue in longterm, but whole windows restart will do the trick → but again for only several hours.
Strange thing #2 is that it is not happening from the beginning, but after couple of hours running.

Is there any way how to find out what is really happening on each client and why some mandatory readings becoming blank? Is windows somehow mangling memory readings/sharing processes? Is there better way how to separate them on same machine?

any ideas appreciated.
Thanks!

1 Like

In same windows process or different VMs for each bot? Mine is in separate VMs on same server.

same machine, different users (not using VM’s)

so after trying some options what is going on, i’ve found out that code which is responsible for getting tooltips (and usually works normally) is not working when memorymeasurements are lost.

			var moduleUnknown = MemoryMeasurementAccu?.ShipUiModule?.FirstOrDefault(module => null == module?.TooltipLast?.Value);

			yield return new BotTask { Effects = new[] { moduleUnknown?.MouseMove() } };

and then bot is kind of stucked, until I physically move mouse over the module to obtain tooltip again.

So my question is more like, have anybody experienced something like that?
In my opinion, when measurement is lost, value is NOT NULL so this bit of code is not in charge anymore as it’s seen as known module.

Big question is what is actually there as a value, because Sanderling developer tool shows tooltion empty as well, but than it does not make much sense, why unknownmodule is not working.

  • it’s only happening when there is more than one instance of eve+bot running

Yes, we have tools to look closer at what is happening. We have a tool to save the memory contents of a windows process. So you can use this on the EVE online client to record its state.
There is a guide on how to use this tool at How to Collect Samples for 32-bit Memory Reading Development

When we have a process measurement, we can use the memory reading code on this as well to inspect where it goes wrong.

A process like the EVE Online client can ask windows how much memory is available or free. It can then adapt the way it stores things in memory to this limitation. So this could lead indirectly to the correlation you described (Dependency of the problem on the number of instances per machine).

An idea is to collect a sample of the EVE online process when you observe such a problem (using the guide from How to Collect Samples for 32-bit Memory Reading Development) and then take a closer look at the memory reading with this sample.

thanks @Viir
so if eve can ask windows about memory and then adjust storing things → which makes sense, why unknownmodule task is not able to get it again?
This is something I dont understand, cos its shown as blank in readings, so obviously that task should step in and populate it again.

will try to save the readings, last time i’ve tried it was not working eg. save to file was disabled

edit:
@Viir ok, I have taken measurements when bot ignored module with propmod (he did not ignored module with armorrepairer and those which are defined in config.)
So there is 300mb zip with tons of file in it, do we have some tool to explore it somehow → specifically how to find readings for ShipUI.Modules in those files?
Thanks!

investigating more further:
when single eve and single sanderling = no errors at all
when two eves and single sanderling = no errors at all
when two eves and two sanderlings = propmod and/or drone windows is ignored after some time

this got me thinking that issue might be related to sanderling which is kind of not able to separate processes correctly after some time.

interestingly, when am starting fresh sanderling identify each eve proces with ease automaticaly. (different logged users)
when its broken or looks like broken and i restart sanderling its not able to identify correct process anymore, its green but selected process belong to another user, i have to select correct one manualy, then it switch and measuring correct one - but usually after a short while its behave incorrectly again.

how is even possible that instance on one user is able to see process on another one?
(maybe running not as administrator will prevent sanderling to be able to see other processes?)

thinking out loud, sorry :slight_smile:

edit: running now three clients + three sanderlings not as administrator and seems to be ok for now. Will be observing it for a while.

note: is running sanderling not as admin still safe? cannot eve detect running non-admin processes to sent them to ccp?

edit 2:
ok so problem is there even not running as admin, it’s definitely better than running it as admin, but after a while modules are not read correctly, but rest of the bot is working like normally.
So definitely there is a problem with modules part.

edit 3:
so kind of confirmed, logged different ships with different number of modules, and sanderling which supposed to read 5modules is (even with selected correct process) switching to process which contains 3modules → and therefor it’s not reading modules correctly then.
Still strange, but definitely happening.

Dude, sounds like you are jumping to conclusions and have too low a data set.