Improved was finding memory objects in the scenarios you described.
I uploaded the release for you to test with at https://nofile.io/f/DnJDAtoKXbo/2018-07-25.Sanderling.ExploreProcessMeasurement.Debug.zip
In this archive, there is a file called Sanderling.ExploreProcessMeasurement.exe. When you run this, you should get the prompt I demonstrated how to work with in my post at Memory objects getting lost - #39 by Viir
So after a while when I’m playing more with multiple sanderlings/eve processes on same machine I can say, that on single instance/bot there are basically no issues at all, it can run 24/7 without a single issue.
However, when there are two clients running and two bots running - of course binded to proper eveprocess, problems starts to appear after couple of hours.
I does look like some memory readings are lost, eg. modules are blank, some buttons are blank or even drone window is blank, not always and not on both sanderlings at same time. Usually one client is broken while second one is working normally (same build same configuration).
Weirdly enough bot is not really trying to do anything with it, it does look like stucked, but usually hovering above modules get him back on track.
So my guess is, both processes are somehow sharing readings (which is unfortunate) and then one of the client shit self and start behaving weirdly.
Strange thing is, that eve restart and bot restart will usually not correct the issue in longterm, but whole windows restart will do the trick → but again for only several hours.
Strange thing #2 is that it is not happening from the beginning, but after couple of hours running.
Is there any way how to find out what is really happening on each client and why some mandatory readings becoming blank? Is windows somehow mangling memory readings/sharing processes? Is there better way how to separate them on same machine?
any ideas appreciated.
Thanks!
In same windows process or different VMs for each bot? Mine is in separate VMs on same server.
same machine, different users (not using VM’s)
so after trying some options what is going on, i’ve found out that code which is responsible for getting tooltips (and usually works normally) is not working when memorymeasurements are lost.
var moduleUnknown = MemoryMeasurementAccu?.ShipUiModule?.FirstOrDefault(module => null == module?.TooltipLast?.Value);
yield return new BotTask { Effects = new[] { moduleUnknown?.MouseMove() } };
and then bot is kind of stucked, until I physically move mouse over the module to obtain tooltip again.
So my question is more like, have anybody experienced something like that?
In my opinion, when measurement is lost, value is NOT NULL so this bit of code is not in charge anymore as it’s seen as known module.
Big question is what is actually there as a value, because Sanderling developer tool shows tooltion empty as well, but than it does not make much sense, why unknownmodule is not working.
Yes, we have tools to look closer at what is happening. We have a tool to save the memory contents of a windows process. So you can use this on the EVE online client to record its state.
There is a guide on how to use this tool at How to Collect Samples for 32-bit Memory Reading Development
When we have a process measurement, we can use the memory reading code on this as well to inspect where it goes wrong.
A process like the EVE Online client can ask windows how much memory is available or free. It can then adapt the way it stores things in memory to this limitation. So this could lead indirectly to the correlation you described (Dependency of the problem on the number of instances per machine).
An idea is to collect a sample of the EVE online process when you observe such a problem (using the guide from How to Collect Samples for 32-bit Memory Reading Development) and then take a closer look at the memory reading with this sample.
thanks @Viir
so if eve can ask windows about memory and then adjust storing things → which makes sense, why unknownmodule task is not able to get it again?
This is something I dont understand, cos its shown as blank in readings, so obviously that task should step in and populate it again.
will try to save the readings, last time i’ve tried it was not working eg. save to file was disabled
edit:
@Viir ok, I have taken measurements when bot ignored module with propmod (he did not ignored module with armorrepairer and those which are defined in config.)
So there is 300mb zip with tons of file in it, do we have some tool to explore it somehow → specifically how to find readings for ShipUI.Modules in those files?
Thanks!
investigating more further:
when single eve and single sanderling = no errors at all
when two eves and single sanderling = no errors at all
when two eves and two sanderlings = propmod and/or drone windows is ignored after some time
this got me thinking that issue might be related to sanderling which is kind of not able to separate processes correctly after some time.
interestingly, when am starting fresh sanderling identify each eve proces with ease automaticaly. (different logged users)
when its broken or looks like broken and i restart sanderling its not able to identify correct process anymore, its green but selected process belong to another user, i have to select correct one manualy, then it switch and measuring correct one - but usually after a short while its behave incorrectly again.
how is even possible that instance on one user is able to see process on another one?
(maybe running not as administrator will prevent sanderling to be able to see other processes?)
thinking out loud, sorry 
edit: running now three clients + three sanderlings not as administrator and seems to be ok for now. Will be observing it for a while.
note: is running sanderling not as admin still safe? cannot eve detect running non-admin processes to sent them to ccp?
edit 2:
ok so problem is there even not running as admin, it’s definitely better than running it as admin, but after a while modules are not read correctly, but rest of the bot is working like normally.
So definitely there is a problem with modules part.
edit 3:
so kind of confirmed, logged different ships with different number of modules, and sanderling which supposed to read 5modules is (even with selected correct process) switching to process which contains 3modules → and therefor it’s not reading modules correctly then.
Still strange, but definitely happening.
Dude, sounds like you are jumping to conclusions and have too low a data set.
some things can be observed, some cant
this clearly can be observed
and i do have plenty of measurements, but unsure what to do with them as they are hexed + am not really sure if measurements will say anything, as sanderling simply read wrong processes.
So measurement of eve process will not tell anything (imho)
Yes, for example, we have the Sanderling Process Measurement Explorer. With this tool, you can explore a process measurement and use the paths which are displayed in the Sanderling API Explorer to tell it which part of the Sanderling memory measurement to show you.
I compiled and uploaded binaries of this App, so people can use it without having to depend on Visual Studio. You can download the tool from https://nofile.io/f/DnJDAtoKXbo/2018-07-25.Sanderling.ExploreProcessMeasurement.Debug.zip
There is also this example of how to use this app:
Welcome to the Sanderling process measurement explorer.
Available commands are:
----
Command 'load-process-measurement-from-file':
Load a process measurement from the given file path.
----
Command 'read-memory-measurement-from-process-measurement':
Read a sanderling memory measurement from the currently loaded process measurement.
----
Command 'explore-memory-measurement':
Explore the sanderling memory measurement at the given path.
----
>load-process-measurement-from-file "C:\Users\John\Desktop\demo-sanderling\2018-07-10.MeasurementNoColumnHeader.zip"
Loaded process measurement with SHA1 of 05B7534F1DE715DB35F4E8EF5F37356C96218ABC
>read-memory-measurement-from-process-measurement
memory measurement read succesfully
567 UIElements found.
>explore-memory-measurement .WindowOverview?.FirstOrDefault()?.ListView?.ColumnHeader
Object found at path '.WindowOverview?.FirstOrDefault()?.ListView?.ColumnHeader':
Sanderling.Interface.MemoryStruct.IColumnHeader[]
Type.FullName: Sanderling.Interface.MemoryStruct.IColumnHeader[]
----
Properties of this object:
-
Length: 3
LongLength: 3
Rank: 1
SyncRoot: Sanderling.Interface.MemoryStruct.IColumnHeader[]
IsReadOnly: False
IsFixedSize: True
IsSynchronized: False
>
thanks @Viir
unfortunatelly ShipUIModule is not in that tool defined. And ShipUi has no tooltiptext, what do?
What is the console output from your test with the tool?
>read-memory-measurement-from-process-measurement
memory measurement read succesfully
669 UIElements found.
>explore-memory-measurement
Object found at path '':
Sanderling.Interface.MemoryStruct.MemoryMeasurement
Type.FullName: Sanderling.Interface.MemoryStruct.MemoryMeasurement
----
Properties of this object:
-
SessionDurationRemaining:
UserDefaultLocaleName: en-EN
VersionString:
ScreenSize: A = 1 330, B = 965
Menu:
Tooltip:
ShipUi: Sanderling.Interface.MemoryStruct.ShipUi
Target: Sanderling.Interface.MemoryStruct.ShipUiTarget[]
InflightBracket: Sanderling.Interface.MemoryStruct.IInSpaceBracket[]
ModuleButtonTooltip:
SystemMenu:
Neocom: Sanderling.Interface.MemoryStruct.Neocom
InfoPanelButtonCurrentSystem: Sanderling.Interface.MemoryStruct.UIElement
InfoPanelButtonRoute: Sanderling.Interface.MemoryStruct.UIElement
InfoPanelButtonMissions:
InfoPanelButtonIncursions:
InfoPanelCurrentSystem: Sanderling.Interface.MemoryStruct.InfoPanelSystem
InfoPanelRoute: Sanderling.Interface.MemoryStruct.InfoPanelRoute
InfoPanelMissions:
Utilmenu:
AbovemainMessage:
AbovemainPanelGroup:
AbovemainPanelEveMenu:
WindowOther:
WindowStack: Sanderling.Interface.MemoryStruct.WindowStack[]
WindowOverview: Sanderling.Interface.MemoryStruct.IWindowOverview[]
WindowChatChannel: Sanderling.Interface.MemoryStruct.WindowChatChannel[]
WindowSelectedItemView: Sanderling.Interface.MemoryStruct.IWindowSelectedItemView[]
WindowDroneView: Sanderling.Interface.MemoryStruct.IWindowDroneView[]
WindowPeopleAndPlaces:
WindowStation:
WindowShipFitting:
WindowFittingMgmt:
WindowSurveyScanView:
WindowInventory: Sanderling.Interface.MemoryStruct.IWindowInventory[]
WindowAgentDialogue:
WindowAgentBrowser:
WindowTelecom:
WindowRegionalMarket:
WindowMarketAction:
WindowItemSell:
WindowProbeScanner: Sanderling.Interface.MemoryStruct.IWindowProbeScanner[]
>
and ShipUI
>explore-memory-measurement .ShipUi
Object found at path '.ShipUi':
Sanderling.Interface.MemoryStruct.ShipUi
Type.FullName: Sanderling.Interface.MemoryStruct.ShipUi
----
Properties of this object:
-
Center: Sanderling.Interface.MemoryStruct.UIElement
Indication: Sanderling.Interface.MemoryStruct.Container
HitpointsAndEnergy: Sanderling.Interface.MemoryStruct.ShipHitpointsAndEnergy
SpeedLabel: Sanderling.Interface.MemoryStruct.UIElementText
EWarElement: Sanderling.Interface.MemoryStruct.ShipUiEWarElement[]
ButtonSpeed0: Sanderling.Interface.MemoryStruct.UIElement
ButtonSpeedMax: Sanderling.Interface.MemoryStruct.UIElement
Module: Sanderling.Interface.MemoryStruct.ShipUiModule[]
Readout: Sanderling.Interface.MemoryStruct.IUIElementText[]
SpeedMilli:
Timer:
SquadronsUI:
ButtonText:
InputText:
LabelText:
Sprite:
Region: Min0 = 0, Min1 = 0, Max0 = 0, Max1 = 0
InTreeIndex:
ChildLastInTreeIndex:
RegionInteraction: Sanderling.Interface.MemoryStruct.ShipUi
Id: 0
>
As Modules tooltipstext are populated to memParsed, I guess it’s read from something else and then parsed by Sanderling, right?
To get back to your question:
In the logs I see you explored paths ‘’ and path ‘.ShipUi’. Did you expect ShipUIModule or tooltiptext in any of those? If so why? I do not understand yet were these are coming from.
I searching the Sanderling repository for those:
I do not understand how tooltipstext would be connected to Sanderling.
Github does not find this in the repository: Repository search results · GitHub
We couldn’t find any code matching ‘tooltipstext’ in Arcitectus/Sanderling
I meant “ToolTipLast” (typing on ipad is not really great usually )
When ToolTipLast is empty → and which is empty if it is broken, I’m wondering why
var moduleUnknown = MemoryMeasurementAccu?.ShipUiModule?.FirstOrDefault(module => null == module?.TooltipLast?.Value);
yield return new BotTask { Effects = new[] { moduleUnknown?.MouseMove() } };
this bit of code won’t step in (as it does after undock)
Only reason from my perspective is, that this variable is not actually NULL and therefore this bit of code is not working (but it’s just my wild guess which I can’t really confirm)
Issue with this is, that we dont know why it’s missing, cos in those readings there is nothing like ToolTipLast defined so I cant’ search if from EVE process.
Or can I?
you can see on screenshot, that tooltip is missing from module[0] even when hovered mouse over.
note: more observation during the day told me, that FIRST sanderling launched never fails, it’s just running whole day without any issue.
SECOND or THIRD sanderling launched always fail after like 2-3hours of running.
Separated users, same machine.
And issue is always the same : missing ToolTipLast and thus not activating/deactivating propmod and/or dronewindow is being ignored.
So after some fiddling around, I’m testing ToolTipLast for null and for empty string, and it seems like this helped.
No real idea why it’s not null tho.
It feels like workaround more than solution tho.
I looked up ToolTipLast. The ToolTipLast I found is available under MemoryMeasurementAccu. The properties under MemoryMeasurementAccu are not read from memory but derived from the memory measurement which is available under MemoryMeasurement. The Accu part of the symbol stands for accumulation, as the properties in there are an accumulation of what was part of the memory measurement in the past.
So, to see where the value of ToolTipLast is coming from, I see no need to consider contents of the EVE Online client. I would follow the code in the Framework which assigns to this property:
In the screenshot, I do not see the complete path, maybe the screenshot is clipped at the top. My guess is that the path is under MemoryMeasurementAccu. Because of this, I would not expect the properties in there to reflect what is on the Screen in the EVE Online client. So there is not yet a connection to even when hovered mouse over.
To see what is on the Screen in the EVE Online client, look under the properties under the node MemoryMeasurement. The ship module tooltip is at MemoryMeasurement.Value.ModuleButtonTooltip.
To clarify this, I wrote a guide at Sanderling Framework - Differences between MemoryMeasurement, MemoryMeasurementParsed and MemoryMeasurementAccu
Thank you @Viir that helped a lot!
@Viir can you please give me a hint what exactly determine how long bot is hovering above given module to read and parse modulename?
Scenario is: CPU under heavy load, bot is cycling on modules to read them, but bot moves to another module faster then needed for read/parse module name and save that to memAccu.ToolTipLast.
Thanks
edit:
Ok so the actual code
var moduleUnknown = MemoryMeasurementAccu?.ShipUiModule?.FirstOrDefault(module => null == module?.TooltipLast?.Value);
Is simply not enough to be sure that module is really recognized.
I was debugging it for couple of days, and what is happening is, that if bot is quicker than all information from ToolTip is read/parsed and he leaves to another module, then ToolTipLast IS populated, but not with all values, especially with crucial part
LabelText
Which is used on many places determining propulsion, repairers etc.
So what you wanna do to gather all needed is change that line above like this
var moduleUnknown = MemoryMeasurementAccu?.ShipUiModule?.FirstOrDefault(module => null == module?.TooltipLast?.Value?.LabelText?.Any());
This force bot to hover above modules as long as needed to ALL fields are filled with information.
It’s been a mystery really, but … hope it helps somebody.
Cheers