kelltom / os-bot-color Goto Github PK

I like this project very much, but I have some problems in local debugging. I want to ask these questions. First, whether Python 3.8 is applicable. My local environment is Python 3.8, but some code can be directly run after modification. However, I am worried that the code in other places may conflict with Python 3.8. In addition, in the Creating Your First Bot tutorial in the wiki, the Note mentioned in the subtitle Test the Bot: " If you are running RuneLite v1.9.11.2 or greater, OSBC may ask you to locate the Profile Manager folder as well. This is typically located in C:\Users<username>.runelite\profiles2. You must select the profiles2 folder, not the .runelite folder.", there is no C:\Users<username>.runelite\profiles2. path on my local computer. How is this path generated?

Problem
RuneLite Home Views are used as a starting point for users. It tells them about the game they are about to bot on, why it needs to be launched with customized settings, and gives the ability to do so. Despite offering the same functionality, Home Views and the game_launcher utility are separated entirely. Both need to send updates/information to different parts of the UI, so merging the functionality is complex.

On top of this, these RuneLite Home Views and launcher methods are catered specifically to the official RuneLite client - meaning games that have "pseudo-RuneLite" implementation (E.g., Ikov uses a 317 client with a custom RL layer on top) cannot make use of either. The problem arises when you create a new bot for one of these games; since the game has RL features, it should be an instance of RuneLiteBot -- but that forces it to rely on RL Home Views, which will not work since the game isn't really RuneLite.

Solution
This is a complex issue that requires some architectural changes. I don't know the solution right now. If OSBC is going to get to a point where bots can be added as single files, then Home Views either need to be removed or made such that they can be customized from the Bot side of things.

_OSR-20

Problem
Mouse click distribution seems to be the primary way that bots are detected. Clicking in the same spot over and over is an easy way to get a ban. Normally distributed clicks are an improvement, but still, show an obvious pattern over time.

Solution
This video series is a good experiment to show the issue and solution. For objects we are clicking, we need to select multiple reference/base points as "targets", and distribute clicks based on them with some skewness. The exact implementation is a bit beyond me at the moment, but there has been some good discussion in Discord about how it could be done.

as discussed in #35

Describe the bug

If your display has scaling changed from 100% the program cannot resize the client to start the bot correctly. It also messes with the osrs COLOR menu also. When I had my 4k display set to 225% scaling the OSRS COLOR display was nearly full screen and couldn't be resized smaller. This scaling also effected the bot as it couldn't see the full in game client when it changed to its fixed size.

To Reproduce

right click desktop select " Display settings"
Under "Scale and layout" change scaling to anything other then 100%
Run the bot and see how it fails to start correctly as it cannot resize to fixed classic layout.

Screen Shots

This is the smallest the menu can be at with 225% scaling.

This is what the in game client looks like at 225% scaling after bot has started running. It also causes the game to crash each time.

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

• Currently for what i noticed from the code. Launching the game is reliant on user having basic osrs account and not jagex account. Game launching is impossible when user has turned their account into jagex account.

Describe the solution you'd like
A clear and concise description of what you want to happen.

• Solution would be to create few extra steps on launching to enable running through jagex account.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

• Possibly just launching the game yourself through jagex account and then selecting the launched runelite window and applying the profile needed.

Additional context
Add any other context or screenshots about the feature request here.

• For context I am willing to give this a try myself. I finished my bachelor's in CS and I was keen on doing this sort of project myself only to find out this kind of application already exists. IF time gives, I will try to create solution myself on this problem and create pull request if possible.

as discussed in ...

Hey,

I don't have a way to message you so can we use this thread as a Q&A board? Or is there another avenue you prefer?

Is your feature request related to a problem? Please describe.
If you want to right click on an item in your inventory and select one of the options, you can't using the nice rect=window views because the options span both the game_view and the control_panel.

Describe the solution you'd like
A new Window.py class attribute that includes the whole RuneLite game window

_OSR-17

There are many use cases where it makes sense to grow or shrink a Rectangle (or even the bounding box of a RuneLiteObject). We should add functions to those classes to do so.

These functions should be able to manipulate the rectangle based on percentages or pixel values, on any axis. We must also be careful to adjust any properties that this might have side effects on (e.g., width and x_max).

_{From SyncLinear.com | OSR-29}

This is a bit of a weird bug. The stop() function is called from the UI's stop button, which correctly stops the bot. However, when you call this function within your script's code, it doesn't set the Bot Status on the UI.

The solution is to shuffle around the statement that sets the status to STOPPED.

_{From SyncLinear.com | OSR-38}

The Morg API utility is missing a function to check if the inventory is full.

Similar logic from the StatusSocket utility could be used to implement this.

_{From SyncLinear.com | OSR-39}

Problem
In real life, humans move the mouse around constantly. They don't sit and wait for their character to complete an action; they anticipate the completion and prepare for it. At the moment, OSBC lets you move the mouse in a blocking manner. Despite the mouse movements looking convincing, the pattern in which they occur is robotic looking.

Solution
We can implement an asynchronous mouse utility. This utility could have a function that runs on a thread with the start of the bot, and it could loop, constantly moving the mouse according to the user's demands. When a demand is met, it can then decide it's own movements to emulate a human just fiddling the mouse around. The challenge is deciding how the user should give these commands. Do we make a special class called MouseCommand that dictates all the arguments the usual move_to() function accepts and allow users to assign it to some global variable that the loop function is watching?

This can be implemented in tandem with #29

_OSR-21

Traceback (most recent call last):
File "C:\Users\Mammoth\PycharmProjects\pythonProject3\OSRS-Bot-COLOR\src\OSBC.py", line 6, in
import customtkinter
ModuleNotFoundError: No module named 'customtkinter'

This may be a simple fix but i'm pretty new to coding and have tried several different methods to fix this. Any help would be appreciated.

The Profile Manager is a new RuneLite feature for managing properties files (profiles). This issue seems to be affecting anyone with a new install of RuneLite.

Findings from Discord user:

"Looks like if your on a new install of runelite you can't launch with the --config. You can launch with --profile. Runelite Will create a profile that names and points to our path for temp.properties and names the profile to the path. It even launches with that profile when you launch runelite through OSBC. Unfortunately it refuses to load that profiles settings. Now if you import the profile and name it something like OSBC and change the launch args to

EXECARG2 = f"--profile=OSBC --sessionfile=bot_session"

It will load the profile correctly.

I believe this has something to do with signature they are adding to the end of profile names."

_{From SyncLinear.com | OSR-45}

A user reported the following error:

Exception in thread Thread-6:
Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64__qbz5n2kfra8p0\lib\threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "c:\Users\user\Documents\OSRS-Bot-Shared\src\model\bot.py", line 42, in run
    self.target()
  File "c:\Users\user\Documents\OSRS-Bot-Shared\src\model\osrs\combat\combat.py", line 117, in main_loop
    if self.mouse.click(check_red_click=True):
  File "c:\Users\user\Documents\OSRS-Bot-Shared\src\utilities\mouse.py", line 99, in click
    return self.__is_red_click(mouse_pos_before, mouse_pos_after)
  File "c:\Users\user\Documents\OSRS-Bot-Shared\src\utilities\mouse.py", line 139, in __is_red_click
    cursor_sct = Rectangle.from_points(top_left_pos, bottom_right_pos).screenshot()
  File "c:\Users\user\Documents\OSRS-Bot-Shared\src\utilities\geometry.py", line 70, in screenshot
    res = np.array(sct.grab(monitor))[:, :, :3]
  File "C:\Users\user\Documents\OSRS-Bot-Shared\src\env\lib\site-packages\mss\base.py", line 76, in grab
    return self._grab_impl(monitor)
  File "C:\Users\user\Documents\OSRS-Bot-Shared\src\env\lib\site-packages\mss\windows.py", line 261, in _grab_impl
    self._data = ctypes.create_string_buffer(width * height * 4)  # [2]
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64__qbz5n2kfra8p0\lib\ctypes\__init__.py", line 63, in create_string_buffer
    buftype = c_char * init
ValueError: Array length must be >= 0, not -76104

_{From SyncLinear.com | OSR-35}

Problem
Mouse clicks are currently implemented just by using pag.click(). While being unlikely, it might be possible that the duration between mouse button presses is tracked for anti-bot purposes.

Solution
We could add a normal distributed pause between mb presses. One recording of gameplay showed the following:

• lower_bound = 30ms
• upper_bound = 1500ms
• avg = 133.37
• num_of_click = 9033

This can be achieved using the RandomUtil function, truncated_normal_sample().

as discussed in Discord

The StatusSocket constructor creates a localhost server on port 5000. The StatusSocket server does not automatically destroy itself when the class instance leaves the main_loop scope. If a user starts/stops a bot that uses StatusSocket, it seems that the server lingers. It is a daemon to the OSBC app itself, though, so it will stop when the program is closed, but it would probably be ideal if it wasn't running unnecessarily.

It is probably a good idea to implement this as a context manager, or at least provide a method that allows bot developers to explicitly stop the server when their bots stop.

class StatusSocket:
    gameTick = 0.603

    def __enter__(self):
        t_server = Thread(target=self.__RSERVER)
        t_server.daemon = True
        t_server.start()
        print("thread alive:", t_server.is_alive())
        return self

    def __exit__(self, exc_type, exc_value, traceback):
        self.stop()

    def __RSERVER(self, port=5000):
        try:
            self.httpd = HTTPServer(("127.0.0.1", port), RLSTATUS)
            self.httpd.serve_forever()
        except OSError:
            print("Status socket already running.")

    def stop(self):
        if hasattr(self, 'httpd') and self.httpd:
            self.httpd.shutdown()

def main_loop(self):
    # Setup API
    with StatusSocket() as api_s:
        ...
        # do some stuff with the api

    # The HTTP server is automatically shut down when the block exits

_{From SyncLinear.com | OSR-48}

Describe the bug
When installing via the instructions and I try to run the demo bot I get the following error:

Traceback (most recent call last):
  File "C:\Users\sivar\PycharmProjects\OSRS-Bot-COLOR\src\OSRS Bot COLOR.py", line 1, in <module>
    from controller.bot_controller import BotController
  File "C:\Users\sivar\PycharmProjects\OSRS-Bot-COLOR\src\controller\bot_controller.py", line 5, in <module>
    from model.bot import Bot, BotStatus
  File "C:\Users\sivar\PycharmProjects\OSRS-Bot-COLOR\src\model\__init__.py", line 1, in <module>
    from .alora import *
  File "C:\Users\sivar\PycharmProjects\OSRS-Bot-COLOR\src\model\alora\__init__.py", line 1, in <module>
    from .alora_bot import AloraBot
  File "C:\Users\sivar\PycharmProjects\OSRS-Bot-COLOR\src\model\alora\alora_bot.py", line 6, in <module>
    import utilities.bot_cv as bcv
  File "C:\Users\sivar\PycharmProjects\OSRS-Bot-COLOR\src\utilities\bot_cv.py", line 5, in <module>
    import cv2
  File "C:\Users\sivar\PycharmProjects\OSRS-Bot-COLOR\env\lib\site-packages\cv2\__init__.py", line 181, in <module>
    bootstrap()
  File "C:\Users\sivar\PycharmProjects\OSRS-Bot-COLOR\env\lib\site-packages\cv2\__init__.py", line 175, in bootstrap
    if __load_extra_py_code_for_module("cv2", submodule, DEBUG):
  File "C:\Users\sivar\PycharmProjects\OSRS-Bot-COLOR\env\lib\site-packages\cv2\__init__.py", line 28, in __load_extra_py_code_for_module
    py_module = importlib.import_module(module_name)
  File "C:\Users\sivar\Anaconda3\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "C:\Users\sivar\PycharmProjects\OSRS-Bot-COLOR\env\lib\site-packages\cv2\gapi\__init__.py", line 290, in <module>
    cv.gapi.wip.GStreamerPipeline = cv.gapi_wip_gst_GStreamerPipeline
AttributeError: partially initialized module 'cv2' has no attribute 'gapi_wip_gst_GStreamerPipeline' (most likely due to a circular import)

To Reproduce
Steps to reproduce the behavior:
run python ".\src\OSRS Bot Color.py"

object.addProperty("latest msg", msg);

Morg API has a "latest msg" property. We should add functions that use this to the Morg API utility.

https://github.com/MorgApps/morghttpclient/blob/master/src/main/java/com/morghttpclient/HttpServerPlugin.java

_{From SyncLinear.com | OSR-28}

Describe the bug
This error is raised when executing a osbc script. when creating an object of the CTkLabel class from the customtkinter package. The CTkLabel constructor is passed a text_font argument which is not supported. The check_kwargs_empty() function raises a ValueError exception with a message indicating that the text_font argument is not supported.

To Reproduce
Steps to reproduce the behavior:
run python .\src\OSBC.py

Expected behavior
error output:
Traceback (most recent call last):
File "C:\Users\mike\Desktop\OSRS-Bot-COLOR-main\src\OSBC.py", line 326, in
app = App() # Add the "test=True" argument to the App constructor call.
^^^^^
File "C:\Users\mike\Desktop\OSRS-Bot-COLOR-main\src\OSBC.py", line 38, in init
self.build_ui()
File "C:\Users\mike\Desktop\OSRS-Bot-COLOR-main\src\OSBC.py", line 72, in build_ui
self.label_1 = customtkinter.CTkLabel(master=self.frame_left, text="Scripts", text_font=("Roboto Medium", 14))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\mike\AppData\Local\Programs\Python\Python311\Lib\site-packages\customtkinter\windows\widgets\ctk_label.py", line 90, in init
check_kwargs_empty(kwargs, raise_error=True)
File "C:\Users\mike\AppData\Local\Programs\Python\Python311\Lib\site-packages\customtkinter\windows\widgets\utility\utility_functions.py", line 18, in check_kwargs_empty
raise ValueError(f"{list(kwargs_dict.keys())} are not supported arguments. Look at the documentation for supported arguments.")
ValueError: ['text_font'] are not supported arguments. Look at the documentation for supported arguments.

Feature
Add remote mouse movements/clicks so that the user's cursor is not occupied by OSBC. This feature will be supplied to the mouse utility as an option that can be toggled by users via a settings panel.

This issue is being created to jot down some information/options at our disposal.

_OSR-18

Problem
Many OSBC bots require users to mark specific objects in the game. For instance, with Agility bots, we can't rely on the Agility plugin to mark obstacles for us (for various reasons), so users must do it manually. This would be very tedious to do each time they load the client.

Solution
Users should be given the option to save the current session's RL settings by name. Then, when they go to launch the game client via OSBC, they can select the RuneLite settings file they'd like to launch from.

Problem
Currently, points of interest on screen (e.g., game view, inventory area, minimap area, etc.) are hardcoded coordinates - meaning bots rely on the game client to be fixed to the top-left of the user's screen. This is annoying because moving the game client for any reason would require restarting a bot.

Solution
The pre-defined points/rectangles of interest should be computed properties based on the game client window's position.
I haven't thought this through much so I'm just going to experiment until something works.

Hi, I used to be a member of the discord group. Recently have wanted to get back into scripting and I notice the discord is pretty private. Would I be able to get another invite? My discord name is McFaddles#3683, you can look back and see that I've posted. Thank you.

Problem
At the moment, the Mouse Utility offers simple mouse movements by allowing randomized tweens and normally distributed movement durations. While this is a viable solution temporarily, there is room for improvement.

Solution
Pyclick is a library for generating human-like mouse movements based on the Bézier curve. Implementing this into the Mouse Utility would greatly improve it.

The documentation for this library is lacking, so investigation into the source code and/or projects that use this library may be required. See the dependents list.

Should likely be implemented after #34
As discussed in #33

Problem
The current implementation of the RuneLite Settings replacer is not ideal. It requires multiple clicks and it has lasting side effects that must be corrected manually by the user (i.e., when the user wants to revert to non-botting settings).

Solution
Perhaps the best solution would be to have a button that lets the user launch their game client with arguments. This would launch the client while pointing to a different settings.properties file. This button can be in two potential spots: the HomeView for a game, or the BotView itself. If it exists in the HomeView, then there is one settings file for all bots for a particular game -- but placing this button in the BotView could allow developers to add a unique settings.properties file for each bot. Then, developers do not need to depend on some "master settings" file.

Alternatives I've Considered

• Adding a toggle switch to the HomeView that replaces the settings.properties file.

Concerns

• Some of the RuneLiteBot functions depend on certain plugins (checking HP values based on HP sidebars, opponent information, and certain tag colors). If we give developers free rein to make their own settings files, perhaps bots will not work as expected. I suppose developers will just need to be diligent in making sure their bot works with whatever settings configuration they assign to it.

Examples

OR

Problem:
Currently, bots rely on the developer putting frequent calls to status_check_passed() to listen for keyboard interrupts and status changes. This isn't ideal because it causes clutter in bot scripts, clutter in Bot and RuneLiteBot functions, and status checks can easily be forgotten by the developer or placed too sparsely, making keyboard interrupts feel sluggish. This feature was coded early on and not much thought was put behind it.

At the moment, the keyboard interrupts are captured on the UI, then the UI tells the bot to adjust its status, then the main bot loops listens for changes in that status and reacts. I feel like this is the wrong way of doing it.

Solution:
A system should be developed that allows for listening to keyboard interrupts and status changes on separate threads. Ideally, we'd want a setup that prevents the developer from needing to use a status-checking function at all. Maybe they can be daemon threads of the main thread property within Bot. I think the UI should maybe tell the bot what the user is requesting the status should be, then the bot can react, then notify the UI of the new status once the request has been met. So, kind of the opposite of what is implemented now.

This would also solve the issue of the UI status saying a status has been changed prematurely. For instance, when the user presses the stop button on the UI, it will say "STOPPED" before the thread has truly been terminated. This then allows the user to press play, and now we have duplicate threads running in parallel.

Challenges:

• How we would handle pausing a bot. As far as I know, it's not possible to pause a thread from another thread (let alone pause one at all).

This issue was brought up in Discord. A user mentioned that an LLMHF_INJECTED flag is put on hardware inputs in PyAutoGUI, and it may be a vulnerability for bot detection. More info here: https://stackoverflow.com/questions/45295482/pyautogui-dont-work-in-game-window

A possible workaround is to swap to PyDirectInput: https://pypi.org/project/PyDirectInput/.

Concerns: Some utilities in OSBC may require some functions of PyAutoGUI that don't exist in PyDirectInput. I'm thinking this might be the case in the Window utility, but I'm not entirely sure off the top of my head.

_{From SyncLinear.com | OSR-47}

Problem
Pressing Ctrl to start usually works fine. But sometimes, you have to press Ctrl x2 to stop. You won't have the issue if you hold Ctrl down long enough before releasing.

Root cause
To paraphrase from pynput's documentation, the keyboard listener callbacks are invoked directly by an OS thread. The Window.initialize() method takes about 300ms to finish on my system. During the 300ms, it blocks the OS thread, blocking any keyboard events.

So, it does not detect the Ctrl key being released. The member variable "self.pressed" does not get set to False, which means an additional Ctrl is needed to set it to False when trying to stop. Only then it will work as normal, and this is where the second Ctrl press will come in to stop the bot.

Solution
The simplest is to move the bot stop and play logic to __on_release(). Ideally, we could consider moving the window.initialize() to another thread...or the main bot one. I've tested the former (moving logic to on_release) and it works fine, albeit that any incoming key presses gets added to an OS queue. I would like your input first though. I am happy to do the PR for any solution you recommend. Great project btw!

_OSR-19

Problem
PyAutoGUI uses an older Windows API for simulating KBM inputs, and that API is becoming more and more deprecated as Windows updates. This does not necessarily mean PyAutoGUI will not work for OSRS bots, but migrating to a more modern library will reduce the likelihood of encountering undesirable KBM behaviours.

Solution
PyDirectInput is the first step of the solution, which utilizes DirectInput scan codes. An extended fork of this project exists, pydirectinput-rgx that implements the missing functions from PyDirectInput. This prevents us from requiring both PyAutoGUI and PyDirectInput.

As discussed in #33

Is your feature request related to a problem? Please describe.
In my using of this function, I've encountered the problem of it detecting a single RuneLiteObject as being multiple, i.e. it spits out a List with duplicates in it.

Describe the solution you'd like
deduplication.

Describe alternatives you've considered
In my local copy I've patched the function like so:
before creating the RuneLiteObject and added it to the List objs, I run a check comparing its euclidean distance (np.linalg.norm is very fast) between centers to every other RLO in objs currently. If it's less than 10 pixels, I chuck it instead of appending.

My setup is a bit different to yours, I just snagged this function, so this isn't a direct drag-and-drop patch, but here's the code I have in my local copy to explain the idea. This has totally eradicated my duplicate problem.

if len([o for o in objs if o.distanceToPoint(Point(center[0],center[1])) < 10]) == 0:
    objs.append(RuneLiteObject(x_min, x_max, y_min, y_max, width, height, center, axis, self.client))

To improve the human-like behavior of the scripts, Its helpful to have time differences in repetitive mouse actions. Adding a parameter for time jitters would be really helpful:

from:

def move_to(self, point: tuple, duration=0.3, variance=0):
        '''
        Moves mouse to a point on screen with a random movement pattern.
        Args:
            point: x, y tuple of the destination point
            duration: duration of the movement
            variance: maximum pixel variance in final x and y position
        '''
        # tweens = [pag.easeOutBounce, pag.easeInBounce, pag.easeInBack,
        #           pag.easeInCirc, pag.easeInCubic, pag.easeInElastic,
        #           pag.easeInExpo, pag.easeInOutBounce, pag.easeInOutQuad,
        #           pag.easeInOutQuart, pag.easeInQuart, pag.easeInQuint,
        #           pag.easeInOutBack, pag.easeOutSine]
        # tween = tweens[np.random.randint(0, len(tweens))]
        tween = pag.easeInOutSine
        x, y = point
        if variance != 0:
            x += np.random.randint(-variance, variance)
            y += np.random.randint(-variance, variance)
        pag.moveTo(x, y, duration=duration, tween=tween)

to

def move_to(self, point: tuple, duration=0.3, variance=0, time_variance=0):
        '''
        Moves mouse to a point on screen with a random movement pattern.
        Args:
            point: x, y tuple of the destination point
            duration: duration of the movement
            variance: maximum pixel variance in final x and y position
            time_variance: the variance absolute of duration. This number is always posotive (gaussian with mean 0)
        '''
        # tweens = [pag.easeOutBounce, pag.easeInBounce, pag.easeInBack,
        #           pag.easeInCirc, pag.easeInCubic, pag.easeInElastic,
        #           pag.easeInExpo, pag.easeInOutBounce, pag.easeInOutQuad,
        #           pag.easeInOutQuart, pag.easeInQuart, pag.easeInQuint,
        #           pag.easeInOutBack, pag.easeOutSine]
        # tween = tweens[np.random.randint(0, len(tweens))]
        tween = pag.easeInOutSine
        x, y = point
        if variance != 0:
            x += np.random.randint(-variance, variance)
            y += np.random.randint(-variance, variance)
        pag.moveTo(x, y, duration=duration + np.abs(rd.gaus(0, time_duration)), tween=tween)

Would fix this.

Also I understand that certain mousemovements can be annoying (I've had pyautogui auto quite because my mouse hit a corner during the course of a quadratic/curcular mouse movement)

but its really helpful to seem human to have variety, so perhaps setting a default to:

def move_to(self, point: tuple, duration=0.3, variance=0, time_variance=0, mouse_movement=None):
        '''
        Moves mouse to a point on screen with a random movement pattern.
        Args:
            point: x, y tuple of the destination point
            duration: duration of the movement
            variance: maximum pixel variance in final x and y position
            time_variance: the variance absolute of duration. This number is always posotive (gaussian with mean 0)
            mouse_movement: the mouse movement to use in pyautogui's moveTo. Default easeInOutSine

        '''
         if mouse_movement is None:
                 tween = pag.easeInOutSine
         else:
                  tweens = rd.choice([pag.easeOutBounce, pag.easeInBounce, pag.easeInBack,
                                                      pag.easeInCirc, pag.easeInCubic, pag.easeInElastic,
                                                      pag.easeInExpo, pag.easeInOutBounce, pag.easeInOutQuad,
                                                      pag.easeInOutQuart, pag.easeInQuart, pag.easeInQuint,
                                                      pag.easeInOutBack, pag.easeOutSine])
        
        x, y = point
        if variance != 0:
            x += np.random.randint(-variance, variance)
            y += np.random.randint(-variance, variance)
        pag.moveTo(x, y, duration=duration + np.abs(rd.gaus(0, time_duration)), tween=tween)

If this is something you would like changed please let me know and I will make a fork -> pull request!

Problem:
We have a new utility that lets you record a sequence of clicks to a CSV file along with screenshots for each (#25). At the moment, to make use of the results this utility produces, a developer needs to manually sift through Pandas data frames to extract click data and use it.

Solution:
A function should be made that accepts the results of the utility as input and builds a fully functional loop from it. This loop could use the supplied screenshots to do checks prior to clicking to make sure the player is clicking the right spot.

The function might look like:
def run_mouse_recording(folder: )

That way, a mouse recording can be used within the main_loop() of complex bots; so when a mouse recording aborts for some reason or another, it can be handled gracefully in the main_loop() so that the recording can be resumed again.

Other thoughts:

• Do we want to pass the path of the folder as the argument to that function, or the contents of the folder?
• Is class Bot the right place to put this function? Might need to think about that.
• Changes to the mouse recorder utility might need to be made.
• Maybe there's a way better way of doing this - I'm open to ideas.

Discussed in https://github.com/kelltom/OSRS-Bot-COLOR/discussions/22

Describe the bug
functions inside runelite_cv like self.get_nearest_tag(self.TAG_PURPLE) arent registering blocks on agility courses.
Am I using these functions wrong?
Heres is the full code/what ive tried:

# 1 (ab is the bot instance)

ab.mouse.move_to(ab.get_nearest_tag(ab.TAG_PURPLE))
ab.mouse.click()


# 2 

        map = self.rect_game_view
        map = (map.start.x, map.start.y, map.end.x-map.start.x, map.end.y-map.start.y)
        pag.screenshot(imageFilename='images/temp/temp_screenshot.png', region=map)
        path_tagged = isolate_colors('images/temp/temp_screenshot.png', [color], "get_all_tagged_in_rect")
        contours = get_contours(path_tagged)
        coords = rd.choice(contours[0])[0]
        print(contours[0])
        self.mouse.move_to((coords[0], coords[1]), .2, 0, .001)
        self.mouse.click()

screenshot for reference:

Describe the bug
This is a bug with the ocr.find_text function. Sometimes, it fails to pick up a certain letter depending on other letters included in the search.

For instance, here is an input image for testing with a function call like this: ocr.find_text('Deposit', self.win.control_panel, ocr.BOLD_12, clr.WHITE)

"Deposit":

"Deposit-1":

"Deposit-5" failure:

...

Any string in full below the top option is not searchable, except for Cancel. Additionally, if we exclude the letter p from the searches, it can find the strings. With some further investigation, it is confirmed that right-click menu text lines are too close together for find_text to work reliably.

This issue will be isolated to cases where the text is too close together. Unfortunately, this applies to chatbox text too. Perhaps the solution is to make helper functions that ignore the top row of pixels for each letter. Another potential solution would be to add a confidence param to the function. Lowering the confidence does seem to bypass this issue:

_OSR-15

Problem
Taking screenshots of the client and scanning them can only take us so far. It would be better if we had a way to get ground-truth information about the game state without doing this.

Solution
Luckily, there are official plugins that stream game state information to a local host server, such as Status Socket. Using this plugin requires us to write a module that starts a local host server and allows us to query the JSON data it receives.

Concerns
I've run into issues with multithreading in Tkinter before, even when UI code is restricted to the main thread. I'm guessing we just fire up the server on program start and let it run until it closes.

Starter Code
Here's the server base courtesy of BTWIUSEGENTOO#9187 in Discord.

from http.server import BaseHTTPRequestHandler, HTTPServer
import simplejson

class ServerHandler(BaseHTTPRequestHandler):
    data_bytes: bytes

    def _set_headers(self):
        self.send_response(200)
        self.send_header('Content-type', 'text/html')
        self.end_headers()

    def do_POST(self):
        global player_data
        self._set_headers()
        self.data_bytes = self.rfile.read(int(self.headers['Content-Length']))
        self.send_response(200)
        self.end_headers()
        player_data = simplejson.loads(self.data_bytes)

    def log_request(self, *args):
        return

def server(port=5000):
    httpd = HTTPServer(('127.0.0.1', port), ServerHandler)
    httpd.serve_forever()

And a query method:

def find_item(id_list):
    '''
    Finds the inventory positions of all the items in the given list.
    Args:
        id_list: A list of item ids.
    Returns:
        A list of inventory positions of the found items.
    '''
    if not id_list:
        return []
    item_list = []
    inventory = player_data['inventory']
    for id_item in id_list:
        item_list.extend(item['index'] for item in inventory if item['id'] == id_item)
    if not item_list:
        return []
    item_list.sort()
    return item_list

This function has been requested a few times. It's pretty useful for checking minimap colors.

We should also consider making this function return the location of the found color so it can be clicked.

_{From SyncLinear.com | OSR-27}

At the moment, Rectangle and RuneLiteBot have functions for getting the distance of an instance to its container Rectangle's center. The way this is implemented currently requires our `Rectangles and RuneLiteObjects to be dependant on other instances of these classes. The only real benefit of this is sorting based on the container (and, for lazy reasons, locating a RuneLiteObject relative to the monitor origin).

Here's how the sorting key function is implemented in Rectangle:

    # TODO: Consider changing to this to accept a Point to check against; `distance_from(point: Point)`
    def distance_from_center(self) -> Point:
        """
        Gets the distance between the object and it's Rectangle parent center.
        Useful for sorting lists of Rectangles.
        Returns:
            The distance from the point to the center of the object.
        """
        if self.reference_rect is None:
            raise ReferenceError("A Rectangle being sorted is missing a reference to the Rectangle it's contained in and therefore cannot be sorted.")
        center: Point = self.get_center()
        rect_center: Point = self.reference_rect.get_center()
        return math.dist([center.x, center.y], [rect_center.x, rect_center.y])

I think this sorting key function could be made general (e.g., accept a Point as an argument and get distance from it instead of a container). Implementing this is a little tricker for RuneLiteObjects though.

Since the runelite_cv.py >extract_objects function only considers an image, the containing Rectangle needs to be added as a reference to that shape after its creation. This allows us to know its position on screen relative to the origin of the monitor rather than relative to some png image we no longer have access to. That's where the whole idea of Rectangle references came from, and it's pretty lazy.

It's likely that extract_objects should just accept a Rectangle and Color(s) as input instead of an image, just like the OCR utility functions. Then, the RLObject's properties can be inherently relative to the monitor at creation-time, and a dynamic sorting key function can be used to get distance from any point.

TLDR:

• runelite_cv.py's extract_objects function should be re-designed to accept a raw Rectangle & Color set instead of an image, and should define the resulting RuneLiteObject with the Rectangle argument in mind.
• Rectangle reference properties can be removed from both Geometry classes.
• Sorting key functions in both classes should be re-designed to get distance from a given point rather than the container.

_{From SyncLinear.com | OSR-40}

I didn't even realize the OSRS Wiki had an API and I feel like an idiot.

The SpriteScaper utility currently scrapes the webpage with BS4, but it's likely better to just use the official API.

Some simple example usage is shown here: https://www.osrsbox.com/blog/2018/12/12/scraping-the-osrs-wiki-part1/

https://oldschool.runescape.wiki/api.php?action=opensearch&search=abyssal&format=json&limit=20

_{From SyncLinear.com | OSR-44}

Problem
PIL is simply not the fastest option for capturing the screen in real-time. Since OSBC bots rely on constant screen capture, we should opt for the fastest libraries.

Solution
python-mss is an ultra-fast alternative using ctypes. It can be the replacement.

Notes
PIL functions are used in many places throughout OSBC, so this will result in lots of refactoring. Additionally, we will need to ensure that the new solution is compatible with the way the Rectangle named tuple is implemented (which might not be an issue).

As discussed in #33

Hello, I just discovered this amazing project and am trying to tinker with it. I think I may have found a few bugs or am having issues.
1: I don't think RuneLite is being started with custom_settings.properties correctly because it doesn't seem the correct plugins or settings are being installed/turned on. I don't think this is my main issue because I've looked at the file and the documentation and have manually configured RuneLite to match what ya'll say to do. I do have to change the confidence=0.15 on the search_img_in_rect method in imagesearch.py or else I get:
Window.__locate_chat(): Failed to find chatbox.
Window.__locate_control_panel(): Failed to find control panel.
Window.__locate_game_view(): Failed to locate game view. Missing minimap, chat, or control panel.
2: I'm testing out the default Combat and Woodcutter scripts and getting the same error, screenshot pasted below (added some of my own print statements). It doesn't seem that player_data is getting populated. I have Status Socket plugin installed with endpoint set to http://localhost/5000 and the Morg HTTP Client plug installed, both are turned on. I added print statements to do_POST in RLSTATUS class in status_socket and it doesn't appear that this method is ever hit so player_data = JSON.loads(self.data_bytes) never gets populated and is just left as player_data = {}.

Thank you so much for taking the time to create this project and listen to my ramblings. Let me know if I can give any more relevant info. I would like to continue tinkering and possibly help contribute.