Comments (20)
would this also apply to the pirkey_m0?
from adafruit_circuitpython_bundle.
@jerryneedell Good point, yes. Since we're not selling it currently, I wasn't thinking about that.
from adafruit_circuitpython_bundle.
oh yeah thats a good idea
put /libextra first, .frozen second, then /lib ?
from adafruit_circuitpython_bundle.
@ladyada other way around: most libraries in lib
, then .frozen
, then libextra
which will be only the ones that are usually frozen, especially cpx and crickit.
from adafruit_circuitpython_bundle.
i meant the searchpath :)
from adafruit_circuitpython_bundle.
There are two issues being covered here, and I agree that both need to be addressed. I'm creating a second issue with the second topic so we can focus on each on accordingly.
The least invasive way to do this is to change sys.path
. It would all be handled behind the scenes at that point. It seems like if we're going to do it, we should put it as ['', '/', '.frozen', '/lib']
. That leaves us a way to test updated libraries without needing to do a build with them frozen in every time. In situations where people are dragging individual libraries, they're often putting them on CIRCUITPY
not in /lib
, so we're still going to have to address that as it happens.
Having multiple zip files for bundles is also a good option. We have 5 boards with frozen libraries, and would need to maintain the current two main bundle versions, which comes to 7 total (I think). We've already introduced people to the idea of downloading something specific to their board with CircuitPython, so this isn't a new concept. This would be a more invasive way to resolve this issue because we're now introducing 5 more options into the bundle download situation and will obviously run into the issue of people needing to update their bundle and people having the wrong bundle loaded.
I think moving some of the libraries to a different folder is the least useful way to do it as it can't reasonably address all the libraries frozen into the various builds without running into issues on boards that don't have libs frozen in.
Regardless of how we do this people will have to update something: either the firmware or the lib bundle. So the decision comes down to where to we want to make the change and what thing do we want people to have to update.
from adafruit_circuitpython_bundle.
What if we took a step back and tried to solve the original problem of memory use?
Perhaps the best thing is to use spare flash space to host libraries pulled from flash. Or, better yet, leave the raw code bits from the mpy file on disk until they are actually used. This will make better use of ram for all cases instead of the specific cases that frozen modules covers. It'll also leave the bundle as is.
from adafruit_circuitpython_bundle.
I agree with Kattni's thoughts that simply flipping .frozen
and lib
in the search path is the easiest solution. A long time ago we talked about putting .frozen
first, but were then worried about not being able to update frozen libraries without a firmware update. At that time the libraries were much less mature. Now they are in good shape. But we see support cases where people run out of RAM because they have duplicate copies of the the frozen libs in their lib
directory.
I think it's a nice aspirational idea to have an "mpy cache" in flash, which would get updated dynamically if a newer mpy was available in the filesystem on import. But that's a significant implementation task.
We can talk about this in the meetings today and later too.
from adafruit_circuitpython_bundle.
My concern with having .frozen first by default is that is then becomes more difficult to use an updated version of a library. I suppose we could just test by putting the test candidate at the root level instead of in /lib then it would get used before .frozen.
I am a little uncomfortable intentionally having two version of a library installed - one in .frozen and one in /lib. For my workflow, I just remove the redundant entries from /lib then the current search path is fine.
It may be simpler to hide this by putting .frozen first and that is will work a lot of users. It really depends on how much we want to educate the user to the internal workings.
from adafruit_circuitpython_bundle.
If a module exists as frozen then I think there has already been an implicit decision that it should be used first over whatever is in the lib folder. Further, the situation of having a frozen module but wanting to use a non-frozen version instead is kind of an advanced technique. How about this?
Put .frozen
first so this would import frozen version:
import foo
If you want to get around that, use a different import syntax to import version in lib folder:
import lib.foo as foo
from adafruit_circuitpython_bundle.
@caternuson makes a good point - as did @kattni - using a non-default version of the library is an advanced technique - so it makes sense for it to be more complex.
I agree that swapping /lib and .frozen is the best way to go.
That said, I still think we should consider distributing different .zip files for each board so we are not intentionally having two versions of the library present.
from adafruit_circuitpython_bundle.
Another idea - just remove always frozen modules from the bundle.
- CPX - it's frozen for the only board that uses it
- crickit - frozen in specific firmware
from adafruit_circuitpython_bundle.
Not sure about crickit - since we don't produce images for all the feathers that can be used with the feather crickit. For example feather_m4_express. Will it work with any of the basic feather M0's? I have not tried those. What about the feather ESP8266?
from adafruit_circuitpython_bundle.
crickit and seesaw are not always frozen, and if you're using ESP8266 or M4 yeah you'd install that by hand
from adafruit_circuitpython_bundle.
I don't like having special cases for crickit and cpx. They have a long term cost during development because they require special treatment forever more. For example, the networking PR requires extra effort because the new code fits in all express boards except those with frozen libraries.
There are a number of ideas for memory optimization that would help all of our users here. Reducing memory use would remove the need to have these libraries frozen in at all, simplify the download options for CircuitPython and simplify development and maintenance.
from adafruit_circuitpython_bundle.
@tannewt percentage-wise, how much space does dropping the line numbers in .mpy gain? I'm worried it will make support harder, since we won't be able to tell where library code is breaking when presented with backtraces.
from adafruit_circuitpython_bundle.
I understand the sentiment of not having special cases. But with only 20kB for the heap on the M0, having frozen modules makes an enormous amount of difference toward being able to write programs of significant size on the CPX and Crickit, versus only being able to write toy programs. The CPX library is about 8kB, the LIS3DH library is about 6kB, and the crickit library is about 6kB. These libraries have already been tuned to be smaller. Completely removing these large chunks of .mpy from RAM makes a huge difference. 1kB here and there will not solve the problem.
The M4 and nRF52840 solve the problem in the long run, but there's no large-RAM CPX yet, and we still need to support the large number of boards already sold. At some point I think we should stop adding features to the M0 builds, and save those features for the large-memory boards.
One way to think about the frozen modules is not that they are Python modules that deserve to be compilable and be in RAM, but that they are specialized builtin modules that happen to be coded in Python instead of C. We could have written the libraries above in C to put them in flash, but it's a much more arduous undertaking.
My own personal experience has been trying to write a relatively simple but "real" HID keyboard/mouse program that also supported a character LCD display on the M0. I did all kinds of pruning on the libraries, but without frozen modules, it was impossible.
from adafruit_circuitpython_bundle.
@dhalbert The behavior without line numbers is that the stack trace says line 1 of the file instead of the actual line. This would require us to reproduce the issue when using the py file but I don't think that prevents debugging.
I ran some approximate numbers yesterday by measuring CPX build sizes with libraries turned on and off and with line number info or not. The number is the number of bytes free in flash. Its a decent approximation of ram size I believe as well.
6928 w/o lines
5484 w/ lines
29372 w/o frozen libs
27440 w/bus device
24592 w/circuitplayground
20100 w/HID
24632 w/lis3dh
26456 w/neopixel
28456 w/thermistor
So, the frozen libs are 23888 bytes and line number info is 1444 bytes (6%). That won't solve our problems but it would help.
There are a couple of other issues with frozen modules:
- Upgrading them requires a new version of CircuitPython. This is currently coming up in my discussions with the KMK keyboard folks (KMKfw/kmk_firmware#52). They want to freeze their code in but then they must rerelease CircuitPython for every version of their code. Instead, it'd be awesome for them to use stock CircuitPython but then have an easy install/upgrade on top of it.
- Freezing libs is too easy of an answer when running out of memory and causes design issues to be ignored. For example, the Seesaw library is 10696 bytes (when frozen in)! I doubt it needs to be. This impacts all uses of the Seesaw library but freezing it for Crickit only solves it for those builds.
Ultimately, I think the problem is that we don't have good insight into why we run out of memory. Making it easier for anyone to get a diagram of the heap would help this. We could even automate heap charts for every library.
So, for example, for every bundle we could generate heap charts of all the libraries after import and one for every example. That would make the impact of code changes on memory much more apparent.
Improving library sizes will help all embedded use of them. Changing memory behavior in CircuitPython will as well. Freezing modules doesn't.
from adafruit_circuitpython_bundle.
This is coming up constantly now. The most reasonable solution is to switch the path to ['', '/', '.frozen', '/lib']
to resolve this. It's not reasonable right now to fix all of the memory issues we're seeing in the M0 builds. There is no quick answer to fixing the underlying issue, and we need something in place now.
I discussed it again with @caternuson because he's a perfect example of the support people who are dealing with this. If we make this change, the thing that will have to happen is for users to upgrade to the latest version of CircuitPython, a step which is much simpler to identify and explain than explaining frozen modules and the reason for deleting the library from the /lib folder.
I will be filing an issue with the same info on the CircuitPython repo as this is a change that needs to be made to CircuitPython, and not to the bundle.
from adafruit_circuitpython_bundle.
Superseded by adafruit/circuitpython#1310
from adafruit_circuitpython_bundle.
Related Issues (20)
- Many libraries with bad circup instructions HOT 5
- FR: Add type hints to circuit python libraries HOT 1
- Include import names in the library list HOT 1
- "Drivers" vs "Helpers" is not so helpful as a top-level distinction HOT 1
- Note that adafruit_bus_device and adafruit_pixelbuf are available as native
- AttributeError: module 'pulseio' has no attribute 'PulseOut' HOT 1
- Add NAU7802
- Bundle being rebuilt with no library changes HOT 4
- LSM6DS libraries were combined drivers.rst needs updated
- key stuck with macropad HOT 4
- adafruit_neopxl8.mpy doesn't have NeoPixel attribute
- request - port TM1640 driver from micropython
- Source code release zip is mostly empty directories. HOT 6
- mcp2515 error
- Adafruit_CircuitPython_Bundle-main HOT 1
- ValueError: incompatible .mpy file
- TypeError: object of type 'Pin' has no len() HOT 1
- Libraries: docs/conf.py gives version as "1.0"
- docs/drivers.rst does not include import name
- Request Inclusion of .MPY Files in Github Repository HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from adafruit_circuitpython_bundle.