Current thinking is best names are boot-ARMv5.c and boot-ARMv7.c, this seems to be truthful and less fiddly than boot-d678X.c which needs to keep changing name to be honest.

bmp.c does

BMP_LOCK( bmp_fill( 0x0, BMP_W_MINUS, BMP_H_MINUS, BMP_TOTAL_WIDTH, BMP_TOTAL_HEIGHT ); )

Color with index 0 is RGBA #00000000. Current redraw code simply ignores it, which means we don't clear RGBA buffer. This ends up not erasing background in some circumstances, like Arkanoid.

IMO since D45 explicitly erased whole Canon buffer that we used, on FEATURE_VRAM_RGBA we should erase both indexed and RGBA buffers.

Found with file_man.mo: Digic 7 and 8 have a different fio_file struct, with additional fields. Modules are universal, compiled without per-cam ifdefs and thus wrong structure definition is used, ending with file manager wrong behavior.

Hardcoding struct to Digic78 variant made it usable on Digic78 models, but obviously broken for older ones.

This apply to any structure or function interface ifdef that may exists in code. We usually wrap functions into old interfaces, but that's impossible for structures.

Digic6 is ARMv7-R, which has hw division, most other Arm don't. Gcc is aware and emits idiv and sdiv instructions. On other Arm, there's some external symbols that code calls into, including e.g. __aeabi_idiv. Modules expect these and so fail dep checking on -R.

This is a provisional fix: e345717

This forces the linker to include code and symbols for the expected exports into the magiclantern binary. Magiclantern itself on D6 will still have idiv and sdiv instructions, so the only inefficiency is a small code size increase there, and the fact that modules when loaded on D6 will use software division even though hardware division is available (no worse than other platforms).

We could do some trickery so that these symbols, on ARMv7-R builds, resolved to the hardware operations. You'd still get the call overhead per division but the code would be small and presumably faster.

We would need to be sure that the module builds couldn't inline the now much smaller code, since this would mean if the first build for a module was from -R, the next non-R platform zip build would include broken code. This may already be true, I don't know.

This is a single digic cam, not dual. Ensure code is correct after it was mistakenly listed as dual.

Menu entries that immediately trigger crashes:

• Movie -> Movie tweaks
  • fixed by e552d2e
• Help -> Key shortcuts
  • fixed by 1067473
• Help -> Complete user guide
  • fixed by e552d2e
• Help -> About Magic Lantern
  • fixed by 1067473

On D6 and up additional_version doesn't work properly. We point this at \n in char buffer that contains firmware version in RAM. However Canon code leaves there 1.0.0 on boot and updates that slightly later in boot process, I'm not sure where.

Address is correct, however our code needs to wait for the value to be updated, or we need to find a better way to update the string.

Current code assumes stubs / consts can provide a fixed location for Ximr structs. This is true on some cams, but not others. E.g. 200D uses fixed address, R uses a pointer at fixed address to refer to address returned by an allocator.

For the latter cams we are currently dumping the address of the Ximr struct from running cams, which is not ideal - the allocators appear deterministic but this means ROM analysis is not enough to get display working. We should change the code to work with the fixed pointers.

In 05a99a7 I blindly believe font file to be valid if just header magic matches.

bfnt_load_from_card should be updated to:

• verify if loaded size > header size
• parse header
• count if maximum offset fits in loaded file

As already mentioned in #26 there is a sporadic glitch on the percentage row which leads into displaying random characters sometimes (see https://www.youtube.com/watch?v=wMBgHcIxlRs). It's probably some issue of bmp_printf.

So far we can only see this on RP.

Newer cams don't have an embedded bitmap format font. RBF can be used in most places, but there are a few where this doesn't work. Mostly for very early debug messages, but also for some icons and menus.

Kitor implemented the ability to load a compatible bfnt from card and the font format is not complicated - converting an existing bitmap font with a compatible license is the obvious choice (and making replacement icons for those cases, similar but distinct from Canon icons).

See "bfnt read fail" in rbf_font.c

I forked a possible good font to use: https://github.com/reticulatedpines/spleen/
BDF fonts are text, and this one is BSD licensed, so we could put into our repo directly. We would need to credit the BSD code, I'm not sure where that would go (about screen? Not much space though).
I made a bdf2fnt converter which could run at build, so we never need to keep large binaries in our repo.

Compilation guide largely inspired by the github action : https://github.com/reticulatedpines/magiclantern_simplified/pull/63/files
so much old info from google, this guide tries to fix that. Thanks for this repo.

sudo apt-get install python2 libc6:i386 libncurses5:i386 libstdc++6:i386 autoconf python2.7-minimal wget gcc build-essential mercurial libtool git libglib2.0-dev libpixman-1-dev zlib1g-dev libgtk2.0-dev xz-utils mtools netcat-openbsd lua5.4

wget https://launchpad.net/gcc-arm-embedded/5.0/5-2016-q3-update/+download/gcc-arm-none-eabi-5_4-2016q3-20160926-linux.tar.bz2
tar -xvf gcc-arm-none-eabi-5_4-2016q3-20160926-linux.tar.bz2
mv gcc-arm-none-eabi-5_4-2016q3 ~/

git clone https://github.com/reticulatedpines/magiclantern_simplified.git
cd magiclantern_simplified
make

make should output on first line : Using /home/myuser/gcc-arm-none-eabi-5_4-2016q3-old/bin/arm-none-eabi-gcc (found in home directory).

git_summary.py errors like this if someone downloads zip of the repo and tries to build it:
fatal: not a git repository (or any parent up to mount point /mnt)

We should either not fail, and make a build with a "not git managed" style message for the user, or, we should print a nicer message explaining why it failed and possible fixes.

See the related #17

On 7D2, during early boot, I was getting hangs from cam. Blink debugging located this as after the call to reloc_entry() in copy_and_restart(). Swapping the following stubs swaps whether it hangs, 80000f78 works okay:

37 //ARM32_FN(0xfe6510d0, create_init_task) // unthunked Thumb at 80000f78 38 THUMB_FN(0x80000f78, create_init_task)

Bugs this early in boot process are somewhat concerning. One candidate from the cause is patch_thumb_instruction(). Possibly bad implementation for encoding some addresses? Maybe something to do with allowable range of branch?

Because 7D2 doesn't emulate this far, I can't check how it's being encoded.

The parent repo stamped builds with hg commit info. I removed this to migrate to git. It would be nice to restore this functionality and extend to git, so the build determines which is managing the source and uses appropriate tools. If neither, don't fail.

See:
3163d41
4a29a34

contrib/ptpcam gives alot of warnings when compiling and tends to crash when exiting with ctrl+c while it is running. We may refactor it a bit to make it more stable.

See get_photo_cmos_iso_start_550d in dual_iso.c. Other cams also exhibit this problem due to dual_iso hard-coding a heap address that is not in fact fixed. 650d and 700d have had a few reports.

Use qemu + gdb to determine if the same approach as for 550d can be used, and if it can be generalised.

We use "naked" on some functions, e.g. backtrace_getstr() (here, presumably to avoid disturbing register contents before doing the backtrace), but, GCC states: "While using extended asm or a mixture of basic asm and C code may appear to work, they cannot be depended upon to work reliably and are not supported"
https://gcc.gnu.org/onlinedocs/gcc/ARM-Function-Attributes.html

We use both extended asm and C in functions marked naked, which is not supported.

I couldn't find a way to get GCC to warn about this behaviour, so reliably detecting it may be impossible. We don't use it very often so at a minimum we should audit and fix all naked functions.

CONFIG_MEM_2GB cams don't have enough virtual address space to use the old cache/uncache scheme.

Is the workaround in fio-ml.c appropriate? Is it performant? Is there a better way of mapping between these types of address?

Both mlv_lite and silent modules list lossless.o as a dependency. mlv_lite/Makefile has it like this:

MODULE_OBJS=mlv_lite.o ../mlv_rec/mlv.o ../silent/lossless.o

This means if lossless.o is first built during silent module, and mlv_lite happens to build later, mlv_lite will overwrite lossless.o, making the timestamp later than silent.mo. This leads to successive makes in the modules dir building silent module twice, which is at least wasteful. Maybe this causes real problems, too?

mlv_rec/mlv.o should suffer from the same problem.

It's not clear to me exactly how MODULE_OBJS leads to the build occuring at all, possibly implicit make rules, .o from .c of same name?

I'd guess the fix is to correctly list the shared dependency. With that done, parallel builds should know lossless.o must be built before either, and it should only get built once.

If you power on cam, and instead of Delete to enter ML, you long-press Q, the Q spinner will display, fill up, and then remain, with the cam in a strange state.

I don't know why long-Q even does anything on this cam, probably we can disable it. We use Av to back out of menus, and Q/Set to enter them.

If Global Draw is enabled, starting video recording shows red circle in upper right. This then disappears within a few seconds - do we perhaps overwrite it? No such problem with GD disabled. Confirmed on 200D and 750D.

MMU tables from R5:

00001000-00001FFF -> 00000000-00000FFF (-1000) O:NCACH I:WB,WA  P:RW   
00002000-3FFFFFFF -> 00002000-3FFFFFFF (   +0) O:NCACH I:WB,WA  P:RW   
40000000-BEFFFFFF -> 40000000-BEFFFFFF (   +0) O:NCACH I:NCACH  P:RW   
BF000000-DEFFFFFF -> BF000000-DEFFFFFF (   +0) Device           P:RW XN
DF000000-DFFFFFFF -> DF000000-DFFFFFFF (   +0) O:NCACH I:WB,WA  P:RW   
E0000000-E7FFFFFF -> E0000000-E7FFFFFF (   +0) O:WB,WA I:WB,WA  P:R    
E8000000-EFFFFFFF -> E8000000-EFFFFFFF (   +0) Strongly-ordered P:RW XN
F0000000-F7FFFFFF -> F0000000-F7FFFFFF (   +0) O:WB,WA I:WB,WA  P:R    
F8000000-FFFFFFFF -> F8000000-FFFFFFFF (   +0) Strongly-ordered P:R  XN

The important bit is BF000000-DEFFFFFF -> BF000000-DEFFFFFF ( +0) Device P:RW XN
On D78 models it was RAM up to 0xD0000000.

Implications currently unknown, except maybe QEMU assumptions.

No current devs really understand the build system. I'm not a make expert, but it seems overly complicated to me for the small number of use cases in active use.

The "minimal subdir" builds have been the source of multiple difficult bugs, and getting platform builds to clean them is hard, due to the bizarre way they try to build; it's easy to hit infinite recursion.

Build objects should go in a subdir of the source being built; this limits the chance of racey edits / removes of files and makes cleanup trivial.

Unused build rules should be removed (the stub auto-formatting stuff, the install targets, etc).

Try to reduce the amount of includes, complex functions, general magic.

crop_rec_4k_mlv_snd turns out to be another important branch that we want code from. In the normal pattern for ML development, this branch contains several different features, most unrelated to the branch name.

One such feature is a thread safety effort, utilising clang static analysis. See refs to REQUIRES, mutex.h and a new makefile target "tasks".

I think this works by invoking the build target, which triggers some kind of static analysis. I don't know if it's functional. I don't know if it's comprehensive. I don't know how to use or maintain it. Static analysis for thread safety is a cool concept, but I have no idea if this code is worth the maintainence cost.

Should we back out this feature?

Hi all,
Thank you all very much for your work.
I just tested canon basic scripting and it works on M6 II (digic 8). So I dumped the latest rom with firmware v1.1.1.
I understand that you guys are not working on this camera; but just in case this might be helpful.
Let me know if you want these files. Also let me know if there is anything I can help.
Thanks for your time!

General issues (Digic 6)

• D6.. winsys_sem (Semaphore_Winsys_v aka vsync semaphore) DNE on 750, but it exists on 5D4 (and D.78X)
  • Usage in D6.. removed in bae1c85
  • Without semaphore, screen tearing is possible while redrawing ML GUI.
  • TODO: Investigate when it was introduced by Canon, adapt code accordingly.

General issues (Digic 6 and up):

• D678 Even when no Canon overlays are visible, Canon will draw some on actions (eg. shutter speed change)
  • Canon overlays will flicker as we actively overdraw whole screen while zebra_should_run() (94c0e03)
  • To be investigated: possibility to kill Canon GUI drawing in this mode
    • Guess: this was achieved by bmp_on() / bmp_off() on D45
    • In FEATURE_COMPOSITOR_XCM we could just mute Canon layer(s), but there should be more generic solution.
• D678: HTP can have multiple levels now. Adapt to same style display as ALO.

General issues (Digic 8):

• D..8 PROP_LENS_DYNAMIC_DATA / lens struct size issues
  • RAW values (and field sizes) for ISO, AV, TV had changed.
• D..8 CONFIG_COMPOSITOR_DEDICATED_LAYER implementation for EOS RP and up (code to setup our own layer)
  • R code is specific to R; RP, 250D and even R6 (Digic X) seems to have unified code
  • Different structures are used to setup GUI layers
  • Can be probably achieved using just XCM function calls, without touching RENDERER / VMIX structs.
  • Refer to Chapter 4 for clues: Compositors, layers, contexts in RGB and YUV - How Digic 7(6?)+ draw GUI
• D..8 Most dial modes are displayed properly except Fv + SCN + Movie Mode are displayed as ?

Partially fixed:

• D..8 338dd8e Detection if overlays are clear to draw is broken:
  • TODO: check possibility to improve delay by handling props from 958c07f
• D678 Many (not yet enabled) code paths search for dialog struct in a broken way
  • Addressed for enabled cases by 338caa1
  • New implementation checks if given handler is not null, instead of checking gui_task_list topmost window
  • Require expansion with each new case enabled
  • TODO: check if it can be backported to D45
  • There's a brief moment on app transitions (eg LV->Main menu) where old code crashes, probably due to gui_task_list.current being null (see below)

    struct gui_task * current = gui_task_list.current;
    struct dialog * dialog = current->priv;

Fixed issues:

• 750D 618e5c2 GUI modes are wrong
• D67. 939a14c PROP_LV_LENS: PROP_LV_LENS_D67 added, new method of data fetching needed
• D..8 fa47a53 PROP_LENS_STATIC_DATA has different lengths on R/RP/250D/M50.
• D..8 103f33d With CONFIG_COMPOSITOR_DEDICATED_LAYER our LV overlays wont disappear unless ML menu open/close.
• D..8 dc65526 PROP_PIC_QUALITY2 decoding data fails on R.
• D..8 aa8eabe dialog_struct needs update for other cameras than R.
• D..8 aa3f67d On start camera will not display any Canon GUI (black screen, only ML GUI is visible).
• D..8 aa3f67d With CONFIG_COMPOSITOR_DEDICATED_LAYER screen sometimes is left not cleaned.
• D..8 aa3f67d Rewrite R compositor support, leave space for RP+ and Ximr implementations.
• D678 338caa1 After exit ML menu in LV mode, LV becomes black until Canon menu is open and closed.
• D678 338caa1 liveview_display_idle() dialog handlers based LV detection is fundamentally broken.

R5 has a total of 5(?)GB of RAM. There's no ARM PAE enabled. According to MPU tables, there's almost DIGIC8 mapping (see #80) so 1 gig of cacheable RAM mirrored to 1G uncacheable + almost another 1G of uncacheable.

FIO function gained a few new entries: ReadIBus and WriteIBus. There's some translation going on, as BufferAddr is CPUAddr[%#x:%#x]

Digic X is missing "old" SRM / RscMgr at all. A few stubs that exist here and there are just dummy functions that throw an error or assert.

debug_assert(0,"Resource::RscMgrOldApiStub.c",0x2b);

System is substituted by RscMgr2. No smemshowfix command - replaced by res_memshow <int>.

Example output:

res_memshow 4
     4022:  26713.744 [RSCC] ===== Map 0
     4023:  26713.749 [RSCC] ----- Allocated -----
     4024:  26713.753 [RSCC] USER_RENDERING_WORK
     4025:  26713.756 [RSCC] USER_RENDERING_IMAGE_WORK
     4026:  26713.758 [RSCC] USER_BMPVRAM
     4027:  26713.761 [RSCC] USER_AE_WORK
     4028:  26713.763 [RSCC] USER_AE_ENGINE_WORK
     4029:  26713.765 [RSCC] USER_DAF_RAW
     4030:  26713.767 [RSCC] USER_IMGVRAM_1
     4031:  26713.769 [RSCC] USER_IMGVRAM_2
     4032:  26713.771 [RSCC] USER_IMGVRAM_3
     4033:  26713.773 [RSCC] USER_IMGVRAM_4
     4034:  26713.776 [RSCC] USER_STILL_LV_WORK_1
     4035:  26713.778 [RSCC] USER_STILL_LV_WORK_2
     4036:  26713.780 [RSCC] USER_STILL_LV_WORK_3
     4037:  26713.784 [RSCC] USER_INDEXVRAM
     4038:  26713.786 [RSCC] USER_META_STORE_WORK
     4039:  26713.788 [RSCC] USER_DEFECT_PIX_BUF
     4040:  26713.791 [RSCC] USER_TVAF_WORK
     4041:  26713.793 [RSCC] USER_IBP_DATA
     4042:  26713.795 [RSCC] USER_AF_PINTDATA
     4043:  26713.799 [RSCC] USER_HDMI_RAW_META_STORE_WORK
     4044:  26713.801 [RSCC] USER_MEM3_CH2_B
     4045:  26713.805 [RSCC] ------ Waiting ------
     4046:  26713.815 [RSCC] =====

===

Some functions report addresses in form of more-than-32bit values:

K421[1]>VramRead
     3909:   7226.072 [DISP] PNL Addr:0x300000000 W:1024 H:682

channel = Peripheral::MemifCommon::CheckIbusAddress((layer->ibus).lo,(layer->ibus).hi,"Ximr/XimrApiSub.c",480);
Peripheral::MemifArbiter::MemifCheckPriority(channel,1);
buf = Peripheral::MemifCommon::CheckIbusAddress((layer->ibus).lo,(layer->ibus).hi,"Ximr/XimrApiSub.c",480);
Peripheral::MemifSelfRefresh::FUN_e08c7dfc(buf);

Number above bit 31 seems to be related to "ibus address", as:

      if (((size != 0x80000000) && (size != 0x20000000)) && (size != 0x40000000)) {
        bl_printf("Unexpected Size : %08x\n");
        return 1;
      }

Interesting function name: SystemIF::MapCpuBuffer.c MapCpuBufferToIbusBuffer(), UnmapCpuBuffer() seems to suggest there's some memory banking happening?

dibus evshell command may give a clue how to at least read other ibus channels(?):

K421[1]>dibus
0x100000000:     04 f0 1f e5 00 00 00 e0  - 00 00 00 00 01 00 00 00 
0x100000010:     c6 01 00 00 00 60 fc df  - 28 76 19 00 88 be 2a 00 
0x100000020:     7c ab 2a 00 7c ea 0f 00  - 28 ab 2a 00 00 00 00 00 
0x100000030:     fe 7f ff df fe ff ff ff  - fe ff df ff fb ff ff 7f 
0x100000040:     bd db df ff f8 5f fe ff  - ea ff ff ff ef ff ef bd 
0x100000050:     7f ff df ff df 7f ff fe  - fb fd fe fc df fe d7 ff 
0x100000060:     ff fe ff 7f fb ff fd 7f  - fe ff ff fc ff fe cf fd 
0x100000070:     ff ff f5 fd ff 7f ff ff  - fd d2 ff ff ff ff ff ff 
0x100000080:     ff 7f ff ba df fd fd ff  - df bf fd bb e7 3b ff ff 
0x100000090:     ff fd fe ff ef ff ff ff  - f7 ff ff df ff ff ef fb 
0x1000000a0:     ff ef d5 ff fe ff fe f7  - ff 2f df ff ff 7f bf ff 
0x1000000b0:     ff ff fd ff fb ef ff ff  - ff 7f f7 7f ff ff fb ff 
0x1000000c0:     ff 7a f7 3f af 7f df ff  - ff fe df ff cf df ff f9 
0x1000000d0:     ff d7 ff ff ff df ee ff  - ff ff ef f9 ff ef ff ff 
0x1000000e0:     f7 ff de f4 bf fb dd fb  - df fe 7f fd fa fd db dd 
0x1000000f0:     df fb ff ff f5 ef ff df  - ff fb ff ce ff ee ff e5 
 dibus returned 256(0x100)
K421[1]>dibus 2
0x200000000:     06 01 00 00 00 22 01 60  - 01 ff ff a0 00 00 00 00 
0x200000010:     00 00 00 00 00 00 00 00  - 00 00 00 00 00 00 00 00 
0x200000020:     00 00 00 00 00 00 00 00  - 00 00 00 00 00 00 00 00 
0x200000030:     00 00 00 00 00 00 00 00  - 00 00 00 00 00 00 00 00 
0x200000040:     00 00 00 00 00 00 00 00  - 00 00 00 00 00 00 00 00 
0x200000050:     00 00 00 00 00 00 00 00  - 00 00 00 00 00 00 00 00 
0x200000060:     00 00 00 00 00 00 00 00  - 00 00 00 00 00 00 00 00 
0x200000070:     00 00 00 00 00 00 00 00  - 00 00 00 00 00 00 00 00 
0x200000080:     00 00 00 00 00 00 00 00  - 00 00 00 00 00 00 00 00 
0x200000090:     00 00 00 00 00 00 00 00  - 00 00 00 00 00 00 00 00 
0x2000000a0:     00 00 00 00 00 00 00 00  - 00 00 00 00 00 00 00 00 
0x2000000b0:     00 00 00 00 00 00 00 00  - 00 00 00 00 00 00 00 00 
0x2000000c0:     00 00 00 00 00 00 00 00  - 00 00 00 00 00 00 00 00 
0x2000000d0:     00 00 00 00 00 00 00 00  - 00 00 00 00 00 00 00 00 
0x2000000e0:     00 00 00 00 00 00 00 00  - 00 00 00 00 00 00 00 00 
0x2000000f0:     00 00 00 00 00 00 00 00  - 00 00 00 00 00 00 00 00 
 dibus returned 256(0x100)
K421[1]>dibus 3
0x300000000:     fd 25 c0 df 04 fa 51 50  - 5f 03 fc 29 c0 df 02 03 
0x300000010:     56 40 9f 04 02 4e 40 5f  - 04 ff 29 a0 df 01 00 1e 
0x300000020:     e0 9f 04 fa 45 90 1f 03  - fd 25 d0 df 01 0b 2e 10 
0x300000030:     20 02 14 1e f0 9f 02 1a  - 3e 50 e0 04 08 56 e0 1f 
0x300000040:     04 ff 35 70 9f 04 ff 35  - a0 1f 02 00 16 00 a0 01 
0x300000050:     fb 41 90 1f 04 fc 31 80  - 5f 02 00 1a e0 5f 03 fc 
0x300000060:     41 e0 20 05 fb 45 c0 e1  - 04 fd 59 80 62 04 ff 21 
0x300000070:     a0 a0 02 fe 35 70 5f 03  - ff 21 a0 1f 02 ff 29 c0 
0x300000080:     1f 02 00 46 d0 5f 02 01  - 1a 10 e0 03 03 4e e0 1f 
0x300000090:     04 06 36 d0 5f 02 02 2a  - d0 9f 02 fd 1d 00 e0 01 
0x3000000a0:     ff 19 20 a0 01 05 1a e0  - 1f 01 03 12 e0 df 01 02 
0x3000000b0:     22 20 60 02 0d 22 10 a0  - 01 06 1a 10 a0 02 fe 31 
0x3000000c0:     c0 df 02 01 2a a0 9f 01  - ff 2d a0 df 04 fa 4d 40 
0x3000000d0:     5f 03 04 22 e0 df 01 03  - 1e 50 e0 01 00 1e 40 60 
0x3000000e0:     04 01 36 10 60 03 fc 51  - 30 df 03 fc 35 b0 df 03 
0x3000000f0:     00 56 00 9f 05 fe 51 20  - 1f 03 fb 29 d0 df 01 fc

In fact, while looking for existing XCM layer data it was on "2nd" IBus:

K421[1]>dryshDry[DivaPUX]> Dry[DivaPUX]> xd 0x22788900
           0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
22788900: 00 00 00 ff 00 00 00 ff 00 00 00 ff 00 00 00 ff  ................
22788910: 00 00 00 ff 00 00 00 ff 00 00 00 ff 00 00 00 ff  ................
22788920: 00 00 00 ff 00 00 00 ff 00 00 00 ff 00 00 00 ff  ................
22788930: 00 00 00 ff 00 00 00 ff 00 00 00 ff 00 00 00 ff  ................
22788940: 00 00 00 ff 00 00 00 ff 00 00 00 ff 00 00 00 ff  ................
22788950: 00 00 00 ff 00 00 00 ff 00 00 00 ff 00 00 00 ff  ................
22788960: 00 00 00 ff 00 00 00 ff 00 00 00 ff 00 00 00 ff  ................
22788970: 00 00 00 ff 00 00 00 ff 00 00 00 ff 00 00 00 ff  ................
22788980: 00 00 00 ff 00 00 00 ff 00 00 00 ff 00 00 00 ff  ................
22788990: 00 00 00 ff 00 00 00 ff 00 00 00 ff 00 00 00 ff  ................
227889a0: 00 00 00 ff 00 00 00 ff 00 00 00 ff 00 00 00 ff  ................
227889b0: 00 00 00 ff 00 00 00 ff 00 00 00 ff 00 00 00 ff  ................
227889c0: 00 00 00 ff 00 00 00 ff 00 00 00 ff 00 00 00 ff  ................
227889d0: 00 00 00 ff 00 00 00 ff 00 00 00 ff 00 00 00 ff  ................
227889e0: 00 00 00 ff 00 00 00 ff 00 00 00 ff 00 00 00 ff  ................
227889f0: 00 00 00 ff 00 00 00 ff 00 00 00 ff 00 00 00 ff  ................

K421[1]>dibus 2 0x22788900 
0x222788900:     00 00 00 ff 00 00 00 ff  - 00 00 00 ff 00 00 00 ff 
0x222788910:     00 00 00 ff 00 00 00 ff  - 00 00 00 ff 00 00 00 ff 
0x222788920:     00 00 00 ff 00 00 00 ff  - 00 00 00 ff 00 00 00 ff 
0x222788930:     00 00 00 ff 00 00 00 ff  - 00 00 00 ff 00 00 00 ff 
0x222788940:     00 00 00 ff 00 00 00 ff  - 00 00 00 ff 00 00 00 ff 
0x222788950:     00 00 00 ff 00 00 00 ff  - 00 00 00 ff 00 00 00 ff 
0x222788960:     00 00 00 ff 00 00 00 ff  - 00 00 00 ff 00 00 00 ff 
0x222788970:     00 00 00 ff 00 00 00 ff  - 00 00 00 ff 00 00 00 ff 
0x222788980:     00 00 00 ff 00 00 00 ff  - 00 00 00 ff 00 00 00 ff 
0x222788990:     00 00 00 ff 00 00 00 ff  - 00 00 00 ff 00 00 00 ff 
0x2227889a0:     00 00 00 ff 00 00 00 ff  - 00 00 00 ff 00 00 00 ff 
0x2227889b0:     00 00 00 ff 00 00 00 ff  - 00 00 00 ff 00 00 00 ff 
0x2227889c0:     00 00 00 ff 00 00 00 ff  - 00 00 00 ff 00 00 00 ff 
0x2227889d0:     00 00 00 ff 00 00 00 ff  - 00 00 00 ff 00 00 00 ff 
0x2227889e0:     00 00 00 ff 00 00 00 ff  - 00 00 00 ff 00 00 00 ff 
0x2227889f0:     00 00 00 ff 00 00 00 ff  - 00 00 00 ff 00 00 00 ff 

On older models there's a family of MEMIF functions. On R5 those are completely different when compared to any DIGIC 8 model + reference a lot of IBus addresses. Memif/MemifDynamicWindow.c, Peripheral::MemifDynamicWindow.c

When attempting to allocate own buffer for XCM, Assert went from MemifArbiter:

     4016:    962.031 Canon layer: 0x22ff8900
     4017:    962.036 Found 2 layers
     4018:    970.731 Our layer: 0x01ef7700
     4178:   1026.219 [DEVICE] ERROR MemifCheckPriority Invalid Priority!!
     4179:   1026.225 [DEVICE] ERROR pid(1) is not registered in DRAM-Ch1
     4180:   1026.243 [STARTUP] ERROR ASSERT : Peripheral::MemifArbiter.c
     4181:   1026.248 [STARTUP] ASSERT : Task = CtrlSrv
     4182:   1026.250 [STARTUP] ASSERT : Core 0
     4183:   1026.252 [STARTUP] ASSERT : Line 133
     4184:   1026.258 [STARTUP] < StackDump >

Both mlv_lite and mlv_rec perform very similar functions. Functionality should be merged while keeping acceptable performance. If necessary, optional expensive functionality could be behind a toggle in a menu (metadata per frame?).

First need to determine required functionality (mlv_rec does more, except, mlv_lite does lossless compression... anything else it's better at?). Card spanning is important. Both should do sound, need to confirm this.

Some discussion of differences here: https://discord.com/channels/671072748985909258/671072748985909264/1142858075682906132

Just a guess, but I'm assuming we have a significant number of functions in C files that aren't marked static, when they could be.

Doing so would allow the compiler to better optimise, including removing the code from object files entirely if never called. This might be a useful size reduction in magiclantern.

Should be largely automatable with some combination of objdump and maybe semgrep to find usages.

We use uart_printf() quite a bit in early code debugging where DryosDebugMsg() doesn't work. But, it's annoying because it's a stub some cams don't have (not needed for real builds) so you have to guard it and it's verbose.

Some macro so real builds resolve to nothing and consciously overridden builds (ML_DEV_BUILD=y)? That way it can be left in at no cost.

qprintf() would be better in most ways but we don't always emulate far enough for this to be practical.

D45 use a version / configuration of ARM that has different guarantees to D678. E.g. D6 forbids division by zero, D78 use MMU to disallow access to zero page. These are allowed on D45 and so there are bugs where they unintentionally occur.

We should trigger these conditions on purpose in qemu on D45:

• check they behave the same as on cam
• add a switch to qemu to fail on division by zero (and access to 0x0 etc) on D45
• run regression tests on D45 to see what existing code will fail
• fix those problems so D678 is okay

But see null_pointer_check() which requires access through null pointer on D45 to try and detect buggy Canon code.

The portable rom dumper tool allows getting rom dumps from many cams, using the same ML code:
https://www.magiclantern.fm/forum/index.php?topic=16534.0

As far as I can tell, the way this is managed in source control is unusual and difficult to maintain: a separate branch, "recovery" is used, with radical changes to core source files and make profiles. No attempt seems to have been made to plan for integrating this with other branches at a future date.

See e.g. src/disp_direct.c which very clearly shows this if you diff between recovery and unified (or lua_fix), via "hg diff src/disp_direct.c -r recovery:lua_fix". If you try and diff the whole branch... well, enjoy yourself, it's a 57k line diff. Because each branch will have had multiple other, different, branches merged into them at various points over the 10+ years these branches have been separate.

Do we want the functionality of portable rom dumper? If so, how do we merge this code? My first thought is a different make target, and instead of maintaining two incompatible copies of e.g. disp_direct.c, we have something like src/disp_direct_portable.c or maybe src/disp_direct.c and src/portable/disp_direct.c.

"make" in platform/ will use -j16 etc, but only within one cam, these are deliberately serialised to avoid race conditions (rather than fix the actual bugs, which is a different question...).

"make disk_image" is quite slow (needs to spin up a qemu vm to mount the disk image) and should be safe to parallelise. It depends on the zip for each cam, so it should be possible to keep the cam builds serial, and trigger a parallel disk_image job at the end.

See platform/Makefile:

 19 # => use .NOTPARALLEL in the upper-level Makefiles only
 20 # parallel build is still used within each platform
 21 .NOTPARALLEL:
 22 #

I think critix might have spotted this. Here's my comment from dual_iso.c:

    // SJE TODO gain better understanding of why modern digic cams,
    // apparently including 1300D, fail to take the sem if 2nd param is 0.
    // On prior cams, 0 works fine.  But there are other uses with 0 on D678
    // that seem okay!  E.g. the menu sem.
    take_semaphore(isoless_sem, 1);

isoless_sem is one we make ourselves, so I'd guess this is an API change on Canon's side.

We should audit all our uses of take_semaphore() and see if the problem happens elsewhere. Potentially this could break a lot of things.

ARMv7 cams have Thumb, but modules are built only as ARMv5 / ARM, for compatibility reasons. We have a custom loader via tcc anyway, so we could perhaps build these as fat binaries, containing a Thumb and non-Thumb build, with the loader detecting the best choice to load.

The Thumb builds have quite a lot less memory footprint (and presumably some speed improvement, as well as lower I-cache pressure).

ARMv5te supports Thumb (just 1, not 2?) so it might be an option there as well. This might mean three builds per file is optimal, but we have a lot more disk space than memory so I think this would be fine.

Connected with #31 (visible due to that).

Arkanoid uses index 0 as background color. For example

         // erase elements that changed their position (to minimize flicker)
         if (e->old_x != e->x || e->old_y != e->y)
         {
            //kitor: color index 0 is transparent on D45, we draw it as #00000000
            arkanoid_draw_elem(e, e->old_x, e->old_y, 0); 

This works on D45 and allows to play over LV.

With how we implemented opacity handling I see no easy way out for D678.
Only thing that comes to mind so far is replace hardcoded index 0 with constant: COLOR_TRANSPARENT for D45 and COLOR_BLACK for D6+ and replace clrscr() on D6+ with COLOR_BLACK fill. Feature to play over LV won't work anymore (it will be always black background) but at least it would work properly.

First I'd like to thank you for the latest updates:
It allowed me to successfully build, install and boot the latest dev branch on three cameras (5DII, 6D and 550D).

The image for the 1100D was building and installing fine, but unfortunately it is blinking SOS when booting.

Maybe it's worth to mention that

• it starts blinking directly after inserting the card, even if the camera is still turned off.
• pressing SET when closing the card-slot does not prevent the crash
• the latest "official" build (from 2018-07-03) is working fine, so the camera is ok

The screen stays black, so I have no more info than that.

I'm going to build qemu-eos and see if it's reproducible there.
But since the documentation is slightly outdated it might take a bit ;-)

850D MMU code works (tested locally, repo has the patch disabled in mmu_patches.h). However, even with no patch enabled (possibly only with? Haven't yet tested with enabled), attempting to enter LV immediately fails and hangs the cam, with a crash log created.

Two logs so far observed:

ASSERT: nullptr
at MainFrame::ConcurrentFlow.c:421, **INT-1Bh**:e058906f
lv:1 mode:1

Or:

ASSERT: nullptr
at AutoExposure::EosAE_MainState.c:2497, EosAE:227290b
lv:1 mode:1

Possibly this relates to kitor's problems with MMU builds on EVF cams, LV probably gets automatically started.

crop_rec_4k_mlv_snd turns out to be another important branch that we want code from. In the normal pattern for ML development, this branch contains several different features, most unrelated to the branch name.

One such feature is changes to SRM allocator code, which look more tested / advanced than lua_fix branch. We should validate if D678 is still broken around SRM with this code.

As discussed in #51 ptpcam does not shut down properly but tends to crash when it is terminated via ctrl+c. It's not critical but it would be a nice fix.

Selftest module is quite useful. But it's a lot of tests, tries to run all of them, and for the new cams, we expect it to fail a lot.

An option to terminate the test on the first failure (while still logging it), would speed things up when working on improving compliance on new cams.

Both R and RP (untested on M50) have random crashes related to EvfCap task. Recently @coon42 got nice trace [1] that sent us into stateobj_lv_spy() from state-object.c. Disabling state objects use (implemented in a89d71f) mitigates the issue, but requires investigation in future.

stateobj_lv_spy() is replacement state transition function that we install in EVF_STATE. Quick static analysis yielded nothing, except that we might be oversimplifying. Real state transition function has a couple of checks that if I'm not mistaken - we have not implemented.

[1]

7968.524 in menu_open
8090.038 [LVEVFC] ERROR SendEventEvfDev : [12][e005edfb]
8090.081 [STARTUP] ERROR ASSERT : LiveView::EvfCapState.c
8090.106 [STARTUP] ASSERT : Task = EvfCap
8090.109 [STARTUP] ASSERT : Core 0
8090.114 [STARTUP] ASSERT : Line 370
8090.120 [STARTUP] < StackDump >
8090.123 [STARTUP] SP: 0x00213D44
(...)

[DM] FROM Write Complete!!!
     3667:  38111.302 SHUTDOWN REASON 1

bench module depends on icache_unlock() and dcache_unlock() in mem_perf.c. These don't exist on ARMv7. We can't use a camera CONFIG define to guard this because modules aren't built in a camera context. This means bench doesn't get included on builds for modern cams because they fail the dep check.

From other modules, it appears the fix / bodge is to do something like this:
extern WEAK_FUNC(ret_0) void raw_lv_request();

Work out the appropriate fake function to replace these with and check if this is an appropriate solution.

In crop_rec_4k branch, state-object.c, we have this new guard, should it also apply to D678?
#if !defined(CONFIG_DIGIC_V)
digic_iso_step()
This is because digic_iso_step() has moved for CONFIG_DIGIC_V, but still we want to know which D678 should be. Maybe it will help with something?

What should the guard really be?

Digic 5 is known to have fast (hw accel?) operations on buffers. These ops include subtraction: https://www.magiclantern.fm/forum/index.php?topic=13408.0

We could subtract raw video frames from a keyframe (every x-th? The prior?) before compressing. This should compress better and could be restored perfectly in post. Would this overall be an increase in effective data rate?

At least on R5, GUI XCM buffer is no longer 960x540. In fact it is 2048x1080 with offset 152,30.

For now this is bodged in #62, but requires a better solution.

One of possible answers is to use XCM and draw on our own layer. This is blocked by #82, as if buffer is just malloc'd, XCM crashes with invalid IBus address.

Today's Ubuntu LTS of 22.04 LTS lacks candidate for package installation named Python

joelwindows7@JOEL-ROG-GL503GE:~/sauce/simplified/magic-lantern/contrib/qemu$ ./install.sh

This will setup QEMU for emulating Magic Lantern.
Thou shalt not be afraid of compiling stuff on Linux ;)
Continue? [y/n] y

*** You have a valid ARM GCC/GDB already installed - using that one.
*** Checking dependencies for Ubuntu...

Not installed: python
Not installed: python-pip
Not installed: python-docutils

*** Installing dependencies for Ubuntu...

Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Package python is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
However the following packages replace it:
  2to3 python2-minimal:i386 python2:i386 python2-minimal python2 dh-python python-is-python3

Package python-docutils is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'python' has no installation candidate
E: Package 'python-docutils' has no installation candidate
joelwindows7@JOEL-ROG-GL503GE:~/sauce/simplified/magic-lantern/contrib/qemu$ uname -a
Linux JOEL-ROG-GL503GE 5.15.68.1-microsoft-standard-WSL2 #1 SMP Mon Sep 19 19:14:52 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
joelwindows7@JOEL-ROG-GL503GE:~/sauce/simplified/magic-lantern/contrib/qemu$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.1 LTS
Release:        22.04
Codename:       jammy
joelwindows7@JOEL-ROG-GL503GE:~/sauce/simplified/magic-lantern/contrib/qemu$

Then, you cannot detect if package named just python this way anymore.

Okay, what Python this script uses? yeah just detect it by explicit name like python2, python3 whatever.. I think..

Hello
How are you?
I want to remove some unnecessary language options for Canon camera menu?
Can you provide a firmware which supports ONLY English, Japanese and Chinese?
Thanks

How to reproduce

Change this line to this:

rst2htmlCommands = ["rst2html5", "rst2html5.py"]

To simulate someone who only has rst2html5 on their computer.

Now make zip in any platform/camera.

Some modules will have missing module strings errors:

[ CC       ]   dual_iso.o
In file included from dual_iso.c:60:
../../src/module.h:345:10: fatal error: module_strings.h: No such file or directory
  345 | #include "module_strings.h"
      |          ^~~~~~~~~~~~~~~~~~

Overall will fail to build.

Explanation

It becomes clear why this happens if you go to modules/adv_int (or any module with a long README.rst), and run both of these commands:

cat README.rst | grep -v -E "^:([^:])+:.+$" | rst2html.py | python2 ../html2text.py -b 700 | sed "s/\r$//"

cat README.rst | grep -v -E "^:([^:])+:.+$" | rst2html5.py | python2 ../html2text.py -b 700 | sed "s/\r$//"

Output 1

# Advanced Intervalometer

Advanced ramping and exposure control for the intervalometer

Create keyframes from current camera settings and specify the
frame at which the keyframe is applied. You will select which
parameters you would like to set in the keyframe and which you
would like to ignore. The module ramps (linearly) the vaules of
selected parameters from one keyframe to the next while the
intervalometer is running.

# Usage

You should create an initial keyframe at 1, that has the settings
you are going to start with, that way the module knows what to
ramp from. If you don't do this the module won't ramp to your
first keyframe, it will simply set the values when it gets there (it
doesn't know how to calculate the ramp, b/c it doesn't know what
the values started as).

# Exposure ramping

To ramp basic expo controls (Av, Tv, ISO) set them like normal in
the canon GUI as if you were taking a picture, then go into the
ML menu under 'Shoot' -> 'Intervalometer' -> 'Advanced
Intervalometer' -> 'New Keyframe' and create a new keyframe.
Turn on or off to specify which parameters are to be included in
the keyframe. The current values are displayed on the right for
you, but you cannot change them from here.

# Focus ramping

For Focus you do specify the offset to focus to from the 'New
Keyframe' menu. The module will calculate the number of focus
steps it need to take each frame to arrive at your offset that
you specify. It steps from wherever the lens is focused at the
start. Negative values mean focus closer, positve is towards
infinity. You need to be in LiveView for the focusing to work and
make sure there is enough time after the exposure is taken,
before the next one starts for the focusing to take place (I
recommend at least 2s, maybe more depending on camera and
lens)

# Misc notes

You can ramp the intervalometer period as well, this you also
specify from within the new keyframe menu

Once you have created keyframes you can view them with the
'List Keyframes' menu. You can also save your keyframe
sequence to file, and then reload them later. If not saved, any
keyframes you created will be lost when the camera is turned
off. Keyframes are saved and loaded from a file called "SEQ.TXT"
in the ML/SETTINGS directory.

When you have created keyframes and are ready to begin, make
sure the advanced intervalometer is turned on, then turn the
intervalometer on as usual from the ML Shoot menu.

# AutoETTR compatibility

This module is compatible with AutoETTR so long as you don't try
to ramp the parameters that AutoETTR is trying to change. You
can ramp Av and let AutoETTR take care of Tv and ISO, or you
can ramp Av and Tv and let AuttoETTR only set ISO, or put
AutoETTR to link to Canon shutter (then your Tv will basically be
setting the slowest Tv parameter of AutoETTR)

Output 2

# Advanced Intervalometer

Advanced ramping and exposure control for the intervalometer

Create keyframes from current camera settings and specify the
frame at which the keyframe is applied. You will select which
parameters you would like to set in the keyframe and which you
would like to ignore. The module ramps (linearly) the vaules of
selected parameters from one keyframe to the next while the
intervalometer is running.

## Usage

You should create an initial keyframe at 1, that has the settings
you are going to start with, that way the module knows what to
ramp from. If you don't do this the module won't ramp to your
first keyframe, it will simply set the values when it gets there (it
doesn't know how to calculate the ramp, b/c it doesn't know what
the values started as).

## Exposure ramping

To ramp basic expo controls (Av, Tv, ISO) set them like normal in
the canon GUI as if you were taking a picture, then go into the
ML menu under 'Shoot' -> 'Intervalometer' -> 'Advanced
Intervalometer' -> 'New Keyframe' and create a new keyframe.
Turn on or off to specify which parameters are to be included in
the keyframe. The current values are displayed on the right for
you, but you cannot change them from here.

## Focus ramping

For Focus you do specify the offset to focus to from the 'New
Keyframe' menu. The module will calculate the number of focus
steps it need to take each frame to arrive at your offset that
you specify. It steps from wherever the lens is focused at the
start. Negative values mean focus closer, positve is towards
infinity. You need to be in LiveView for the focusing to work and
make sure there is enough time after the exposure is taken,
before the next one starts for the focusing to take place (I
recommend at least 2s, maybe more depending on camera and
lens)

## Misc notes

You can ramp the intervalometer period as well, this you also
specify from within the new keyframe menu

Once you have created keyframes you can view them with the
'List Keyframes' menu. You can also save your keyframe
sequence to file, and then reload them later. If not saved, any
keyframes you created will be lost when the camera is turned
off. Keyframes are saved and loaded from a file called "SEQ.TXT"
in the ML/SETTINGS directory.

When you have created keyframes and are ready to begin, make
sure the advanced intervalometer is turned on, then turn the
intervalometer on as usual from the ML Shoot menu.

## AutoETTR compatibility

This module is compatible with AutoETTR so long as you don't try
to ramp the parameters that AutoETTR is trying to change. You
can ramp Av and let AutoETTR take care of Tv and ISO, or you
can ramp Av and Tv and let AuttoETTR only set ISO, or put
AutoETTR to link to Canon shutter (then your Tv will basically be
setting the slowest Tv parameter of AutoETTR)

rst2html5 produces different html header tags, causing html2text.py to produce different markdown header tags (two # instead of one)