see 724bc61 for more context, but basically because npm rebuild is called on the nodejs-project via nodejs-mobile in a gradle task, it attempt to run the postinstall script, which references patch-package, even though at that point, the node module is already patched and patch-package is not available

Currently useDraftObservations() uses Async Storage to store persisted state. Switch to the mmkv zustand model that is being used in this repo. This hook also stores photo promises (photos that are being processed). photo promises are not stored in persisted memory. This logic should remain in a context

To Do:

• Create a usePersistedDraft Observation Hook that stores relevant data in persisted state
• Create a PhotoPromisesContext, that stores photos as they are being processed
• Create a useDraftObservation hooks that encapsulates the logic of persisted drafts and photo Promises

@ErikSin the LocationContext is mapping values on LocationObjectCoords that are 0 to undefined: https://github.com/digidem/CoMapeo-mobile/blob/1e85bb5dc5e931ba7bff3e8fa6f2917c19e6a522/src/frontend/contexts/LocationContext.tsx#L127 I think this should be:

const newCoord = Object.entries(location.coords).map(
  ([key, val]) => [key, val === null ? undefined : val] as const,
);

Any reason to map from null to undefined? For mapping an object and preserving types I've found map-obj to be helpful. See also filter-obj

First screen at launch

• User launches CoMapeo for the first time
• User sees the permissions prompt for the devices Camera and Location
• User is invited to Get Started

Todo

• Display onboarding screen with logo
• Display permissions request for Camera
• Display permissions request for Location
• Create button Get Started button

Design

*** Updated design: https://digidem.slack.com/archives/D025UPW0TUG/p1697140727438439
***old design: https://www.figma.com/file/uzynHXR0xgJu19lsCoJyXN/%F0%9F%90%9A-Shell-App?type=design&node-id=1069-4606&mode=design&t=8p1fA5Pu7n24D7E5-0

Copydeck

https://docs.google.com/spreadsheets/d/18x135fNGqwryXcBoJGvtNSJJMHuPB48Vp0o1l16UyzY/edit?usp=sharing

Bring over the category chooser screen, with a mocked API of the presets

To Do

• mock preset API
• create ui
• create a draftObservationAction that updates the tags
• selecting a category should update draft
• show a placeholder icon

Realized I didn't set up any of that stuff for this repo so incorporate:

• prettier (formatting)
• husky (pre-commit formatting)
• lint-staged (combined with husky to apply on staged files)

Create a release workflow (triggered by Github actions).

Workflows:

• Deploy Release
• Deploy Release Candidate

To Do

• Renames the apks with the current datetime and a reference the git commit and the bitrise build slug
• Upload QA variants to amazon s3 buckets
• Get Keystore key
• Build backend
• Build Translations
• Set Version Name
• Prepare Js bundle

Should follow Mapeo Mobile's Bitrise YAML

Extracted from #81 (comment)

We hardcode the native deps that we use but this is prone to breaking if we update any backend deps. For future reference, I wrote some adapted code in https://github.com/digidem/nodejs-mobile-prebuilds/blob/main/build.js#L118 to get this information.

From Mapeo Mobile, bring the src/frontend/navigation folder, and install all the required dependecies.

To Do:

• port over navigation folder from mapeo-mobile
• port over navigation related typing from mapeo-mobile
• port over custom navigation hooks from mapeo-mobile
• port over custom header

Thoughts from Gregor:

For every native dependency that Mapeo Core Next uses, we should create a fork of every one and build the prebuilds that are hosted in these repos. When installing the backend dependencies, download these prebuilds so that they can be used by NodeJS Mobile. This allows us to avoid need the NDK to build them from source via NodeJS Mobile, which will reduce friction for local development, CI, and releases.

Can use this to build the prebuilds:

https://github.com/staltz/prebuild-for-nodejs-mobile

Create screen that allows the user to choose what format their coordinates are in.

To Do

• Coordinate Format State should be a global state
• Include Decimal Degrees
• Include DMS
• Include `UTM

Screen Shot:

To Port Over

• images folder
• animations folder
• public folder
• sharedComponents/icons folder

Bring over the custom Bottom Sheet Modal Implementation from Mapeo Mobile

To Do

• Bring SharedComponents/BottomSheet folder
• Bring SharedComponents/BottomSheetModal folder
• Create e2e tests confirming that the modals open and close

The client implementation needs to handle two scenarios that can cause race-conditions / lost messages:

1. nodejs-mobile has not started when the front-end starts, which means that messages sent via nodejs.channel will go nowhere and not receive a response.
2. nodejs-mobile has already started when the front-end starts, e.g. if the front-end is slow to start, or if the front-end is reloaded (which does not reload the nodejs-mobile process). This is important because if we are listening for a "backend started" event to solve (1), then we would never receive that event in scenario (2).

I suggest wrapping this into the MessagePortLike abstraction, and queuing messages until we know nodejs-mobile has started and nodejs.channel is functioning. That way the rest of the app / api does not need to worry about this. This means listening for a "started" message that we will need to send from the backend when it first loads, to solve (1), and probably sending a "get state" message to the backend, and the backend to respond with the "started" message when it receives this message, to solve (2).

In order to actually use Mapeo Core we need to get native modules to work with NodeJS Mobile React Native. The general plan is to create and host prebuilds of the relevant native modules outside the project, and then have a build script in this repo that pulls the prebuild and places it in the directory required by the NodeJS Mobile build process.

Rough steps are:

• 1. Identify the modules that we need prebuilds for.

Doing a quick check using npx native-modules on the mapeo core repo lists these:

better-sqlite3
crc-universal
fs-native-extensions
napi-build-utils # not needed
quickbit-native
simdle-native
sodium-native
udx-native
napi-macros/example # not needed
sodium-universal -> sodium-native # ideally de-duped 

The main ones are better-sqlite3 and (maybe) sodium-native (although using sodium-universal may allow us to tackle it later on). Hypercore ecosystem introduced quite a few more but hopefully they're not too tricky to get built (since they're rather small projects). There may be other ones coming for peer deps such as Fastify that aren't listed.

• 2. Create a repo that has a CI action to create prebuilds for our desired architectures and then upload as a GitHub artifacts.

  • most likely can use https://github.com/staltz/prebuild-for-nodejs-mobile for the prebuild step

• 3. Create a (postinstall?) script for this repo that pulls the desired prebuild artifacts and moves them to the correct directories. These directories are specified by NodeJS Mobile's React Native Android build process.

• 4. [maybe] Updating the backend build script that bundles the backend code. we do this to improve startup time and while I'm not sure what changes will be needed, I'm anticipating that something will need to be done here.

Research and implement the best solution of storing encrypted key

To do

• Research best practices for storing encrypted keys in React Native
• Implement storage
• update mmkv to use an encrypted key from this storage

The new logging for mapeo-core makes the code run really slowly, because it logs every state event during sync. We should not enable this by default, and also should probably reduce the amount of logging so that the slowdown is not quite so huge.

Task: Remove this line: https://github.com/digidem/CoMapeo-mobile/blob/e4b88cdafbb430a3c6a448c8ce0e57558904ac77/src/backend/src/app.js#L46

The following contexts need to be refactored to use the new usePersistedState Architecture

To Refactor

• LocationContext
• DraftObservationContext*
• SettingsContext*

*All the logic should be transferable to the zustand store and the context can be deleted

To be copy and pasted

• PermissionsContext

To be done at a later time

• Config Context (new architecture, waiting for backend)
• MapImportsContext (descoped)
• ObservationsContext (Erik is working on)

Being done in another ticket

• IntlContext
• SecurityContext

Can be ported over as is

• useIsFullyFocused
• useIsMounted
• useNavigationWithTypes
• useWifiStatus

Listen to the nodejs-mobile process and only renders children when the server is up and running and listening.
On initial app load it waits for the server before dismissing the native splash screen.
If it doesn't hear a heartbeat message for timeout, it displays a screen to the user so they know something is wrong.

To do

• Frontend initiates the backend using the nodejs channel messaging (done in Splash Screen)
• Wait for the start messaage before showing the app (done in Splash Screen)
• Show error message if it does not correctly start
• When the app goes into the background, pause the server, and when it comes back to the foreground, resume the server
  

Developer Experience (DX) for Mapeo Mobile has not been great. Some key challenges include:

1. Setting up the Android build environment (Android Studio, SDK, Gradle) is a challenge for any contributor.
2. Compounding (1), our backend code in nodejs-mobile includes native C++ code, which needs to be compiled from source using NDK. Setting up NDK and configuring npm & node-gyp for the build environment is an additional challenge on top of the Android Studio setup.
3. Updating React Native involves complex updates to all the files in the android and ios folders, and keeping that in track with our own modifications to these folders is major challenge.
4. We often have challenges with native (Java) modules not working with different versions of React Native.
5. Our use of v8 (rather than JSC) for a JS engine creates additional headaches when upgrading. This change was initially made for better performance and a lower memory footprint, but Hermes might be the better supported option going forwards.
6. Building the app on Bitrise is a constant challenge, because the machine setup changes on Bitrise's end, and new versions of React Native require different SDK tools to be installed.

One route towards improving the developer experience is to make more use of the Expo ecosystem. We already use some Expo modules in Mapeo Mobile — they are often the best maintained react native modules and are quickly updated to support new versions of RN. There are two ways we can leverage Expo, with different resource requirements:

1. Expo "bare" project

Expo can work with an existing "bare" React Native app. We can use Expo Application Services (Expo's PaaS for building and publishing apps), and we can use the expo modules & cli to facilitate local development. However we still need to manage the contents of android and ios folders ourselves. With a bare project we can create prebuilds, which allow developers to develop the front end and view on a device/emulator without needing the Android or iOS build setup: the frontend JS code can load in the prebuild, which can be built by one person / CI every time we change a native module, and then developers can install that prebuild and develop Mapeo in that.

2. Expo fully managed project

If you start an expo project from scratch, Expo fully manages the "native" side of the app, and the android and ios files can be generated by expo at build-time. As long as we do not edit those folders then we don't need to check them in, and we don't need to worry about them for updates. Expo supports native modules that are not included in the Expo SDK via plugins. Plugins are a config that tells expo how to modify the generated native code. For example Mapbox-gl maps includes config for installing into an expo project.

The advantage of (2) is that it should simplify the process of updating and maintaining the app - we should never need to touch any gradle config or android or ios config, since Expo will generate all of that for us. There are some key development challenges for (2) though:

1. We will need to create an Expo plugin configuration for nodejs-mobile-react-native.
2. We need to figure out the relevant expo config needed for things like our debug builds that depend on changes we have made to the native side of the app. I think most things we need are supported in Expo though, the only exception being the config to build the Mapeo for ICCAs app from the same codebase, but we do not plan to do that with Mapeo 6, so we can drop it.

Improvements beyond Expo

One of the biggest challenges is configuring the build environment for native C++ modules that we use on the nodejs-mobile side. Configuring NDK correctly has consistently proved a major headache. nodejs-mobile however does support prebuilds, which remove the need to do any building from source at build time. Solution: We setup some CI processes to automatically build a publish prebuilds for the native node modules that we use, e.g. better-sqlite3 and sodium-native. We publish these with a separate name, but will be forks of the original modules e.g. @mapeo/better-sqlite3. We can automate updating these when the original module updates. This means we can set up the build environment once in CI, and for developers they only need to download these prebuidls.

Another option we can look at for a nice DX is to conditionally require non-expo native modules so that developers can use the Expo Go client: https://docs.expo.dev/bare/using-expo-client/. This would mean having a placeholder for the map screen and mocking the Mapeo API, because it would not have the nodejs process. However the advantage would be that you can view Mapeo in the browser, or very easily start developing on the screens (other than the Map screen) with just the regular Expo Go client.

Create a Zustand store which we will incrementally replace all the contexts/providers eventually/

To Do

• Create Zustand Store
• Create Hooks, separating action from state
• Create a createPersistedState hook, integrating persisting state middleware (mmkv)

For Simplicity, the createPersistedState hook should use the same pattern as Mapeo Mobile, expect make the key a type, that contains hardcoded strings as keys. Also, because we are using mmkv, which is synchronous, we do not need access to Status

type persistedStorageKeys = 'Settings' | 'Security' // example keys

function createPersistedState(key:persistedStorageKeys) : <S>(initialValue: S) => readonly [S, ZustandStateSetter<S>] { ....}

Incomplete

Needs to be refactored

• useAllObservations
• useObservation
• useDraftObservation
• usePersistedState
• useProjectInviteListener
• useSettingsValue
• useExperiments
• usePermissions
• useUpdateNotifierEffect

Requires Backend

• useDeviceId
• useDeviceInfo
• useMapAvailability
• useMapImportProgress
• useMapImports
• useMapStyles
• useSelectedMapStyle

Can be ported over as is

• useIsFullyFocused
• useIsMounted
• useNavigationWithTypes
• useWifiStatus

Include Bug Snag for reporting bugs.

To Do

• Set up Bug Snag
• Attach it to the Digital-Democracy Account
• Create README.md to track all instances of bug snag throughout the app

Bring over observation list. Use Tanstack query to fetch observations (and mock api)

To do

• bring over observation and observationList
• use Tanstack Query

Due to issues with Expo, upgraded React-Native Dependecy from 70.xx to 71.13.

This allowed the adding of expo module

This also meant an increase in version of mmkv, and the older version was not compatible with RN > 71.

Extracte from #81 (comment)

Currently we rely on the dev environment Node runtime i.e. whatever is running the script, but this will break if we need to change that for reasons related to things like React Native's CLI.

Not sure if there's a good way of finding out which node version the nodejs-mobile runtime is using, aside from just something that's hardcoded.

It's been released recently but requires some updates to the native prebuilds setup we have, so would prefer to tackle it later on (also give Staltz some time to address any potential issues with the new release)

Description:

For better-sqlite3, receive the following error when running ../../scripts/rebuild-android.sh better-sqlite3 arm:

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: file: Release/obj.target/sqlite3/gen/sqlite3/sqlite3.o is not an object file (not allowed in a library)

In general, this process shouldn’t be using XCode’s libtool but instead should be using the libtool provided by the Android NDK.

Debugging

The outcomes of running node-gyp rebuild --release (armeabi-v7a) are slightly different:

Mobile via nodejs-mobile-react-native gradle config (Node 12, npm 6), which succeeds:

> [email protected] build-release /Users/andrewchou/GitHub/digidem/mapeo-mobile/android/build/nodejs-native-assets-temp-build/nodejs-native-assets-armeabi-v7a/nodejs-project/node_modules/better-sqlite3
> node-gyp rebuild --release

  touch ba23eeee118cd63e16015df367567cb043fed872.intermediate
  LD_LIBRARY_PATH=/Users/andrewchou/GitHub/digidem/mapeo-mobile/android/build/nodejs-native-assets-temp-build/nodejs-native-assets-armeabi-v7a/nodejs-project/node_modules/better-sqlite3/build/Release/lib.host:/Users/andrewchou/GitHub/digidem/mapeo-mobile/android/build/nodejs-native-assets-temp-build/nodejs-native-assets-armeabi-v7a/nodejs-project/node_modules/better-sqlite3/build/Release/lib.target:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; cd ../deps; mkdir -p /Users/andrewchou/GitHub/digidem/mapeo-mobile/android/build/nodejs-native-assets-temp-build/nodejs-native-assets-armeabi-v7a/nodejs-project/node_modules/better-sqlite3/build/Release/obj/gen/sqlite3; node copy.js "/Users/andrewchou/GitHub/digidem/mapeo-mobile/android/build/nodejs-native-assets-temp-build/nodejs-native-assets-armeabi-v7a/nodejs-project/node_modules/better-sqlite3/build/Release/obj/gen/sqlite3" ""
  touch Release/obj.target/deps/locate_sqlite3.stamp
  /Users/andrewchou/Library/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/bin/armv7a-linux-androideabi21-clang '-DNODE_GYP_MODULE_NAME=sqlite3' '-DUSING_UV_SHARED=1' '-DUSING_V8_SHARED=1' '-DV8_DEPRECATION_WARNINGS=1' '-DV8_DEPRECATION_WARNINGS' '-DV8_IMMINENT_DEPRECATION_WARNINGS' '-D_LARGEFILE_SOURCE' '-D_FILE_OFFSET_BITS=64' '-D__STDC_FORMAT_MACROS' '-DOPENSSL_NO_PINSHARED' '-DOPENSSL_THREADS' '-DSQLITE_DQS=0' '-DSQLITE_LIKE_DOESNT_MATCH_BLOBS' '-DSQLITE_THREADSAFE=2' '-DSQLITE_USE_URI=0' '-DSQLITE_DEFAULT_MEMSTATUS=0' '-DSQLITE_OMIT_DEPRECATED' '-DSQLITE_OMIT_GET_TABLE' '-DSQLITE_OMIT_TCL_VARIABLE' '-DSQLITE_OMIT_PROGRESS_CALLBACK' '-DSQLITE_OMIT_SHARED_CACHE' '-DSQLITE_TRACE_SIZE_LIMIT=32' '-DSQLITE_DEFAULT_CACHE_SIZE=-16000' '-DSQLITE_DEFAULT_FOREIGN_KEYS=1' '-DSQLITE_DEFAULT_WAL_SYNCHRONOUS=1' '-DSQLITE_ENABLE_MATH_FUNCTIONS' '-DSQLITE_ENABLE_DESERIALIZE' '-DSQLITE_ENABLE_COLUMN_METADATA' '-DSQLITE_ENABLE_UPDATE_DELETE_LIMIT' '-DSQLITE_ENABLE_STAT4' '-DSQLITE_ENABLE_FTS3_PARENTHESIS' '-DSQLITE_ENABLE_FTS3' '-DSQLITE_ENABLE_FTS4' '-DSQLITE_ENABLE_FTS5' '-DSQLITE_ENABLE_JSON1' '-DSQLITE_ENABLE_RTREE' '-DSQLITE_ENABLE_GEOPOLY' '-DSQLITE_INTROSPECTION_PRAGMAS' '-DSQLITE_SOUNDEX' '-DHAVE_STDINT_H=1' '-DHAVE_INT8_T=1' '-DHAVE_INT16_T=1' '-DHAVE_INT32_T=1' '-DHAVE_UINT8_T=1' '-DHAVE_UINT16_T=1' '-DHAVE_UINT32_T=1' '-D_GLIBCXX_USE_C99_MATH' '-DNDEBUG' -I/Users/andrewchou/GitHub/digidem/mapeo-mobile/node_modules/nodejs-mobile-react-native/android/libnode/include/node -I/Users/andrewchou/GitHub/digidem/mapeo-mobile/node_modules/nodejs-mobile-react-native/android/libnode/src -I/Users/andrewchou/GitHub/digidem/mapeo-mobile/node_modules/nodejs-mobile-react-native/android/libnode/deps/openssl/config -I/Users/andrewchou/GitHub/digidem/mapeo-mobile/node_modules/nodejs-mobile-react-native/android/libnode/deps/openssl/openssl/include -I/Users/andrewchou/GitHub/digidem/mapeo-mobile/node_modules/nodejs-mobile-react-native/android/libnode/deps/uv/include -I/Users/andrewchou/GitHub/digidem/mapeo-mobile/node_modules/nodejs-mobile-react-native/android/libnode/deps/zlib -I/Users/andrewchou/GitHub/digidem/mapeo-mobile/node_modules/nodejs-mobile-react-native/android/libnode/deps/v8/include -I./Release/obj/gen/sqlite3  -fPIC -Wall -Wextra -Wno-unused-parameter -std=c99 -w -O3 -O3 -fno-omit-frame-pointer -fPIC  -MMD -MF ./Release/.deps/Release/obj.target/sqlite3/gen/sqlite3/sqlite3.o.d.raw   -c -o Release/obj.target/sqlite3/gen/sqlite3/sqlite3.o Release/obj/gen/sqlite3/sqlite3.c
  rm -f Release/obj.target/deps/sqlite3.a && /Users/andrewchou/Library/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/bin/arm-linux-androideabi-ar crs Release/obj.target/deps/sqlite3.a Release/obj.target/sqlite3/gen/sqlite3/sqlite3.o
  rm -rf "Release/sqlite3.a" && cp -af "Release/obj.target/deps/sqlite3.a" "Release/sqlite3.a"
  /Users/andrewchou/Library/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/bin/armv7a-linux-androideabi21-clang++ '-DNODE_GYP_MODULE_NAME=better_sqlite3' '-DUSING_UV_SHARED=1' '-DUSING_V8_SHARED=1' '-DV8_DEPRECATION_WARNINGS=1' '-DV8_DEPRECATION_WARNINGS' '-DV8_IMMINENT_DEPRECATION_WARNINGS' '-D_LARGEFILE_SOURCE' '-D_FILE_OFFSET_BITS=64' '-D__STDC_FORMAT_MACROS' '-DOPENSSL_NO_PINSHARED' '-DOPENSSL_THREADS' '-DBUILDING_NODE_EXTENSION' '-D_GLIBCXX_USE_C99_MATH' '-DNDEBUG' -I/Users/andrewchou/GitHub/digidem/mapeo-mobile/node_modules/nodejs-mobile-react-native/android/libnode/include/node -I/Users/andrewchou/GitHub/digidem/mapeo-mobile/node_modules/nodejs-mobile-react-native/android/libnode/src -I/Users/andrewchou/GitHub/digidem/mapeo-mobile/node_modules/nodejs-mobile-react-native/android/libnode/deps/openssl/config -I/Users/andrewchou/GitHub/digidem/mapeo-mobile/node_modules/nodejs-mobile-react-native/android/libnode/deps/openssl/openssl/include -I/Users/andrewchou/GitHub/digidem/mapeo-mobile/node_modules/nodejs-mobile-react-native/android/libnode/deps/uv/include -I/Users/andrewchou/GitHub/digidem/mapeo-mobile/node_modules/nodejs-mobile-react-native/android/libnode/deps/zlib -I/Users/andrewchou/GitHub/digidem/mapeo-mobile/node_modules/nodejs-mobile-react-native/android/libnode/deps/v8/include -I./Release/obj/gen/sqlite3  -fPIC -Wall -Wextra -Wno-unused-parameter -std=c++14 -O3 -O3 -fno-omit-frame-pointer -fPIC -fno-rtti -fno-exceptions -std=gnu++1y -MMD -MF ./Release/.deps/Release/obj.target/better_sqlite3/src/better_sqlite3.o.d.raw   -c -o Release/obj.target/better_sqlite3/src/better_sqlite3.o ../src/better_sqlite3.cpp
  /Users/andrewchou/Library/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/bin/armv7a-linux-androideabi21-clang++ -shared -rdynamic -fPIC  -Wl,-soname=better_sqlite3.node -o Release/obj.target/better_sqlite3.node -Wl,--start-group Release/obj.target/better_sqlite3/src/better_sqlite3.o Release/obj.target/deps/sqlite3.a -Wl,--end-group -llog /Users/andrewchou/GitHub/digidem/mapeo-mobile/node_modules/nodejs-mobile-react-native/android/libnode//bin/armeabi-v7a/libnode.so
  rm -rf "Release/better_sqlite3.node" && cp -af "Release/obj.target/better_sqlite3.node" "Release/better_sqlite3.node"
  /Users/andrewchou/Library/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/bin/armv7a-linux-androideabi21-clang '-DNODE_GYP_MODULE_NAME=test_extension' '-DUSING_UV_SHARED=1' '-DUSING_V8_SHARED=1' '-DV8_DEPRECATION_WARNINGS=1' '-DV8_DEPRECATION_WARNINGS' '-DV8_IMMINENT_DEPRECATION_WARNINGS' '-D_LARGEFILE_SOURCE' '-D_FILE_OFFSET_BITS=64' '-D__STDC_FORMAT_MACROS' '-DOPENSSL_NO_PINSHARED' '-DOPENSSL_THREADS' '-DBUILDING_NODE_EXTENSION' '-D_GLIBCXX_USE_C99_MATH' '-DNDEBUG' -I/Users/andrewchou/GitHub/digidem/mapeo-mobile/node_modules/nodejs-mobile-react-native/android/libnode/include/node -I/Users/andrewchou/GitHub/digidem/mapeo-mobile/node_modules/nodejs-mobile-react-native/android/libnode/src -I/Users/andrewchou/GitHub/digidem/mapeo-mobile/node_modules/nodejs-mobile-react-native/android/libnode/deps/openssl/config -I/Users/andrewchou/GitHub/digidem/mapeo-mobile/node_modules/nodejs-mobile-react-native/android/libnode/deps/openssl/openssl/include -I/Users/andrewchou/GitHub/digidem/mapeo-mobile/node_modules/nodejs-mobile-react-native/android/libnode/deps/uv/include -I/Users/andrewchou/GitHub/digidem/mapeo-mobile/node_modules/nodejs-mobile-react-native/android/libnode/deps/zlib -I/Users/andrewchou/GitHub/digidem/mapeo-mobile/node_modules/nodejs-mobile-react-native/android/libnode/deps/v8/include -I./Release/obj/gen/sqlite3  -fPIC -Wall -Wextra -Wno-unused-parameter -O3 -O3 -fno-omit-frame-pointer -fPIC  -MMD -MF ./Release/.deps/Release/obj.target/test_extension/deps/test_extension.o.d.raw   -c -o Release/obj.target/test_extension/deps/test_extension.o ../deps/test_extension.c
  /Users/andrewchou/Library/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/bin/armv7a-linux-androideabi21-clang++ -shared -rdynamic -fPIC  -Wl,-soname=test_extension.node -o Release/obj.target/test_extension.node -Wl,--start-group Release/obj.target/test_extension/deps/test_extension.o Release/obj.target/deps/sqlite3.a -Wl,--end-group -llog /Users/andrewchou/GitHub/digidem/mapeo-mobile/node_modules/nodejs-mobile-react-native/android/libnode//bin/armeabi-v7a/libnode.so
  rm -rf "Release/test_extension.node" && cp -af "Release/obj.target/test_extension.node" "Release/test_extension.node"
rm ba23eeee118cd63e16015df367567cb043fed872.intermediate

• notice: rm -f Release/obj.target/deps/sqlite3.a && /Users/andrewchou/Library/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/bin/arm-linux-androideabi-ar crs Release/obj.target/deps/sqlite3.a Release/obj.target/sqlite3/gen/sqlite3/sqlite3.o
  • most likey derived from https://github.com/nodejs/node-gyp/blob/main/gyp/pylib/gyp/generator/make.py#L222

Mobile Next via rebuild-android.sh (Node 16, npm 8), which fails:

> [email protected] build-release
> node-gyp rebuild --release

  touch ba23eeee118cd63e16015df367567cb043fed872.intermediate
  LD_LIBRARY_PATH=/Users/andrewchou/GitHub/digidem/mapeo-mobile-next/src/backend/node_modules/better-sqlite3/build/Release/lib.host:/Users/andrewchou/GitHub/digidem/mapeo-mobile-next/src/backend/node_modules/better-sqlite3/build/Release/lib.target:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; cd ../deps; mkdir -p /Users/andrewchou/GitHub/digidem/mapeo-mobile-next/src/backend/node_modules/better-sqlite3/build/Release/obj/gen/sqlite3; node copy.js "/Users/andrewchou/GitHub/digidem/mapeo-mobile-next/src/backend/node_modules/better-sqlite3/build/Release/obj/gen/sqlite3" ""
  touch Release/obj.target/deps/locate_sqlite3.stamp
  /Users/andrewchou/Library/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/bin/armv7a-linux-androideabi21-clang -o Release/obj.target/sqlite3/gen/sqlite3/sqlite3.o Release/obj/gen/sqlite3/sqlite3.c '-DNODE_GYP_MODULE_NAME=sqlite3' '-DUSING_UV_SHARED=1' '-DUSING_V8_SHARED=1' '-DV8_DEPRECATION_WARNINGS=1' '-DV8_DEPRECATION_WARNINGS' '-DV8_IMMINENT_DEPRECATION_WARNINGS' '-D_LARGEFILE_SOURCE' '-D_FILE_OFFSET_BITS=64' '-D__STDC_FORMAT_MACROS' '-DOPENSSL_THREADS' '-DOPENSSL_NO_ASM' '-DSQLITE_DQS=0' '-DSQLITE_LIKE_DOESNT_MATCH_BLOBS' '-DSQLITE_THREADSAFE=2' '-DSQLITE_USE_URI=0' '-DSQLITE_DEFAULT_MEMSTATUS=0' '-DSQLITE_OMIT_DEPRECATED' '-DSQLITE_OMIT_GET_TABLE' '-DSQLITE_OMIT_TCL_VARIABLE' '-DSQLITE_OMIT_PROGRESS_CALLBACK' '-DSQLITE_OMIT_SHARED_CACHE' '-DSQLITE_TRACE_SIZE_LIMIT=32' '-DSQLITE_DEFAULT_CACHE_SIZE=-16000' '-DSQLITE_DEFAULT_FOREIGN_KEYS=1' '-DSQLITE_DEFAULT_WAL_SYNCHRONOUS=1' '-DSQLITE_ENABLE_MATH_FUNCTIONS' '-DSQLITE_ENABLE_DESERIALIZE' '-DSQLITE_ENABLE_COLUMN_METADATA' '-DSQLITE_ENABLE_UPDATE_DELETE_LIMIT' '-DSQLITE_ENABLE_STAT4' '-DSQLITE_ENABLE_FTS3_PARENTHESIS' '-DSQLITE_ENABLE_FTS3' '-DSQLITE_ENABLE_FTS4' '-DSQLITE_ENABLE_FTS5' '-DSQLITE_ENABLE_JSON1' '-DSQLITE_ENABLE_RTREE' '-DSQLITE_ENABLE_GEOPOLY' '-DSQLITE_INTROSPECTION_PRAGMAS' '-DSQLITE_SOUNDEX' '-DHAVE_STDINT_H=1' '-DHAVE_INT8_T=1' '-DHAVE_INT16_T=1' '-DHAVE_INT32_T=1' '-DHAVE_UINT8_T=1' '-DHAVE_UINT16_T=1' '-DHAVE_UINT32_T=1' '-D_GLIBCXX_USE_C99_MATH' '-DNDEBUG' -I/Users/andrewchou/GitHub/digidem/mapeo-mobile-next/node_modules/nodejs-mobile-react-native/android/libnode/include/node -I/Users/andrewchou/GitHub/digidem/mapeo-mobile-next/node_modules/nodejs-mobile-react-native/android/libnode/src -I/Users/andrewchou/GitHub/digidem/mapeo-mobile-next/node_modules/nodejs-mobile-react-native/android/libnode/deps/openssl/config -I/Users/andrewchou/GitHub/digidem/mapeo-mobile-next/node_modules/nodejs-mobile-react-native/android/libnode/deps/openssl/openssl/include -I/Users/andrewchou/GitHub/digidem/mapeo-mobile-next/node_modules/nodejs-mobile-react-native/android/libnode/deps/uv/include -I/Users/andrewchou/GitHub/digidem/mapeo-mobile-next/node_modules/nodejs-mobile-react-native/android/libnode/deps/zlib -I/Users/andrewchou/GitHub/digidem/mapeo-mobile-next/node_modules/nodejs-mobile-react-native/android/libnode/deps/v8/include -I./Release/obj/gen/sqlite3  -fasm-blocks -mpascal-strings -O3 -mmacosx-version-min=10.7 -arch x86_64 -w -std=c99 -MMD -MF ./Release/.deps/Release/obj.target/sqlite3/gen/sqlite3/sqlite3.o.d.raw   -c
  rm -f Release/sqlite3.a && ./gyp-mac-tool filter-libtool libtool  -static -o Release/sqlite3.a Release/obj.target/sqlite3/gen/sqlite3/sqlite3.o
rm ba23eeee118cd63e16015df367567cb043fed872.intermediate

• notice: rm -f Release/sqlite3.a && ./gyp-mac-tool filter-libtool libtool -static -o Release/sqlite3.a Release/obj.target/sqlite3/gen/sqlite3/sqlite3.o
  • most likely derived from https://github.com/nodejs/node-gyp/blob/main/gyp/pylib/gyp/generator/make.py#L210

There seems to be difference between how node-gyp flags and options are parsed between the projects. Whether it’s because of the difference in node+npm version or due to configuration errors in rebuild-android.sh is not yet clear.

There's a chance that the export npm_config_format="make-android" is not being used by node-gyp for the mobile-next case, causing it to resort to use mac as the build flavor and thus trying to use XCode's libtool. I tried setting GYP_GENERATORS=make-android but that didn't seem to change anything. Wondering if maybe because --ignore-environment needs to be passed to the node-gyp command somewhere in order to use it.

Another thing to consider is maybe differences in the node-gyp implementation? Didn't dig deep into this but doesn't seem likely.

Useful links for node-gyp:

• Entry where options and flags are used: https://github.com/nodejs/node-gyp/blob/main/gyp/pylib/gyp/__init__.py#L663
• Code that processes the node-gyp --formats flag (or GYP_GENERATORS env variable): https://github.com/nodejs/node-gyp/blob/main/gyp/pylib/gyp/__init__.py#L75
• Link command for Mac: https://github.com/nodejs/node-gyp/blob/main/gyp/pylib/gyp/generator/make.py#L210
• Link command for Android: https://github.com/nodejs/node-gyp/blob/main/gyp/pylib/gyp/generator/make.py#L222
• Where flavor (e.g. mac, linux, android, etc) is determined for GenerateOutput function: https://github.com/nodejs/node-gyp/blob/main/gyp/pylib/gyp/generator/make.py#L2399
  • Source for GetFlavor implmentation: https://github.com/nodejs/node-gyp/blob/main/gyp/pylib/gyp/common.py#L433
• Code that determines what link command is ran for the GenerateOutput function: https://github.com/nodejs/node-gyp/blob/main/gyp/pylib/gyp/generator/make.py#L2472-L2483
• Online documentation for Gyp: https://gyp.gsrc.io/

Add a minimal map screen. Should emulate the code from Mapeo-Mobile. Include all the code does not require info from the backend.

The Map Screen, uses MapView, which is a class based react-component. Refactor this component to a function based component.

To Do:

• copy code over from Mapeo-Mobile
• Take out any code that requires backend
• Refactor Map View into a function-based component
• create e2e test where mapbox is visible on open of app (after splash screen)
• upgrade mapbox-gl (deprecated) to rnmapbox v10

Create github actions to run ci and start the build process.

To Do

• run backend tests
• run frontend test

Create methods for mapeo.observation.getByDocId() and mapeo.observation.getMany() that return mocked data. See proposed Datatype API docs. I think most of the work is already done in https://github.com/digidem/mapeo-core-client-experiments (private repo)

The reason for doing this is to enable integration work to be advanced without completing work on Mapeo Core Next. As long as the shape of the API matches (or is close to) what will eventually be the Mapeo Core API, then we can just swap in the "real" Mapeo core when it is ready. We can move ahead with building out the backend process, the RPC bridge, the client API, react-query hooks, and linking the data into the screens, none of which actually needs the "real" mapeo core to be ready to complete.

We can do most of the hard integration work with this mocked Mapeo. Initially just the two APIs of getByDocId() and getMany() allow us to work on most of the screens that need this data.

• manage startup of the nodejs-mobile process #4
• expose the api on the client via rpc-reflector #3
• create hooks e.g. useObservations() and useObservation() (might want to create a general abstraction using react-query, I think @achou11 or @ErikSin already did some work on this).
• copy across the code for observation list view and observation view to this repo, and connect to the hooks

Issue:

When a BottomSheetModal open, and backdrop is enables, an animated backdrop should show and block all background interactions. Currently it is not showing.

Currently uses custom backdrop

create e2e testing. In MapeoMobile e2e tests can be found in e2e folder.

To Do

• Port Tests and testing environment
• ensure basic e2e test work

Note: Not all tests will work since we are only bring some screens over. So just set up the testing environment and make sure some tests work based on the minimal screens that are available.

Bring over the Settings Screen from Mapeo-Mobile. Have each setting navigate to an empty page.

To Do

• Bring settings screen from mapeo mobile
• Each screen should navigate to an empty page

Bring over the observation edit screen, and update the architecture to use the mmkv/zustand store

To do:

• update useDraftObservations to use new architecture
• update locationView
• bring over screens
• using mock api, save an observation

Port over, from Mapeo Mobile, the About Mapeo Screen.

To do:

• Include Mapeo Version
• Include Mapeo Build
• Include Android Version
• Include Android build number
• Include Phone Model

Screen Shots

Bring Over code/screen from App Passcode

To do

• Port Over App Passcode Screens
• Use new usePersistedState() hook/Zustand Store (instead of context)
• Create e2e Tests

e2e tests

• turning App Passcode experiment on or off makes e2e test available in settings menu
• Clicking App Passcode, navigates user to App Passcode Screens
• Clicking "continue", when no app passcode, create error
• Clicking "continue", when there is an passcode, navigates to confirmation screen
• Clicking "continue" in confirmation screen, when no app passcode, create error
• Clicking "continue"in confirmation screen, when there is an passcode, opens a bottom sheet

Extracted from #81 (comment)

We currently include the ABI version in the name of tarball that we store the prebuilds in. This isn't necessary for modules that use N-API (where ABI compat doesn't make a difference). Only trickiness is knowing which native modules DON'T use N-API and there's probably no deterministic way of knowing that, so those would have to be harrdcoded

Research and propose best flow for release with bitrise, gtihub actions, and crowdin integration

To be considered:

• Translations only
• Currently quite complicated
• ICCA and Mapeo are linked which is unnecessary

TICKET IMCOMPLETE: We should discuss strategies for creating translation only releases.

Create a crowdin page that links to this repo. Create a script that extracts all the messages into one file. That file will be used by crowdin to generate the english.

To be ported over

• List Folder
• Touchables Folder
• Add Button
• Animated Radio
• Button
• Centered View *
• Date Distance *
• GpsPill
• HeaderTitle
• IconButton
• Loading *
• OptionRow
• PhotoView
• Pill
• Select
• Text
• TextButton *
• WifiBar

• indicates component that need to be refactored from js to ts

Do not Port over yet:

• Map
• CameraView
• CustomHeaderLeft
• FormattedData
• HomeHeader
• LocationField
• MapThumbnail
• OfflineAreaCard
• OfflineMapLayers
• PasscodeInput
• PracticeMode
• ThumbnailScrollView

Bring over observation from Mapeo Mobile

To Do

• Observation Edit
• Observation
• Observation Details
• Observations List
• Photos Modal

The current javascript engine in this repo is Hermes.. In Mapeo-Mobile we were not using Hermes due to its lack of compatibility with React-Intl. It seems that (incomplete) support of react-intl has been added to hermes.
Test out our uses of the internationalization apis in this repo to determine whether we need to move away from hermes.

To do

• Determine all the different use cases of react-intl
• Test all the use cases in hermes
• For any broken cases, determine if there is a feasible alternative
• If possible, add React-Intl, or if not possible, switch to jsc.

Project invite in Mapeo-Mobile was created before the API, and under different assumptions. Based on the new project invite flow, and the proposed user flow, refactor the invite flow

Project API doc

To do

• Handle Project Switch
• list project
• persist project user is on
• user needs to bring data over

Files used for invite flow:

• frontend/screens/AddToProjectScreen
• frontend/screens/CreateProject
• frontend/screens/LeaveProject
• frontend/screens/Onboarding
• frontend/screens/ConnectingToDevice
• frontend/screens/JoinRequestModal
• frontend/screens/ProjectInviteModal

2 parts of APi
-Client Api
-Inside project: Invite Method attached to the project

Currently MapeoMobile uses react-native-unimodules for all expo dependencies. Unimodules has been deperecated and now use expo modules. Explanation of unimodules can be found here.

Add the latest versions of the following dependencies, and test:

• expo-camera
• expo-document-picker
• expo-linear-gradient
• expo-localization
• expo-location
• expo-sensors

#28 introduced a hook to persist state. This hook followed the pattern used in Mapeo-Mobile. Unfortuntantly this pattern creates persisted state dynamically. This mean this persisted state still needs to propagated using a context. this is a zustand anti pattern

Create a new hook that statically created the persisted hook, and therefore can be called within the react tree without the use of a context