To code something with Python for analysing data, and code machine learning algorithms.

To initialize environment, run the commands below:

python -m venv .env

source .env/bin/activate

To install dependencies, run "pip install -r requirements.txt" script at the root of the project, to speed up in China, use mirror like to append the script "-i https://pypi.tuna.tsinghua.edu.cn/simple/";

Run "shiny run" script in terminal to run shiny app (coded in the app.py).

Request API key accessing the URL https://fredaccount.stlouisfed.org/apikey, need registration first if not yet

Can search to find series id, such as Chinese Yuan Renminbi to U.S. Dollar Spot Exchange Rate: https://fred.stlouisfed.org/series/DEXCHUS, which id is DEXCHUS and frequency is Daily, the EXCHUS https://fred.stlouisfed.org/series/EXCHUS frequency is monthly

Edit the xchange_rate_analysis.py to replace API key and series id with yours.

No complete yet - TODO

TODO

TODO

TODO

docker compose to start containers for MinIO/Dremio/Nessie by referring to the code repo: https://github.com/miniohq/datalake_ref_arch (note that the code repo is to create data source in Dremio with Nessie for catalog)

Run execute: brew install apache-spark; once done, to run spark-shell for quick running script in terminal but which is optional

24/08/14 21:30:46 WARN FileSystem: Failed to initialize fileystem hdfs://master:8020/user/hive/warehouse: java.lang.IllegalArgumentException: java.net.UnknownHostException: master
24/08/14 21:30:46 WARN SharedState: Cannot qualify the warehouse path, leaving it unqualified.
java.lang.IllegalArgumentException: java.net.UnknownHostException: master

Run "python3 pyspark-iceberg-test.py", at first it will download dependencies (maven) from maven repo, note that it's to download maven jars during executing the python script, which may a bit slow like in China

After execution, you can access to view what table created and data inserted via Spark SQL implemented in the python script mentioned above:

• MinIO: http://localhost:9051/login, with account minioadmin/minioadmin;
• Dremio: http://localhost:9053/login, with account admin/bad4admins;
• Nessie: http://localhost:19120/tree/main.

The pyrightconfig file is used to enable Python virtual environment (so that code navigation works).

• Python 数据分析工具箱
• 用 Python 语言开发 Shiny 应用
• Time-Series Analysis and Forecasting of Foreign Exchange Rate with ARIMA Model
• Time-Series Analysis and Forecasting of Foreign Exchange Rate with SARIMAX Model
• 动手实战人工智能 AI By Doing
• 现代应用统计
• Modern Datalake Reference Tech Stack, docker compose for MinIO/Dremio/Nessie ...
• Manage Iceberg Tables with Spark
• A Developer’s Introduction to Apache Iceberg using MinIO