• Muhammad Arif Faishal Haq (M2008G0843) - Machine Learning - Universitas Gadjah Mada
• Kadek Yuki Andika (M2008G0840) - Machine Learning - Universitas Gadjah Mada
• Yohanes Vito Rizki Darujati (A2239F2147) - Mobile Development - Universitas Kristen Duta Wacana
• Satrio Muslim Wibowo (A2110F1432) - Mobile Development - Institut Sains dan Teknologi Akprind
• Ahmad Fatih (C7010F1126) - Cloud Computing - Universitas Indonesia
• Rifky Manuel Satyana (C2008G0841) - Cloud Computing - Universitas Gadjah Mada

ABA is an application to help toddlers learn to read and write by using multisensory materials. ABA is designed to stimulate active learning that focuses on the toddler learning development stage. We use the Montessori learning methods that provides and environment to increases their curiosity, independence, and exploration that is added with hands-on practices. The method develops two learning domains: cognitive -- which the knowledge of absorbing information and how that develops by appliances; and psychomotor -- which is physicality and how that develops by appliances. In tracking the toddlers' competency development, we rely on Bloom's Taxonomy.

To see the demo, click this link in Youtube.

ABA is currently an android application that uses Google Cloud Platform as the cloud service. There are currently 4 vital features in ABA, namely

1. Learning Section
   • Eksplor Huruf dan Eksplor Angka
   • KBBI Anak
2. Practice Section
   • Latihan Menyusun Huruf
   • Latihan Mengeja Huruf
   • Latihan Mengeja Kata (Demo is shown in the presentation)

To use this feature, toddler can click the alphabets on the screen and the app will output a pronunciation of that particular alphabet. This will not only teach toddler about the alphabet by sight, but also by hearing such that children will memorize how the alphabet is pronounce.

KBBI Anak is a feature for toddler to explore Indonesian words that we have designed to be child-friendly. In this feature, toddlers can know the word, know the meaning of the word, and imagine what the word is like in real-life since we included illustrations with the word.

This feature is a designed as a practice for toddlers after what they have learnt in eksplor huruf. Here toddlers can implement the knowledge of letters.

This feature is designed for toddlers to practice pronouncing the alphabets. Here, toddlers can implement the knowledge of how should the alphabets be pronounced in Indonesian. Hence, this touches cognitive and psychomotor abilities.

1. KBBI Anak
   KBBI Anak is taken from 109 children's e-books from the following link. Each e-book is extracted and merged into one file using PyMuPDF. Then, we stemming each word into a basic word using Sastrawi. Next, we matched each word with the KBBI dataset by Kateglo. We only use words found in the KBBI. We also added the meaning of each word in the KBBI. We only use verbs and nouns in KBBI. All of this we do so that toddlers get words that are easy for them to understand. Next, we also add an illustration to each word. KBBI Anak have more than 3000 words that are ready to be learned.
2. Word Recommendation System
   At KBBI Anak, we created a simple algorithm for word recommendations to be displayed. The words that will be displayed are based on Rimakata. "Rima" used in this recommendation system is a "rima akhir sempurna". If there are no matching words in the dataset, the system will randomize the words to be displayed. In making this recommendation system, we also use the following code. In our application we divide the word dataset into "Mudah", "Sedang" and "Sulit" based on the number of letters.

Latihan Mengeja Huruf is ABA's feature that involves speech recognition. The required dataset for this feature is an audio dataset. This dataset contains the pronounciation of 26 alphabets from A to Z which was collected by our team members. There are 2600 audio files (in wav extension) that used in the training process.

First thing to do is labeling each dataset from 0 to 25 which represent each alphabet respectively. After that we need to extract a feature from audio file called Mel-Frequency Ceptral Coefficients (MFCCs) and then begin to build the model. The model used is a deep learning model containing 2 convolution layers and 1 connected layer. The resulting model is astonishing due to the fact that the average accuracy sits approximately at 98%.

Latihan Mengeja Kata is ABA's feature that involves speech recognition. The required dataset for this feature is an audio dataset. This dataset contains the pronounciation of 5 words which are "Adik", "Ayah", "Ibu", "Kakak", and "Keluarga" which was collected by our team members. There are 200 audio files (in wav extension) that used in the training process.

First thing to do is labeling each dataset from 0 to 4 which represent each word respectively. After that we need to extract a feature from audio file called Mel-Frequency Ceptral Coefficients (MFCCs) and then begin to build the model. The model used is a deep learning model containing 2 convolution layers and 1 connected layer. The resulting model is pretty good due to the fact that the average accuracy sits approximately at 90%.

The cloud computing team is responsible on designing the API for the frontend, a.k.a android application. All of the CC codes are available in the backend folder. We use Node.js and Express.js as the framework. The main code of this can bee seen in app.js file. The routes and handlers can be seen in backend/routes and backend/handlers folders respectively.

The cloud computing team is also responsible for the machine learning deployments. With the help of tensorflowjs, we were able to do this. In Latihan Mengeja Huruf feature, the android application sends the recording in a .m4a format and we were able to catch the file sent using multer. Then, using ffmpeg we were able to convert this file into .wav format. Then, we run a python script -- can be seen in backend/utils/index.py to collect the MFCC matric from the audio file. Finally, we flatten this matrix into an array then make it into a Tensor and insert it to the model that is available in backend/models.

The cloud computing team also design the infrastructure of the cloud using Google Cloud Platform. The initial plan, was to use Regional TCP Load Balancers. Hence, we updated our code and make a new route that is '/' that acts the health checks done by the load balancer. This route is to ensure that the app is healthy and able to send request to the backend services.

Currently, below is the diagram of the design of our GCP's services.

Finally, on time efficiency when making a new compute instance, we made a shell script such that it automatically runs when a new VM is created for Managed Instance Group. The shell script can be seen in backend/startup.sh. Lastly, we use pm2 -- a daemon process manager that will help you manage and keep your application online -- to ensure that our app is running! To do this we install pm2 globally with npm (available in the startup script) and then run app.js with pm2 by doing pm2 start app.js.

We currently have 5 routes, where 4 of them requires firebase authentication token since these will be used in the app. Meanwhile, the '/' as mentioned is for the health checks.

app.js
/* 
@ ROUTES
*/
app.use('/users', middlewareFirebase.decodeToken, userRoute);
app.use('/achievements', middlewareFirebase.decodeToken, achievementRoute);
app.use('/predictions', middlewareFirebase.decodeToken, predictionRoute);
app.use('/rimakatawords', middlewareFirebase.decodeToken, rimakataRoute);
app.use('/', healthRoute);

The code for middlewareFirebase is located in backend/middleware folder. We designed this to be a class. The routes that has GET method are '/users' and '/rimakatawords'. Meanwhile, the route '/achievements' has PUT method. Lastly, the '/predictions' has POST method.

Without further due, let's see the successes result with these routes.

{
    "status":"success",
    "message":"Query user data successfully done!",
    "data": {
        "nama_user":"rifky.satyana08",
        "achv_id":1,
        "user_id":2,
        "eksplor_huruf":["a","b","c"],
        "eksplor_angka":["dua","nol","satu","tujuh"],
        "latMenyusunKatalvl1":5,
        "latMenyusunKatalvl2":1,
        "latMenyusunKatalvl3":0,
        "latMengejaHuruflvl1":["a","n","u"],
        "latMengejaHuruflvl2":["b","c"],
        "latMengejaHuruflvl3":["r","z"]
    }
}

{
    "status": "success",
    "message": "Here are the data!",
    "data": {
        "mudah": [
            {
                "lema": "hela",
                "nilai": "menghela",
                "url": "NA"
            },
            {
                "lema": "gaun",
                "nilai": "baju wanita model Eropa",
                "url": "NA"
            },
            {
                "lema": "onak",
                "nilai": "rotan yang berduri",
                "url": "NA"
            },
            {
                "lema": "biji",
                "nilai": "isi buah (yang apabila ditanam dapat tumbuh)",
                "url": "NA"
            }
        ],
        "sedang": [
            {
                "lema": "serah",
                "nilai": "berserah",
                "url": "NA"
            },
            {
                "lema": "lurah",
                "nilai": "kepala pemerintahan tingkat terendah",
                "url": "NA"
            },
            {
                "lema": "daerah",
                "nilai": "bagian permukaan bumi dalam kaitannya dengan keadaan alam dsb yang khusus",
                "url": "NA"
            },
            {
                "lema": "merah",
                "nilai": "warna dasar yang serupa dengan warna darah",
                "url": "NA"
            }
        ],
        "sulit": [
            {
                "lema": "musyawarah",
                "nilai": "pembahasan bersama dengan maksud mencapai keputusan atas penyelesaian masalah",
                "url": "NA"
            },
            {
                "lema": "anugerah",
                "nilai": "pemberian atau ganjaran dari pihak atas (orang besar dsb) kepada pihak bawah (orang rendah dsb)",
                "url": "NA"
            },
            {
                "lema": "prasejarah",
                "nilai": "bagian ilmu sejarah tentang zaman ketika manusia hidup dalam kebudayaan yang belum mengenal tulisan",
                "url": "NA"
            },
            {
                "lema": "antardaerah",
                "nilai": "antara daerah yang satu dan yang lain",
                "url": "NA"
            }
        ]
    }
}

{
    "status": "success",
    "message": "User's achievements on eksplor huruf successfully updated!"
}

If the user successfully pronounce the alphabet

{
    "status": "success",
    "message": "We have succesfully predict your recording. User's achievements on latihan mengeja huruf successfully updated!",
    "result": 1,
    "updated": true
}

If the user unsuccessfully pronounce the alphabet

{
    "status": "success",
    "message": "We have successfully predict the recording! See you're result.",
    "result": 0
}

We also generate errors such as bad request in a constructive way so that the frontend can handle it the way it should.

Firebase Errors

{
    "status":"fail",
    "type":"auth/id-token-expired",
    "message":"Firebase ID token has expired. Get a fresh ID token from your client app and try again (auth/id-token-expired). See https://firebase.google.com/docs/auth/admin/verify-id-tokens for details on how to retrieve an ID token."
}

File uploaded is not supported by the server.

{
    "status":"fail",
    "type":"server/file-not-supported",
    "message":"We only receive audio file type. You're file type was: video"
}

There are many more error handling in the code. We'll leave for you readers to check it out in the codes!

Now let's take a quick logging at our loggings in Cloud Logging.

As you can see, we monitor the response through Cloud Logging which is made available with Winston and @google-cloud/logging-winston.

The documentation for mobile development team is available at MD Documentation.