This project is created by Kozmos Software Team for Nasa Space Apps Challenge 2023.

NASA offers a variety of “sonifications” – translations of 2D astronomical data into sound –that provide a new way to experience imagery and other information from space. Advanced instruments currently provide hyperspectral (many color) images from space that are 3D (two spatial dimensions and one color dimension), and sophisticated techniques can be used to enhance 2D astronomical images to make video representations called “fly-throughs” that allow viewers to experience what it would look like to move among space objects in 3D (three simulated spatial dimensions). Your challenge is to design a method to create sonifications of these 3D NASA space datasets to provide a different perceptual path that can help us understand and appreciate the wonders of the universe!

The goal is to convert three-dimensional images obtained from NASA's state-of-the-art satellites and telescopes, which consist of two-dimensional and color components, into an artistic musical composition that can be comprehended by visually impaired individuals. This is achieved by developing an Artificial Intelligence-based model to process these complex three-dimensional space images acquired through NASA's advanced technologies into an auditory representation. The approach involves analyzing various parameters, such as the proximity, velocity, and colors of asteroids, stars, and galaxies within the video footage. The resulting data is then associated with specific sound clusters, creating an artistic rendition that enables us to appreciate the wonders of space.

In our devised method, Artificial Intelligence is employed to process these images, facilitating the understanding of the emotional responses these images would evoke in a human observer. Subsequently, to convey these emotions to visually impaired individuals in the most precise and emotionally resonant manner, the aforementioned variables are integrated into our methodology. Moreover, the perception of depth and motion, which is a significant distinction between three-dimensional and two-dimensional imagery, plays a pivotal role in our approach. To enhance this perception, certain functions within the method are reiterated at predetermined intervals. When utilizing this method, it is recommended that users employ high-fidelity technology and headphones to better discern the depth perception.

1. https://www.spaceappschallenge.org/
2. https://www.science.org/content/article/meet-scientist-who-turns-data-music-and-listen-sound-neutron-star
3. https://en.wikipedia.org/wiki/Data_sonification
4. https://keras.io/
5. https://codelabs.developers.google.com/codelabs/cloud-video-intelligence-python3#4