Generate and load more samples to the project. Currently, there are only stringed instrument samples. Use many different VSTs for this.

Create the beginning scene, linked to one of the buttons of startscreen.
Here, the program will guide you through how to navigate through different scenes and user interfaces, focusing especially on shooting rays and pointing.
This issues focuses on the initial set up of the scene by creating objects and the order of the tutorial.

This issue is to study the literature on score-following so that I can next try to implement the theories.

Make the Start Screen scene more visually appealing. It should be streamlined and cool.

Soohwan Park | Personal Goals

what does success for this class mean to you?

• I am very comfortable with Unity and C# to the level where I can dive into VR game development afterwards
• I get Hand Tracking working because that's one of the newest experimental features on Oculus that will be widely used from here on out
• I know what a user-friendly UI looks like in VR after seeing users smoothly run through the full extent of the environment
• I know how to manipulate music in VR

Challenges

what are some challenges you foresee?

• Potential merge conflicts hindering our progress
• Getting behind on where we want to be and not having a solid trajectory

Right now, each component of the backstage acts independently from the rest of the game. This task is to have the music selection, instrument selection, and back-to-front stage doorknob integrate with the game.

There are several features that support user's immersive experience as he's playing along with Ensemble. There are basic motion features that allow the user to move around the stage. These actions are incorporated within the 'pause' menu that allow users to 'restart' or 'move backstage'. This pause itself is triggered by another button that is easily visible to the user.

• Have the music playing in the background that pauses when the user triggers it --> "replay" + "start" buttons pop up --> follow their functions

This issue is to create and sync the audience clapping sounds with the start and end of the performance.

Connect settings scene with start scene. This should be done through a button. Also connect settings to the front stage such that the settings changes are reflected in front stage.

Several instruments have been roughly modeled. This issue focuses on refining those instruments (Guitar) so that they are ready to be staged in a week, specifically the strings of the guitar.

Bryan Shin | Personal Goals

what does success for this class mean to you?

• I am able to fully understand and implement one of the many score following/tempo tracking algorithms described in online research papers
• I am able to show Ensemble to musician peers, and see that this application not only works but is relevant and desirable.
• I release this application on the VR market for the masses to download and try out.

Challenges

what are some challenges you foresee?

• A lot of the research papers online use difficult technical jargon and concepts, which I am unfamiliar with.
• This application may be difficult to release officially because it is targeted at such a specific audience of soloist musicians.
• It will be hard to user-test our application because not many people have access to VR headsets. It will be even harder to find musician peers with VR headsets.

This Epic is for the tutorial scene that teaches a user the basic steps to navigating the application.

Work on a new design and layout for Start Screen UI. Right now, it is not very aesthetically pleasing.

Currently, the game only supports hand interactions in the form of touching objects with the fingertips. However, often times, objects are out of reach of the player's game area, so we need a way to interact with objects from a distance. This involves ray projections and hand-gesture recognition to understand when a user wants to interact with the object and when he/she does not.

From Start Interface, to setting, End Interface, this epic is responsible for guiding the user through the program from start to end.

During Technigala, we found that the user interaction with the backstage was unintuitive. This issue is to figure out a way to make it more seamless and easy for the user.

Ryan Hyun | Personal Goals

what does success for this class mean to you?

• I contribute to a meaningful product that can enhance other people's quality of life.
• I learn how to use new technologies and become better at managing my work.
• I become more industrious and able to handle unfamiliarity and stress in a technical or personal context.

Challenges

what are some challenges you foresee?

• My lack of experience with many technologies could mean lots of learning on the fly.
• I may want to bring the project in directions that may contrast with the views of other members of the team.
• In a time crunch, I tend to have trouble getting things done, so I need to finish my work on time and be efficient.

This issue is to continue refining the animations for the orchestra members and the models.

Create sound files for necessary instruments such as strings, winds, etc. Upload these to Unity as assets for use in note playback.

In order to provide realistic and immersive experience to musicians, it is critical to have reliable 3D models. This sprint focuses on finishing designing the 3D guitar model. This also includes having a firm grasp of the basics of Maya.

Finish the flow of tutorial from beginning to end, including flow to the startscreen.
Design does not need be perfect.

Design a scene that will handle settings for the front stage. This scene should handle things like audience size, orchestra size, etc.

Currently, the backstage environment consists of a basic wall with brick texturing, one table, and one door. However, a real backstage is much more complex. This task is to add features such as signs, props, other people, and general texture details to make a more realistic backstage environment.

Improve user interface and user experience by making the users "know" that they are interacting with objects. (i.e. sound when you pinch, knowing where to pinch)

Set up real-time oculus integration so that when I add edit a script or add an object, I can see the changes in real time. This will be useful for all the upcoming issues, especially hand gesture detection.

This issue is to animate the orchestra members such that they play along to the music soundtrack.

Refine on bugs, the flow, UX, UI, design, everything so that the scene is ready to export.

Once the user is done playing, several buttons pop up to allow the users to play again or choose another music, or instrument. This is similar to the main stage UI, but detect that the music is over.

This issue is to see if it is viable to animate character models using Unity's rigging tools, rather than using external software like Blender to animate the characters. If possible, this could streamline the syncing of the animations to the music.

This issue is to use the BPM value obtained from the tempo tracking script and translate it into a speed multiplier that can be applied directly to a .wav or .mp3 file.

Currently, the backstage scene functionality is completed. A user is able to select a piece, select an instrument, move around in the scene, and move on to the front stage. However, the implementation looks clunky and unappealing. This issue is to create a more realistic backstage environment with more detail, to be more immersive.

To help with merge conflicts, this issue is to split up each of our scenes into multiple portions such that more than 1 person can work on a room at once. For example, front stage could be split up into the front stage models scene and the front stage UI scene.

Refine tutorial design, including the background.
Not a stage in the middle of nowhere, but on a field.

This issue is to continue studying the tempo-tracking literature to find a paper with a tested algorithm, and then figure out how to best implement the algorithm

This issue is to start creating a tangible, workable, prototype for a tempo-tracking algorithm

Establish a set of variables that hold throughout the different scenes in the project. These should keep track of the number of audience and orchestra models desired in the front stage, the type of score-following model to be used, etc.

The current instrument models are crude and overly simplified. This task is to add more faces and vertices to the mesh and more accurate textures to more accurately mimic the instruments.

Building on top of the initial set up and objects, this issues involves adding scripts that handle muting, pausing of the scene, focusing on helping user adjust to the interface.

First, familiarize yourself to the basics of MIDI, wav, and how Unity handles music. Then, import sample classical music files to the main stage scene in Unity. This starts happening after the backstage. If the music is over, ending UI pops up to tell the user that he can either replay or choose new music.

Write a script to change the tempo of an audio source in Unity without affecting pitch. This should prove useful if we decide to adapt a tempo adjustment approach in our accompaniment.

In order to provide realistic and immersive experience to musicians, it is critical to have reliable 3D models. This sprint focuses on building the scaffolding for the 3D guitar model, focusing on the guitar body. This also includes learning the basics of Maya.

Write and test a script that plays a tone given a key, duration, and instrument. It should be fairly simple to lessen the load on the Oculus Quest. Make use of the audio components in Unity.

See if the library using ZeroMQ is suitable for the purposes of this project. https://github.com/off99555/Unity3D-Python-Communication

Write a script that holds a collection of note playback objects dedicated to specific instruments and calls those objects when prompted to do so. This functions as the controller for accompaniment playback.

Finish building the tutorial. Even if it's not fully complete, focus on hand gesture detection (pause) and ray shooting as well as pinching.

This issue is to animate and sync the movement of the instruments with the orchestra member models.

This issue is for an alternative approach to the interactable music. Since score-following is a very complex subject, implementing a simpler tempo-tracking algorithm may be more reasonable.

This is an example of a chair in which the virtual audience sits. This has to be set up correctly to portray an accurate representation of how a real audience is seated in a performance.

Currently, the backstage scene functionality is completed. A user is able to select a piece, select an instrument, move around in the scene, and move on to the front stage. However, the interactions feel clunky and unintuitive. This issue is to touch up the interactions with the music book, instruments, movement tiles, etc to make a more immersive and intuitive interaction experience

This issue is so that we can use an external program to calculate tempo/score tracking, and feed the information to the Oculus, rather than doing calculations on the Oculus itself.