This project focuses on building a data processing module using Python, incorporating advanced features like generators, comprehensions, and object-oriented programming. The goal is to extract and analyze year-over-year revenue and revenue by region from Bamazon Inc.'s sales data spanning 2015 to 2021.

• Mastering Python's advanced data structures and functionalities.
• Efficiently handling and parsing large data files.
• Implementing robust testing strategies with pytest.

• Explored the use of generators for memory-efficient data processing.
• Learned the significance of the __iter__ method in creating iterable objects.

• Utilized list comprehensions to simplify data manipulation and filtering tasks.

• Applied OOP principles to structure the data processing module, enhancing code modularity and reusability.

• Adopted pytest for writing and executing test cases, ensuring code reliability.

• Investigated techniques for processing large datasets without compromising performance.

• Encountered issues with Python module discovery.
• Resolved by correctly setting the PYTHONPATH environment variable.

• Faced difficulties executing the main script due to directory structure.
• Overcame by adjusting the command line invocation to specify the correct module path.

• Identified and fixed syntax errors that were causing script failures.

Successfully developed and tested a Python module capable of:

• Generating insights from large datasets.
• Aggregating sales data to calculate revenue metrics.
• Producing reports including bar graphs and JSON files for data visualization.

• To optimize Python code using multiprocessing, focusing on reducing execution time for data processing tasks.

1. Understanding Multiprocessing: Grasped the basic concepts of parallel processing and Python's multiprocessing module.
2. Code Optimization: Successfully refactored Week 1's single-threaded code to a parallelized version using multiprocessing.
3. Performance Measurement: Measured and compared execution times to quantify performance improvements.

1. Global Interpreter Lock (GIL) Limitations: Overcame Python's GIL restrictions by employing multiprocessing to execute tasks in parallel.
2. Dynamic Process Allocation: Implemented dynamic process allocation based on the system's available CPU cores to optimize performance.

• Technical Precision: Achieved a significant reduction in execution time by parallelizing data processing tasks.
• Educational Growth: Enhanced understanding of multiprocessing and its application for optimizing Python code.
• Communication Clarity: Demonstrated clear and effective communication through inquiries and code modifications.

• Fine-tuning Multiprocessing: Explore the impact of chunk sizes and the difference between process vs. thread pools.
• Understanding Parallelism Patterns: Investigate various parallel computing patterns for structuring computations efficiently.

• The journey through optimizing Python code with multiprocessing has enriched both technical skills and understanding of parallel computing principles. There's a demonstrated ability to effectively apply multiprocessing for performance optimization, showing significant progress in mastering Python for high-performance applications.

1. Explore "High Performance Python" by Micha Gorelick and Ian Ozsvald to deepen knowledge on optimizing Python code.
2. Enroll in Advanced Python Courses focusing on performance optimization and parallel computing on platforms like Coursera or Udemy.
3. Engage in Real-world Projects or contribute to open-source projects involving data processing or scientific computing to apply multiprocessing knowledge practically.

• Make you're using Python version >= 3.9.0
• Install all the modules

pip install -r requirements.txt

Generate test data (useful for unit testing code)

python generate_data.py --type tst

Generate small data (useful for quick testing of logic)

python generate_data.py --type sml

Generate big data (actual data)

python generate_data.py --type bg

PYTHONPATH=../ pytest test.py -s

PYTHONPATH=./w1 pytest w1/test.py -s

PYTHONPATH=../ pytest test.py

PYTHONPATH=./w1 pytest w1/test.py

PYTHONPATH=../ python main.py --type tst

PYTHONPATH=./w1 python -m w1.main --type tst

PYTHONPATH=../ python main.py --type sml

PYTHONPATH=./w1 python -m w1.main --type sml

PYTHONPATH=../ python main.py --type bg

PYTHONPATH=./w1 python -m w1.main --type bg

PYTHONPATH=../ pytest test.py -s

PYTHONPATH=./w2 pytest w2/test.py -s

PYTHONPATH=../ pytest test.py

PYTHONPATH=./w2 pytest w2/test.py

PYTHONPATH=../ python main.py --type tst

PYTHONPATH=./w2 python -m w2.main --type tst

PYTHONPATH=../ python main.py --type sml

PYTHONPATH=./w2 python -m w2.main --type sml

PYTHONPATH=../ python main.py --type bg

PYTHONPATH=./w2 python -m w2.main --type bg