alycialee / beyond-scale-language-data-diversity Goto Github PK

one idea for an experiment:

1. control for alignment e.g., for data sets with roughly same alignment (two figures, 1 low 1 high alignment)
2. then plot CE eval on the y-axis and div coeff on the x-axis. Do we see models trained on mode diverse data sets perform better?

This is mainly to back up our claim that "quality == diversity if target == general data set"

I saw in multiple places

## LLM DIV

comment e.g.,

## LLM DIV
def plot_distance_matrix_heatmap_only(embeddings, labels=None, distance='cosine', show_plot=True, title=None, save_file=None):
    import seaborn as sns
    import pandas as pd
    import matplotlib.pyplot as plt
    distance_matrix = pdist(embeddings, distance=distance)
    if labels is not None:
        distance_matrix = pd.DataFrame(distance_matrix, index=labels, columns=labels)
    sns.heatmap(distance_matrix, cmap='viridis_r')
    if title:
        plt.title(title)
    if save_file:
        _ = plt.savefig("plots/" + save_file + ".png", bbox_inches='tight')
    if show_plot:
        plt.show()

but the input to the function is just a list of embeddings. Numbering for ease of addressing:

1. Is there a reason we didn't simply pip install task2vec and use their code for code that wasn't specific to LLMs?
2. When we do modify it to deal with LLMs, how does it differ from Task2Vec's original code?

Thanks in advance @alycialee

https://github.com/alycialee/beyond-scale-language-data-diversity/tree/main/diversity#quick-start

What is the approximate time taken to run the scripts diversity/runner.sh.
In my GPU server set-up, it takes more than 10 hrs.

Is there any other way for this