This is very interesting work! Sadly, right now the code is extremely complicated and

simple implementation about based HOT 2 CLOSED

AlexiaJM commented on May 26, 2024

simple implementation

from based.

Comments (2)

seyuboglu commented on May 26, 2024

Thanks for the feedback --

In order to create a model with the code in this repo, you can use code that looks something like:

from based.models.gpt import GPT2MixerConfig, GPTLMHeadModel

max_length = 2048

config=GPT2MixerConfig(
    mixer={
      "_target_": "based.models.mixers.slide_attention.SlidingAttention",
      "window_size": 128 ,
      "num_heads": 16
    },
    alt_mixer={
      "_target_": "based.models.mixers.linear_attention.LinearAttention",
      "l_max": 2048,
      "feature_map": {
        "_target_": "based.models.mixers.linear_attention.TaylorExp",
        "input_dim": 16,
      },
      "feature_dim": 16,
      "num_heads": 16,
      "parallel_implementation": "linear"
    },
    alt_mixer_layers=[1,3,7,9],
    n_positions=max_length,
)

model = GPTLMHeadModel(config=config)

We don't currently have plans to release a step-by-step training script. However, the training code in this repo https://github.com/HazyResearch/zoology, which we use for our synthetic language model exerimentsk, is a good bit simpler than the training code in this repo.

from based.

AlexiaJM commented on May 26, 2024

Thanks! That's helpful.

from based.

Recommend Projects