Gradio

loha/llama-3.2-3B-rank32

LOHA

2591

2341

0.4184988627748294

0.6569721374511719

23886561280

19247870771

13446820344

73429560

2025-06-19 16:12:05+00:00

MetaMathQA

meta-llama/Llama-3.2-3B

{'model_id': 'meta-llama/Llama-3.2-3B', 'dtype': 'bfloat16', 'max_seq_length': 768, 'batch_size': 4, 'batch_size_eval': 50, 'max_steps': 5000, 'eval_steps': 250, 'compile': False, 'query_template': 'Question: {query} Think step by step.\nAnswer:', 'seed': 0, 'grad_norm_clip': 1.0, 'optimizer_type': 'AdamW', 'optimizer_kwargs': {'lr': 0.0001, 'weight_decay': 0.1}, 'lr_scheduler': 'cosine', 'use_amp': False, 'autocast_adapter_dtype': True, 'generation_kwargs': {'max_length': 800, 'max_new_tokens': 300}, 'attn_implementation': None}

{'task_type': None, 'peft_type': 'LOHA', 'auto_mapping': None, 'base_model_name_or_path': 'meta-llama/Llama-3.2-3B', 'revision': None, 'inference_mode': False, 'rank_pattern': {}, 'alpha_pattern': {}, 'r': 32, 'alpha': 64, 'rank_dropout': 0.0, 'module_dropout': 0.0, 'use_effective_conv2d': False, 'target_modules': ['q_proj', 'v_proj'], 'exclude_modules': None, 'init_weights': True, 'layers_to_transform': None, 'layers_pattern': None, 'modules_to_save': None}

20000

4198051

0.15.2.dev0

main

4.52.4

3.6.0

2.7.1+cu126

0.46.0

{'transformers-version': '4.52.4', 'transformers-commit-hash': None, 'peft-version': '0.15.2.dev0', 'peft-commit-hash': '5fe7f8f8abe914d313fc3751f2ea92de7718fbaf', 'datasets-version': '3.6.0', 'datasets-commit-hash': None, 'bitsandbytes-version': '0.46.0', 'bitsandbytes-commit-hash': None, 'torch-version': '2.7.1+cu126', 'torch-commit-hash': None}

{'system': 'Linux', 'release': '6.8.0-1029-aws', 'version': '#31-Ubuntu SMP Wed Apr 23 18:42:41 UTC 2025', 'machine': 'x86_64', 'processor': 'x86_64', 'accelerator': 'NVIDIA L40S'}

lora/llama-3.2-3B-rank64

LORA

2017

1853

0.4890068233510235

0.5928755496740341

22540189696

17953927987

12128055669

73415408

2025-06-19 16:55:20+00:00

MetaMathQA

meta-llama/Llama-3.2-3B

{'model_id': 'meta-llama/Llama-3.2-3B', 'dtype': 'bfloat16', 'max_seq_length': 768, 'batch_size': 4, 'batch_size_eval': 50, 'max_steps': 5000, 'eval_steps': 250, 'compile': False, 'query_template': 'Question: {query} Think step by step.\nAnswer:', 'seed': 0, 'grad_norm_clip': 1.0, 'optimizer_type': 'AdamW', 'optimizer_kwargs': {'lr': 0.0001, 'weight_decay': 0.1}, 'lr_scheduler': 'cosine', 'use_amp': False, 'autocast_adapter_dtype': True, 'generation_kwargs': {'max_length': 800, 'max_new_tokens': 300}, 'attn_implementation': None}

{'task_type': 'CAUSAL_LM', 'peft_type': 'LORA', 'auto_mapping': None, 'base_model_name_or_path': 'meta-llama/Llama-3.2-3B', 'revision': None, 'inference_mode': False, 'r': 64, 'target_modules': ['q_proj', 'v_proj'], 'exclude_modules': None, 'lora_alpha': 128, 'lora_dropout': 0.0, 'fan_in_fan_out': False, 'bias': 'none', 'use_rslora': False, 'modules_to_save': None, 'init_lora_weights': True, 'layers_to_transform': None, 'layers_pattern': None, 'rank_pattern': {}, 'alpha_pattern': {}, 'megatron_config': None, 'megatron_core': 'megatron.core', 'trainable_token_indices': None, 'loftq_config': {}, 'eva_config': None, 'corda_config': None, 'use_dora': False, 'layer_replication': None, 'lora_bias': False}

20000

4198051

0.15.2.dev0

main

4.52.4

3.6.0

2.7.1+cu126

0.46.0

{'transformers-version': '4.52.4', 'transformers-commit-hash': None, 'peft-version': '0.15.2.dev0', 'peft-commit-hash': '5fe7f8f8abe914d313fc3751f2ea92de7718fbaf', 'datasets-version': '3.6.0', 'datasets-commit-hash': None, 'bitsandbytes-version': '0.46.0', 'bitsandbytes-commit-hash': None, 'torch-version': '2.7.1+cu126', 'torch-commit-hash': None}

{'system': 'Linux', 'release': '6.8.0-1029-aws', 'version': '#31-Ubuntu SMP Wed Apr 23 18:42:41 UTC 2025', 'machine': 'x86_64', 'processor': 'x86_64', 'accelerator': 'NVIDIA L40S'}

lora/llama-3.2-3B-rank32-lorafa

LORA

2026

1821

0.42987111448066717

0.6509636770486832

20187185152

16257394933

11106307276

36715216

2025-06-19 17:29:01+00:00

MetaMathQA

meta-llama/Llama-3.2-3B

{'model_id': 'meta-llama/Llama-3.2-3B', 'dtype': 'bfloat16', 'max_seq_length': 768, 'batch_size': 4, 'batch_size_eval': 50, 'max_steps': 5000, 'eval_steps': 250, 'compile': False, 'query_template': 'Question: {query} Think step by step.\nAnswer:', 'seed': 0, 'grad_norm_clip': 1.0, 'optimizer_type': 'lora-fa', 'optimizer_kwargs': {'r': 32, 'lora_alpha': 64, 'lr': 0.0001, 'weight_decay': 0.1}, 'lr_scheduler': 'cosine', 'use_amp': False, 'autocast_adapter_dtype': True, 'generation_kwargs': {'max_length': 800, 'max_new_tokens': 300}, 'attn_implementation': None}

{'task_type': 'CAUSAL_LM', 'peft_type': 'LORA', 'auto_mapping': None, 'base_model_name_or_path': 'meta-llama/Llama-3.2-3B', 'revision': None, 'inference_mode': False, 'r': 32, 'target_modules': ['q_proj', 'v_proj'], 'exclude_modules': None, 'lora_alpha': 64, 'lora_dropout': 0.0, 'fan_in_fan_out': False, 'bias': 'none', 'use_rslora': False, 'modules_to_save': None, 'init_lora_weights': True, 'layers_to_transform': None, 'layers_pattern': None, 'rank_pattern': {}, 'alpha_pattern': {}, 'megatron_config': None, 'megatron_core': 'megatron.core', 'trainable_token_indices': None, 'loftq_config': {}, 'eva_config': None, 'corda_config': None, 'use_dora': False, 'layer_replication': None, 'lora_bias': False}

20000

4198051

0.15.2.dev0

main

4.52.4

3.6.0

2.7.1+cu126

0.46.0

{'transformers-version': '4.52.4', 'transformers-commit-hash': None, 'peft-version': '0.15.2.dev0', 'peft-commit-hash': '5fe7f8f8abe914d313fc3751f2ea92de7718fbaf', 'datasets-version': '3.6.0', 'datasets-commit-hash': None, 'bitsandbytes-version': '0.46.0', 'bitsandbytes-commit-hash': None, 'torch-version': '2.7.1+cu126', 'torch-commit-hash': None}

{'system': 'Linux', 'release': '6.8.0-1029-aws', 'version': '#31-Ubuntu SMP Wed Apr 23 18:42:41 UTC 2025', 'machine': 'x86_64', 'processor': 'x86_64', 'accelerator': 'NVIDIA L40S'}

lora/llama-3.2-3B-rank64-rslora

LORA

2069

1871

0.5299469294920395

0.5657225311994553

22538092544

17953927987

12128059444

73415408

2025-06-19 18:02:52+00:00

MetaMathQA

meta-llama/Llama-3.2-3B

{'model_id': 'meta-llama/Llama-3.2-3B', 'dtype': 'bfloat16', 'max_seq_length': 768, 'batch_size': 4, 'batch_size_eval': 50, 'max_steps': 5000, 'eval_steps': 250, 'compile': False, 'query_template': 'Question: {query} Think step by step.\nAnswer:', 'seed': 0, 'grad_norm_clip': 1.0, 'optimizer_type': 'AdamW', 'optimizer_kwargs': {'lr': 0.0001, 'weight_decay': 0.1}, 'lr_scheduler': 'cosine', 'use_amp': False, 'autocast_adapter_dtype': True, 'generation_kwargs': {'max_length': 800, 'max_new_tokens': 300}, 'attn_implementation': None}

{'task_type': 'CAUSAL_LM', 'peft_type': 'LORA', 'auto_mapping': None, 'base_model_name_or_path': 'meta-llama/Llama-3.2-3B', 'revision': None, 'inference_mode': False, 'r': 64, 'target_modules': ['v_proj', 'q_proj'], 'exclude_modules': None, 'lora_alpha': 64, 'lora_dropout': 0.0, 'fan_in_fan_out': False, 'bias': 'none', 'use_rslora': True, 'modules_to_save': None, 'init_lora_weights': True, 'layers_to_transform': None, 'layers_pattern': None, 'rank_pattern': {}, 'alpha_pattern': {}, 'megatron_config': None, 'megatron_core': 'megatron.core', 'trainable_token_indices': None, 'loftq_config': {}, 'eva_config': None, 'corda_config': None, 'use_dora': False, 'layer_replication': None, 'lora_bias': False}

20000

4198051

0.15.2.dev0

main

4.52.4

3.6.0

2.7.1+cu126

0.46.0

{'transformers-version': '4.52.4', 'transformers-commit-hash': None, 'peft-version': '0.15.2.dev0', 'peft-commit-hash': '5fe7f8f8abe914d313fc3751f2ea92de7718fbaf', 'datasets-version': '3.6.0', 'datasets-commit-hash': None, 'bitsandbytes-version': '0.46.0', 'bitsandbytes-commit-hash': None, 'torch-version': '2.7.1+cu126', 'torch-commit-hash': None}

{'system': 'Linux', 'release': '6.8.0-1029-aws', 'version': '#31-Ubuntu SMP Wed Apr 23 18:42:41 UTC 2025', 'machine': 'x86_64', 'processor': 'x86_64', 'accelerator': 'NVIDIA L40S'}

lora/llama-3.2-3B-rank32-dora

LORA

2287

2023

0.4806671721000758

0.6068210340738297

24553455616

19189150515

12490471636

37181760

2025-06-19 18:37:24+00:00

MetaMathQA

meta-llama/Llama-3.2-3B

{'model_id': 'meta-llama/Llama-3.2-3B', 'dtype': 'bfloat16', 'max_seq_length': 768, 'batch_size': 4, 'batch_size_eval': 50, 'max_steps': 5000, 'eval_steps': 250, 'compile': False, 'query_template': 'Question: {query} Think step by step.\nAnswer:', 'seed': 0, 'grad_norm_clip': 1.0, 'optimizer_type': 'AdamW', 'optimizer_kwargs': {'lr': 0.0001, 'weight_decay': 0.1}, 'lr_scheduler': 'cosine', 'use_amp': False, 'autocast_adapter_dtype': True, 'generation_kwargs': {'max_length': 800, 'max_new_tokens': 300}, 'attn_implementation': None}

{'task_type': 'CAUSAL_LM', 'peft_type': 'LORA', 'auto_mapping': None, 'base_model_name_or_path': 'meta-llama/Llama-3.2-3B', 'revision': None, 'inference_mode': False, 'r': 32, 'target_modules': ['q_proj', 'v_proj'], 'exclude_modules': None, 'lora_alpha': 64, 'lora_dropout': 0.0, 'fan_in_fan_out': False, 'bias': 'none', 'use_rslora': False, 'modules_to_save': None, 'init_lora_weights': True, 'layers_to_transform': None, 'layers_pattern': None, 'rank_pattern': {}, 'alpha_pattern': {}, 'megatron_config': None, 'megatron_core': 'megatron.core', 'trainable_token_indices': None, 'loftq_config': {}, 'eva_config': None, 'corda_config': None, 'use_dora': True, 'layer_replication': None, 'lora_bias': False}

20000

4198051

0.15.2.dev0

main

4.52.4

3.6.0

2.7.1+cu126

0.46.0

{'transformers-version': '4.52.4', 'transformers-commit-hash': None, 'peft-version': '0.15.2.dev0', 'peft-commit-hash': '5fe7f8f8abe914d313fc3751f2ea92de7718fbaf', 'datasets-version': '3.6.0', 'datasets-commit-hash': None, 'bitsandbytes-version': '0.46.0', 'bitsandbytes-commit-hash': None, 'torch-version': '2.7.1+cu126', 'torch-commit-hash': None}

{'system': 'Linux', 'release': '6.8.0-1029-aws', 'version': '#31-Ubuntu SMP Wed Apr 23 18:42:41 UTC 2025', 'machine': 'x86_64', 'processor': 'x86_64', 'accelerator': 'NVIDIA L40S'}

lora/llama-3.2-3B-rank32

LORA

1993

1796

0.48218347232752085

0.6069052599668503

22273851392

17710763212

11868689976

36715216

2025-06-19 19:15:35+00:00

MetaMathQA

meta-llama/Llama-3.2-3B

{'model_id': 'meta-llama/Llama-3.2-3B', 'dtype': 'bfloat16', 'max_seq_length': 768, 'batch_size': 4, 'batch_size_eval': 50, 'max_steps': 5000, 'eval_steps': 250, 'compile': False, 'query_template': 'Question: {query} Think step by step.\nAnswer:', 'seed': 0, 'grad_norm_clip': 1.0, 'optimizer_type': 'AdamW', 'optimizer_kwargs': {'lr': 0.0001, 'weight_decay': 0.1}, 'lr_scheduler': 'cosine', 'use_amp': False, 'autocast_adapter_dtype': True, 'generation_kwargs': {'max_length': 800, 'max_new_tokens': 300}, 'attn_implementation': None}

{'task_type': 'CAUSAL_LM', 'peft_type': 'LORA', 'auto_mapping': None, 'base_model_name_or_path': 'meta-llama/Llama-3.2-3B', 'revision': None, 'inference_mode': False, 'r': 32, 'target_modules': ['v_proj', 'q_proj'], 'exclude_modules': None, 'lora_alpha': 64, 'lora_dropout': 0.0, 'fan_in_fan_out': False, 'bias': 'none', 'use_rslora': False, 'modules_to_save': None, 'init_lora_weights': True, 'layers_to_transform': None, 'layers_pattern': None, 'rank_pattern': {}, 'alpha_pattern': {}, 'megatron_config': None, 'megatron_core': 'megatron.core', 'trainable_token_indices': None, 'loftq_config': {}, 'eva_config': None, 'corda_config': None, 'use_dora': False, 'layer_replication': None, 'lora_bias': False}

20000

4198051

0.15.2.dev0

main

4.52.4

3.6.0

2.7.1+cu126

0.46.0

{'transformers-version': '4.52.4', 'transformers-commit-hash': None, 'peft-version': '0.15.2.dev0', 'peft-commit-hash': '5fe7f8f8abe914d313fc3751f2ea92de7718fbaf', 'datasets-version': '3.6.0', 'datasets-commit-hash': None, 'bitsandbytes-version': '0.46.0', 'bitsandbytes-commit-hash': None, 'torch-version': '2.7.1+cu126', 'torch-commit-hash': None}

{'system': 'Linux', 'release': '6.8.0-1029-aws', 'version': '#31-Ubuntu SMP Wed Apr 23 18:42:41 UTC 2025', 'machine': 'x86_64', 'processor': 'x86_64', 'accelerator': 'NVIDIA L40S'}

ptuning/llama-3.2-3B-default

P_TUNING

1918

1707

0.3707354056103108

0.6739867876768112

20937965568

17215688540

11867101593

245880

2025-06-19 19:48:53+00:00

MetaMathQA

meta-llama/Llama-3.2-3B

{'model_id': 'meta-llama/Llama-3.2-3B', 'dtype': 'bfloat16', 'max_seq_length': 768, 'batch_size': 4, 'batch_size_eval': 50, 'max_steps': 5000, 'eval_steps': 250, 'compile': False, 'query_template': 'Question: {query} Think step by step.\nAnswer:', 'seed': 0, 'grad_norm_clip': 1.0, 'optimizer_type': 'AdamW', 'optimizer_kwargs': {'lr': 0.0001, 'weight_decay': 0.1}, 'lr_scheduler': 'cosine', 'use_amp': False, 'autocast_adapter_dtype': True, 'generation_kwargs': {'max_length': 800, 'max_new_tokens': 300}, 'attn_implementation': None}

{'task_type': 'CAUSAL_LM', 'peft_type': 'P_TUNING', 'auto_mapping': None, 'base_model_name_or_path': 'meta-llama/Llama-3.2-3B', 'revision': None, 'inference_mode': False, 'num_virtual_tokens': 20, 'token_dim': 3072, 'num_transformer_submodules': 1, 'num_attention_heads': 24, 'num_layers': 28, 'encoder_reparameterization_type': 'MLP', 'encoder_hidden_size': 3072, 'encoder_num_layers': 2, 'encoder_dropout': 0.0}

20000

4198051

0.15.2.dev0

main

4.52.4

3.6.0

2.7.1+cu126

0.46.0

{'transformers-version': '4.52.4', 'transformers-commit-hash': None, 'peft-version': '0.15.2.dev0', 'peft-commit-hash': '5fe7f8f8abe914d313fc3751f2ea92de7718fbaf', 'datasets-version': '3.6.0', 'datasets-commit-hash': None, 'bitsandbytes-version': '0.46.0', 'bitsandbytes-commit-hash': None, 'torch-version': '2.7.1+cu126', 'torch-commit-hash': None}

{'system': 'Linux', 'release': '6.8.0-1029-aws', 'version': '#31-Ubuntu SMP Wed Apr 23 18:42:41 UTC 2025', 'machine': 'x86_64', 'processor': 'x86_64', 'accelerator': 'NVIDIA L40S'}

prefixtuning/llama-3.2-3B-lr_0.001

PREFIX_TUNING

1959

1662

0.1470811220621683

0.7887116296291351

20912799744

16945051074

11766684083

45875328

2025-06-19 20:20:55+00:00

MetaMathQA

meta-llama/Llama-3.2-3B

{'model_id': 'meta-llama/Llama-3.2-3B', 'dtype': 'bfloat16', 'max_seq_length': 768, 'batch_size': 4, 'batch_size_eval': 50, 'max_steps': 5000, 'eval_steps': 250, 'compile': False, 'query_template': 'Question: {query} Think step by step.\nAnswer:', 'seed': 0, 'grad_norm_clip': 1.0, 'optimizer_type': 'AdamW', 'optimizer_kwargs': {'lr': 0.001}, 'lr_scheduler': 'cosine', 'use_amp': False, 'autocast_adapter_dtype': True, 'generation_kwargs': {'max_length': 800, 'max_new_tokens': 300}, 'attn_implementation': None}

{'task_type': 'CAUSAL_LM', 'peft_type': 'PREFIX_TUNING', 'auto_mapping': None, 'base_model_name_or_path': 'meta-llama/Llama-3.2-3B', 'revision': None, 'inference_mode': False, 'num_virtual_tokens': 200, 'token_dim': 1024, 'num_transformer_submodules': 1, 'num_attention_heads': 8, 'num_layers': 28, 'encoder_hidden_size': 3072, 'prefix_projection': False}

20000

4198051

0.15.2.dev0

main

4.52.4

3.6.0

2.7.1+cu126

0.46.0

{'transformers-version': '4.52.4', 'transformers-commit-hash': None, 'peft-version': '0.15.2.dev0', 'peft-commit-hash': '5fe7f8f8abe914d313fc3751f2ea92de7718fbaf', 'datasets-version': '3.6.0', 'datasets-commit-hash': None, 'bitsandbytes-version': '0.46.0', 'bitsandbytes-commit-hash': None, 'torch-version': '2.7.1+cu126', 'torch-commit-hash': None}

{'system': 'Linux', 'release': '6.8.0-1029-aws', 'version': '#31-Ubuntu SMP Wed Apr 23 18:42:41 UTC 2025', 'machine': 'x86_64', 'processor': 'x86_64', 'accelerator': 'NVIDIA L40S'}

vera/llama-3.2-3B-default

VERA

2025

1820

0.3684609552691433

0.6926896897554398

21596471296

17291123097

11489715316

6821968

2025-06-19 20:53:39+00:00

MetaMathQA

meta-llama/Llama-3.2-3B

{'model_id': 'meta-llama/Llama-3.2-3B', 'dtype': 'bfloat16', 'max_seq_length': 768, 'batch_size': 4, 'batch_size_eval': 50, 'max_steps': 5000, 'eval_steps': 250, 'compile': False, 'query_template': 'Question: {query} Think step by step.\nAnswer:', 'seed': 0, 'grad_norm_clip': 1.0, 'optimizer_type': 'AdamW', 'optimizer_kwargs': {'lr': 0.001}, 'lr_scheduler': 'cosine', 'use_amp': False, 'autocast_adapter_dtype': True, 'generation_kwargs': {'max_length': 800, 'max_new_tokens': 300}, 'attn_implementation': None}

{'task_type': None, 'peft_type': 'VERA', 'auto_mapping': None, 'base_model_name_or_path': 'meta-llama/Llama-3.2-3B', 'revision': None, 'inference_mode': False, 'r': 256, 'target_modules': ['v_proj', 'q_proj'], 'projection_prng_key': 0, 'save_projection': True, 'vera_dropout': 0.0, 'd_initial': 0.1, 'fan_in_fan_out': False, 'bias': 'none', 'modules_to_save': None, 'init_weights': True, 'layers_to_transform': None, 'layers_pattern': None}

20000

4198051

0.15.2.dev0

main

4.52.4

3.6.0

2.7.1+cu126

0.46.0

{'transformers-version': '4.52.4', 'transformers-commit-hash': None, 'peft-version': '0.15.2.dev0', 'peft-commit-hash': '5fe7f8f8abe914d313fc3751f2ea92de7718fbaf', 'datasets-version': '3.6.0', 'datasets-commit-hash': None, 'bitsandbytes-version': '0.46.0', 'bitsandbytes-commit-hash': None, 'torch-version': '2.7.1+cu126', 'torch-commit-hash': None}

{'system': 'Linux', 'release': '6.8.0-1029-aws', 'version': '#31-Ubuntu SMP Wed Apr 23 18:42:41 UTC 2025', 'machine': 'x86_64', 'processor': 'x86_64', 'accelerator': 'NVIDIA L40S'}

ia3/llama-3.2-3B-lr_0.001

IA3

1922

1746

0.41243366186504926

0.6568749620914459

23135780864

18398356439

12023331867

1157064

2025-06-19 21:27:27+00:00

MetaMathQA

meta-llama/Llama-3.2-3B

{'model_id': 'meta-llama/Llama-3.2-3B', 'dtype': 'bfloat16', 'max_seq_length': 768, 'batch_size': 4, 'batch_size_eval': 50, 'max_steps': 5000, 'eval_steps': 250, 'compile': False, 'query_template': 'Question: {query} Think step by step.\nAnswer:', 'seed': 0, 'grad_norm_clip': 1.0, 'optimizer_type': 'AdamW', 'optimizer_kwargs': {'lr': 0.001}, 'lr_scheduler': 'cosine', 'use_amp': False, 'autocast_adapter_dtype': True, 'generation_kwargs': {'max_length': 800, 'max_new_tokens': 300}, 'attn_implementation': None}

{'task_type': None, 'peft_type': 'IA3', 'auto_mapping': None, 'base_model_name_or_path': 'meta-llama/Llama-3.2-3B', 'revision': None, 'inference_mode': False, 'target_modules': ['k_proj', 'down_proj', 'v_proj'], 'exclude_modules': None, 'feedforward_modules': ['down_proj'], 'fan_in_fan_out': False, 'modules_to_save': None, 'init_ia3_weights': True}

20000

4198051

0.15.2.dev0

main

4.52.4

3.6.0

2.7.1+cu126

0.46.0

{'transformers-version': '4.52.4', 'transformers-commit-hash': None, 'peft-version': '0.15.2.dev0', 'peft-commit-hash': '5fe7f8f8abe914d313fc3751f2ea92de7718fbaf', 'datasets-version': '3.6.0', 'datasets-commit-hash': None, 'bitsandbytes-version': '0.46.0', 'bitsandbytes-commit-hash': None, 'torch-version': '2.7.1+cu126', 'torch-commit-hash': None}

{'system': 'Linux', 'release': '6.8.0-1029-aws', 'version': '#31-Ubuntu SMP Wed Apr 23 18:42:41 UTC 2025', 'machine': 'x86_64', 'processor': 'x86_64', 'accelerator': 'NVIDIA L40S'}

ia3/llama-3.2-3B-default

IA3

2005

1783

0.34495830174374525

0.7657463653087616

23137878016

18398566154

12023227429

1157064

2025-06-19 21:59:33+00:00

MetaMathQA

meta-llama/Llama-3.2-3B

{'model_id': 'meta-llama/Llama-3.2-3B', 'dtype': 'bfloat16', 'max_seq_length': 768, 'batch_size': 4, 'batch_size_eval': 50, 'max_steps': 5000, 'eval_steps': 250, 'compile': False, 'query_template': 'Question: {query} Think step by step.\nAnswer:', 'seed': 0, 'grad_norm_clip': 1.0, 'optimizer_type': 'AdamW', 'optimizer_kwargs': {'lr': 0.0001, 'weight_decay': 0.1}, 'lr_scheduler': 'cosine', 'use_amp': False, 'autocast_adapter_dtype': True, 'generation_kwargs': {'max_length': 800, 'max_new_tokens': 300}, 'attn_implementation': None}

{'task_type': None, 'peft_type': 'IA3', 'auto_mapping': None, 'base_model_name_or_path': 'meta-llama/Llama-3.2-3B', 'revision': None, 'inference_mode': False, 'target_modules': ['down_proj', 'v_proj', 'k_proj'], 'exclude_modules': None, 'feedforward_modules': ['down_proj'], 'fan_in_fan_out': False, 'modules_to_save': None, 'init_ia3_weights': True}

20000

4198051

0.15.2.dev0

main

4.52.4

3.6.0

2.7.1+cu126

0.46.0

{'transformers-version': '4.52.4', 'transformers-commit-hash': None, 'peft-version': '0.15.2.dev0', 'peft-commit-hash': '5fe7f8f8abe914d313fc3751f2ea92de7718fbaf', 'datasets-version': '3.6.0', 'datasets-commit-hash': None, 'bitsandbytes-version': '0.46.0', 'bitsandbytes-commit-hash': None, 'torch-version': '2.7.1+cu126', 'torch-commit-hash': None}

{'system': 'Linux', 'release': '6.8.0-1029-aws', 'version': '#31-Ubuntu SMP Wed Apr 23 18:42:41 UTC 2025', 'machine': 'x86_64', 'processor': 'x86_64', 'accelerator': 'NVIDIA L40S'}

lokr/llama-3.2-3B-rank32

LOKR

2352

2152

0.3752843062926459

0.6876721383333206

23565697024

18987698094

13173683073

1131984

2025-06-19 22:33:02+00:00

MetaMathQA

meta-llama/Llama-3.2-3B

{'model_id': 'meta-llama/Llama-3.2-3B', 'dtype': 'bfloat16', 'max_seq_length': 768, 'batch_size': 4, 'batch_size_eval': 50, 'max_steps': 5000, 'eval_steps': 250, 'compile': False, 'query_template': 'Question: {query} Think step by step.\nAnswer:', 'seed': 0, 'grad_norm_clip': 1.0, 'optimizer_type': 'AdamW', 'optimizer_kwargs': {'lr': 0.0001, 'weight_decay': 0.1}, 'lr_scheduler': 'cosine', 'use_amp': False, 'autocast_adapter_dtype': True, 'generation_kwargs': {'max_length': 800, 'max_new_tokens': 300}, 'attn_implementation': None}

{'task_type': None, 'peft_type': 'LOKR', 'auto_mapping': None, 'base_model_name_or_path': 'meta-llama/Llama-3.2-3B', 'revision': None, 'inference_mode': False, 'rank_pattern': {}, 'alpha_pattern': {}, 'r': 32, 'alpha': 64, 'rank_dropout': 0.0, 'module_dropout': 0.0, 'use_effective_conv2d': False, 'decompose_both': False, 'decompose_factor': -1, 'rank_dropout_scale': False, 'target_modules': ['q_proj', 'v_proj'], 'exclude_modules': None, 'init_weights': True, 'layers_to_transform': None, 'layers_pattern': None, 'modules_to_save': None}

20000

4198051

0.15.2.dev0

main

4.52.4

3.6.0

2.7.1+cu126

0.46.0

{'transformers-version': '4.52.4', 'transformers-commit-hash': None, 'peft-version': '0.15.2.dev0', 'peft-commit-hash': '5fe7f8f8abe914d313fc3751f2ea92de7718fbaf', 'datasets-version': '3.6.0', 'datasets-commit-hash': None, 'bitsandbytes-version': '0.46.0', 'bitsandbytes-commit-hash': None, 'torch-version': '2.7.1+cu126', 'torch-commit-hash': None}

{'system': 'Linux', 'release': '6.8.0-1029-aws', 'version': '#31-Ubuntu SMP Wed Apr 23 18:42:41 UTC 2025', 'machine': 'x86_64', 'processor': 'x86_64', 'accelerator': 'NVIDIA L40S'}

adalora/llama-3.2-3B-rank32

ADALORA

2209

1986

0.3904473085670963

0.6862971596717834

22793945088

18203426160

12361399900

35147440

2025-06-19 23:12:19+00:00

MetaMathQA

meta-llama/Llama-3.2-3B

{'model_id': 'meta-llama/Llama-3.2-3B', 'dtype': 'bfloat16', 'max_seq_length': 768, 'batch_size': 4, 'batch_size_eval': 50, 'max_steps': 5000, 'eval_steps': 250, 'compile': False, 'query_template': 'Question: {query} Think step by step.\nAnswer:', 'seed': 0, 'grad_norm_clip': 1.0, 'optimizer_type': 'AdamW', 'optimizer_kwargs': {'lr': 0.0001, 'weight_decay': 0.1}, 'lr_scheduler': 'cosine', 'use_amp': False, 'autocast_adapter_dtype': True, 'generation_kwargs': {'max_length': 800, 'max_new_tokens': 300}, 'attn_implementation': None}

{'task_type': None, 'peft_type': 'ADALORA', 'auto_mapping': None, 'base_model_name_or_path': 'meta-llama/Llama-3.2-3B', 'revision': None, 'inference_mode': False, 'r': 8, 'target_modules': ['q_proj', 'v_proj'], 'exclude_modules': None, 'lora_alpha': 8, 'lora_dropout': 0.0, 'fan_in_fan_out': False, 'bias': 'none', 'use_rslora': False, 'modules_to_save': None, 'init_lora_weights': True, 'layers_to_transform': None, 'layers_pattern': None, 'rank_pattern': {'model.layers.0.self_attn.q_proj.lora_E': [False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False], 'model.layers.0.self_attn.v_proj.lora_E': [True, True, True, True, True, True, True, True, True, False, True, True, True, True, True, False, True, True, True, False, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, False, True, True, True, True, True, True, True, True, True, True, True], 'model.layers.1.self_attn.q_proj.lora_E': [False, False, True, True, False, True, True, False, False, False, False, True, False, False, True, True, True, True, False, False, False, False, False, False, True, False, True, True, False, False, True, True, True, False, True, True, False, False, True, True, True, False, False, False, True, False, True, True, True, True, False, True, True, True, False, False, True, True, False, False, True, True, False, False], 'model.layers.1.self_attn.v_proj.lora_E': [True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, False, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, False, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True], 'model.layers.2.self_attn.q_proj.lora_E': [True, False, True, False, False, False, True, True, True, True, False, True, True, True, False, False, True, False, False, True, False, False, False, False, True, True, False, False, False, False, True, False, False, False, False, False, False, True, True, True, False, False, False, True, True, False, False, False, True, False, True, True, False, True, False, False, False, True, True, False, True, True, False, False], 'model.layers.2.self_attn.v_proj.lora_E': [True, False, False, False, True, True, True, True, False, True, True, True, False, True, False, True, False, True, False, True, False, True, True, True, True, True, False, True, False, False, False, False, True, False, False, False, False, False, True, True, False, False, True, True, False, True, True, True, True, False, False, True, False, True, False, False, False, True, True, False, False, False, True, True], 'model.layers.3.self_attn.q_proj.lora_E': [False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False], 'model.layers.3.self_attn.v_proj.lora_E': [False, False, False, False, False, True, False, True, False, False, False, False, True, False, False, False, False, True, True, True, True, False, True, False, False, False, False, False, False, False, False, False, True, False, False, True, False, False, True, False, True, False, True, True, False, True, False, False, True, False, False, False, False, False, True, False, True, False, False, False, False, True, True, True], 'model.layers.4.self_attn.q_proj.lora_E': [False, False, False, False, False, True, False, False, True, True, False, True, False, False, False, False, False, False, False, True, False, False, True, False, True, False, False, False, False, False, True, False, False, False, False, False, False, False, True, True, False, False, True, False, False, False, True, True, False, False, False, False, False, False, False, False, False, False, False, False, True, True, False, False], 'model.layers.4.self_attn.v_proj.lora_E': [True, False, True, True, False, False, True, False, False, False, True, False, True, True, False, True, False, True, True, False, True, True, False, False, True, True, True, True, False, False, False, False, False, False, True, False, True, False, False, True, True, True, True, True, False, False, False, False, False, True, False, True, True, True, True, True, False, True, True, False, True, True, True, True], 'model.layers.5.self_attn.q_proj.lora_E': [False, True, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False], 'model.layers.5.self_attn.v_proj.lora_E': [True, True, True, True, True, True, True, True, False, True, False, False, True, False, False, True, False, True, False, False, False, False, True, True, False, False, False, False, True, False, True, False, True, True, False, False, True, True, True, True, False, False, True, False, True, False, False, True, True, True, False, True, False, False, False, True, True, True, True, False, False, False, True, True], 'model.layers.6.self_attn.q_proj.lora_E': [False, False, True, True, False, False, True, True, False, False, False, True, False, True, False, True, False, False, False, False, True, True, True, True, False, True, False, True, False, True, False, False, False, True, True, False, False, False, False, True, True, True, False, False, True, False, False, False, False, True, True, False, False, False, True, False, False, False, False, False, False, False, False, False], 'model.layers.6.self_attn.v_proj.lora_E': [False, True, True, True, True, True, True, True, True, True, False, True, True, True, True, True, False, True, True, True, False, False, False, False, True, True, False, False, False, False, True, True, False, True, True, True, False, True, True, True, False, True, True, True, True, False, False, False, True, True, False, False, True, False, True, False, True, True, False, True, False, True, False, True], 'model.layers.7.self_attn.q_proj.lora_E': [False, False, False, False, False, True, False, False, False, False, False, False, False, False, True, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, True, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, True, False, False, False, False, False, False, False, False, False, False, False, False, False], 'model.layers.7.self_attn.v_proj.lora_E': [False, False, True, True, False, True, True, True, True, False, True, True, True, True, True, True, True, True, True, True, False, False, True, True, True, True, True, False, True, False, False, True, True, True, True, False, False, False, True, False, False, True, True, True, False, True, True, True, True, True, True, False, True, True, True, True, True, False, False, False, True, True, True, True], 'model.layers.8.self_attn.q_proj.lora_E': [False, True, False, False, False, False, False, True, False, False, False, False, True, True, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, True, False, False, False, False, True, False, False, True, False, False, False, False, True, False, False, False, False, False, True, False, False, False, False, False, False, False, False, False, False, True], 'model.layers.8.self_attn.v_proj.lora_E': [False, True, False, False, False, True, False, False, False, False, True, True, True, True, True, False, False, True, True, True, False, True, True, True, True, True, False, True, True, False, False, True, True, False, False, True, False, True, False, True, True, False, True, False, True, True, True, False, True, False, False, True, True, True, False, True, True, True, True, False, False, False, False, True], 'model.layers.9.self_attn.q_proj.lora_E': [True, False, True, True, False, False, True, True, False, False, True, False, False, False, False, True, False, True, False, True, False, False, False, True, False, True, False, True, False, True, False, True, True, False, False, True, True, False, False, False, False, True, True, True, False, False, False, False, True, True, True, False, False, False, False, False, True, False, True, False, False, True, False, True], 'model.layers.9.self_attn.v_proj.lora_E': [True, True, False, True, True, True, True, True, True, False, True, True, True, True, True, False, True, True, True, True, True, True, True, True, False, True, True, True, True, True, True, True, False, True, True, False, True, True, True, True, True, True, False, False, True, False, True, False, True, True, True, True, True, True, True, True, True, True, True, True, True, True, False, False], 'model.layers.10.self_attn.q_proj.lora_E': [False, False, False, False, False, False, False, False, False, False, True, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, True, False, False, False, False, False, False, False, False, False, False, True, False, False, False, False, False, False, False, False, False, False, False, False, False], 'model.layers.10.self_attn.v_proj.lora_E': [True, True, False, False, False, True, True, False, False, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, False, False, False, False, True, True, True, True, False, False, False, False, True, False, False, False, True, True, True, False, True, True, False, True, False, False, True, True, False, False, True, True, True, True, True, True, True, False, True], 'model.layers.11.self_attn.q_proj.lora_E': [True, False, False, False, False, True, False, False, False, True, True, False, True, False, False, False, False, False, False, False, False, True, False, False, True, False, False, False, False, False, False, False, False, False, False, False, True, False, False, False, True, True, True, False, True, False, False, False, True, True, False, False, False, False, True, True, False, True, False, True, True, False, False, False], 'model.layers.11.self_attn.v_proj.lora_E': [False, True, True, True, False, True, True, False, True, False, True, True, True, True, True, True, True, False, False, True, False, True, True, True, True, False, False, True, True, True, True, False, True, True, True, True, True, True, True, True, True, False, False, False, False, False, True, False, False, True, True, False, False, True, True, True, True, True, False, True, True, True, False, False], 'model.layers.12.self_attn.q_proj.lora_E': [False, False, False, False, False, False, False, False, False, False, False, False, False, True, False, True, False, False, False, False, False, True, False, False, False, False, False, False, True, False, False, False, False, False, False, False, False, True, True, False, True, False, False, False, False, True, False, True, False, False, True, False, True, False, False, True, False, False, False, False, False, False, False, False], 'model.layers.12.self_attn.v_proj.lora_E': [True, True, True, True, False, True, True, False, True, True, True, True, True, True, True, True, True, False, False, False, False, True, True, False, False, True, True, True, True, True, True, True, True, True, True, False, False, True, False, True, True, True, True, True, False, False, True, False, True, True, True, True, False, True, False, True, False, True, False, True, False, True, True, False], 'model.layers.13.self_attn.q_proj.lora_E': [True, True, False, True, True, True, False, False, True, True, False, True, False, True, False, True, False, False, True, True, False, True, False, True, True, True, True, False, False, True, True, False, False, True, False, True, False, True, True, True, False, False, False, False, True, True, True, True, False, True, False, True, True, True, False, True, False, True, True, False, False, False, True, False], 'model.layers.13.self_attn.v_proj.lora_E': [True, False, True, True, True, False, True, True, True, False, True, True, True, False, True, False, True, False, True, True, True, True, True, True, True, True, True, False, True, True, False, False, True, True, False, False, True, False, False, True, False, False, True, True, True, True, True, True, True, False, True, False, False, True, True, True, False, True, True, False, True, True, True, True], 'model.layers.14.self_attn.q_proj.lora_E': [False, True, False, True, True, False, False, False, True, False, False, True, False, False, True, True, False, True, True, True, False, False, False, True, False, True, False, True, False, False, True, True, True, True, True, False, False, True, True, False, True, True, False, False, True, False, False, False, True, False, True, True, True, False, True, True, True, False, False, True, False, True, True, False], 'model.layers.14.self_attn.v_proj.lora_E': [True, True, True, False, False, False, True, False, False, False, False, True, True, False, False, True, False, True, True, True, False, True, False, False, True, False, True, False, True, True, False, True, False, True, False, False, True, False, False, True, False, True, True, False, True, False, True, False, True, True, True, True, True, True, False, False, True, True, False, True, True, True, True, False], 'model.layers.15.self_attn.q_proj.lora_E': [False, True, True, True, True, True, False, True, False, True, False, True, False, True, True, True, True, True, True, False, True, True, False, True, False, True, False, True, True, True, False, True, False, False, False, True, True, True, True, False, True, True, False, True, False, True, False, False, True, True, False, True, False, True, False, True, True, True, True, True, False, True, True, True], 'model.layers.15.self_attn.v_proj.lora_E': [True, True, True, True, False, True, True, True, True, False, True, True, False, True, True, False, False, True, False, True, True, True, True, True, False, True, True, True, True, True, False, False, False, True, True, True, True, True, True, True, True, True, True, False, True, True, True, True, True, False, True, True, True, True, False, False, False, True, True, True, True, True, True, True], 'model.layers.16.self_attn.q_proj.lora_E': [False, False, False, False, False, False, False, True, True, True, True, False, True, False, True, True, True, False, True, False, True, True, True, False, False, False, True, False, False, False, True, False, True, True, True, True, False, True, True, False, True, True, False, True, True, True, True, False, True, True, False, True, True, False, True, False, False, True, True, True, False, False, True, True], 'model.layers.16.self_attn.v_proj.lora_E': [True, False, True, False, True, False, True, False, False, True, False, True, True, True, True, True, True, False, True, False, True, True, True, False, True, True, False, False, False, True, True, True, False, True, False, False, True, False, False, False, True, False, True, False, True, True, True, False, True, True, False, False, True, True, True, False, True, True, False, True, False, True, False, False], 'model.layers.17.self_attn.q_proj.lora_E': [True, True, True, True, False, True, False, True, False, False, True, True, True, False, True, False, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, False, False, True, True, False, True, True, True, True, False, True, True, False, True, True, True, False, True, False, True, True, True, True, False, True, True, True, True, True, True, True, True], 'model.layers.17.self_attn.v_proj.lora_E': [False, True, True, True, True, True, True, True, True, True, False, True, True, False, False, False, True, True, False, True, True, False, False, True, True, False, False, False, False, True, True, True, False, True, False, True, False, True, True, True, True, False, True, True, True, True, True, True, False, True, True, True, True, True, True, True, True, True, False, False, True, True, True, True], 'model.layers.18.self_attn.q_proj.lora_E': [False, True, False, True, False, True, False, True, True, True, False, True, True, True, False, True, True, False, True, False, False, False, True, True, False, True, True, True, False, True, True, True, True, True, True, False, True, True, True, True, True, True, False, False, True, True, True, True, True, False, True, False, False, False, False, True, False, False, True, False, True, False, True, True], 'model.layers.18.self_attn.v_proj.lora_E': [True, True, True, True, True, True, True, True, False, True, True, False, False, False, True, False, True, True, True, True, True, True, True, True, True, True, True, True, True, False, True, True, True, False, True, True, True, True, True, True, True, True, False, True, True, False, True, True, True, True, True, True, True, True, False, True, True, True, True, True, True, False, False, True], 'model.layers.19.self_attn.q_proj.lora_E': [False, True, False, True, False, True, False, True, True, True, True, True, True, True, True, True, True, True, True, True, False, True, True, True, True, True, True, False, False, True, True, True, True, False, True, False, True, False, True, False, False, False, True, True, True, True, True, False, False, False, False, True, True, True, True, True, True, False, False, True, True, False, True, True], 'model.layers.19.self_attn.v_proj.lora_E': [False, True, False, True, True, True, False, True, False, False, False, False, True, True, True, False, True, False, False, False, False, True, False, True, True, False, True, True, False, False, True, True, True, True, False, False, False, False, False, True, False, False, True, False, False, False, True, True, False, False, False, True, True, True, True, False, True, True, False, True, False, True, True, True], 'model.layers.20.self_attn.q_proj.lora_E': [False, True, False, False, False, False, True, False, False, False, False, False, False, False, False, True, False, False, False, False, False, False, False, False, False, False, False, True, False, False, False, False, False, True, False, True, True, False, False, False, False, True, False, True, False, True, False, False, False, False, True, True, False, False, True, True, False, False, False, False, False, True, False, False], 'model.layers.20.self_attn.v_proj.lora_E': [True, False, True, True, False, False, False, True, True, False, False, True, True, True, False, True, False, True, False, False, False, False, True, False, False, False, True, False, True, False, True, True, True, False, True, False, True, False, True, False, True, True, True, True, False, False, False, False, False, False, False, False, True, False, False, False, True, False, False, True, False, False, True, True], 'model.layers.21.self_attn.q_proj.lora_E': [False, False, True, False, True, True, True, True, True, False, True, True, True, True, True, True, False, False, False, False, True, True, False, True, True, True, True, False, False, False, False, False, False, False, True, True, True, False, True, False, True, False, True, False, False, False, True, False, True, True, True, True, True, False, False, True, True, False, True, True, False, False, True, True], 'model.layers.21.self_attn.v_proj.lora_E': [True, True, True, True, True, False, True, True, False, True, True, False, False, True, True, False, True, True, False, True, True, True, True, False, False, False, False, False, True, True, True, False, True, True, False, True, True, False, True, True, False, False, False, True, True, False, False, False, True, True, False, True, True, False, False, False, False, True, False, False, False, False, False, False], 'model.layers.22.self_attn.q_proj.lora_E': [False, True, False, True, True, False, True, False, False, True, False, False, False, False, False, False, True, False, True, True, False, False, False, False, True, True, True, False, False, True, False, False, False, False, False, True, False, False, False, False, True, False, False, False, True, False, True, False, False, False, True, False, True, True, True, False, False, True, False, False, True, True, False, True], 'model.layers.22.self_attn.v_proj.lora_E': [False, True, True, True, True, False, False, True, True, True, True, False, True, True, False, True, True, True, True, True, True, True, True, True, False, False, True, True, False, True, False, False, True, False, False, False, False, True, False, False, False, True, True, False, False, False, True, True, False, False, True, True, True, True, True, True, True, True, False, False, True, False, True, True], 'model.layers.23.self_attn.q_proj.lora_E': [True, False, True, False, True, True, True, True, True, False, True, True, True, True, True, True, True, True, False, False, True, True, True, True, True, False, True, True, True, True, True, True, True, True, True, True, True, True, True, True, False, True, True, False, True, False, True, False, True, True, True, True, True, True, True, False, True, False, True, True, True, False, True, True], 'model.layers.23.self_attn.v_proj.lora_E': [False, True, False, True, False, False, True, False, True, False, True, True, True, True, True, True, False, True, True, True, True, True, True, True, True, True, True, True, True, False, False, True, False, True, True, False, True, False, True, True, True, True, True, True, True, True, True, True, True, True, False, True, True, True, False, False, False, True, True, False, True, False, True, True], 'model.layers.24.self_attn.q_proj.lora_E': [True, True, True, True, True, True, False, True, True, True, True, False, True, True, True, False, True, True, True, True, True, True, False, False, True, True, True, True, False, True, True, True, True, True, True, True, True, True, False, True, True, True, False, True, True, True, True, False, True, False, True, True, True, True, False, False, False, True, True, True, True, False, False, True], 'model.layers.24.self_attn.v_proj.lora_E': [True, True, True, False, True, False, False, True, True, True, False, True, True, False, False, True, False, False, False, False, True, True, True, False, True, False, False, True, False, True, False, True, True, False, True, True, False, False, False, True, False, False, True, True, False, True, True, False, False, True, True, True, True, False, False, True, True, True, False, True, False, True, True, True], 'model.layers.25.self_attn.q_proj.lora_E': [False, False, False, False, True, True, False, True, True, False, False, False, False, False, False, False, True, False, False, True, False, False, True, False, False, False, True, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, True, False, False, False, False, False, True, False, False, False, True, True, True, False, False, False, False, False, False, False], 'model.layers.25.self_attn.v_proj.lora_E': [False, False, False, True, False, False, False, True, True, False, False, True, False, True, True, True, False, False, False, False, True, False, False, False, True, True, True, True, False, False, False, False, True, False, False, False, False, True, False, True, False, False, False, False, False, False, False, False, False, False, False, False, False, False, True, False, False, False, False, False, False, False, False, False], 'model.layers.26.self_attn.q_proj.lora_E': [True, False, False, True, False, False, False, False, False, False, True, False, True, False, True, True, True, False, False, True, True, True, False, False, True, True, False, False, True, False, True, True, False, False, False, True, True, False, False, False, True, False, False, False, True, True, False, False, True, False, True, True, False, True, False, False, True, True, True, False, True, True, True, True], 'model.layers.26.self_attn.v_proj.lora_E': [False, False, True, False, True, False, False, False, True, False, False, False, False, False, True, False, False, False, False, False, False, False, True, False, False, True, False, True, False, True, True, False, False, False, False, False, False, False, False, False, True, False, False, False, False, False, False, False, True, True, False, False, False, True, False, True, False, True, False, False, False, True, False, False], 'model.layers.27.self_attn.q_proj.lora_E': [True, False, False, True, True, False, False, True, True, False, False, False, True, True, False, True, False, False, True, False, False, True, True, True, True, False, False, True, True, False, False, False, False, True, True, True, False, True, False, False, False, True, False, True, True, True, False, False, False, True, True, True, True, True, False, False, False, False, True, False, False, False, True, False], 'model.layers.27.self_attn.v_proj.lora_E': [False, False, True, True, True, True, True, True, True, False, False, False, True, False, False, False, True, True, False, False, False, True, False, True, True, True, True, True, False, True, True, False, True, False, True, True, False, True, True, False, False, True, False, True, True, False, False, True, False, True, True, True, False, False, True, False, False, True, True, True, True, True, False, True]}, 'alpha_pattern': {}, 'megatron_config': None, 'megatron_core': 'megatron.core', 'trainable_token_indices': None, 'loftq_config': {}, 'eva_config': None, 'corda_config': None, 'use_dora': False, 'layer_replication': None, 'lora_bias': False, 'target_r': 32, 'init_r': 64, 'tinit': 200, 'tfinal': 500, 'deltaT': 1, 'beta1': 0.85, 'beta2': 0.85, 'orth_reg_weight': 0.5, 'total_step': 5000}

20000

4198051

0.15.2.dev0

main

4.52.4

3.6.0

2.7.1+cu126

0.46.0

{'transformers-version': '4.52.4', 'transformers-commit-hash': None, 'peft-version': '0.15.2.dev0', 'peft-commit-hash': '5fe7f8f8abe914d313fc3751f2ea92de7718fbaf', 'datasets-version': '3.6.0', 'datasets-commit-hash': None, 'bitsandbytes-version': '0.46.0', 'bitsandbytes-commit-hash': None, 'torch-version': '2.7.1+cu126', 'torch-commit-hash': None}

{'system': 'Linux', 'release': '6.8.0-1029-aws', 'version': '#31-Ubuntu SMP Wed Apr 23 18:42:41 UTC 2025', 'machine': 'x86_64', 'processor': 'x86_64', 'accelerator': 'NVIDIA L40S'}

vblora/llama-3.2-3B-default

VBLORA

2210

1962

0.36997725549658833

0.7142883945703506

22181576704

17635223797

11735344663

4864912

2025-06-19 23:49:12+00:00

MetaMathQA

meta-llama/Llama-3.2-3B

{'model_id': 'meta-llama/Llama-3.2-3B', 'dtype': 'bfloat16', 'max_seq_length': 768, 'batch_size': 4, 'batch_size_eval': 50, 'max_steps': 5000, 'eval_steps': 250, 'compile': False, 'query_template': 'Question: {query} Think step by step.\nAnswer:', 'seed': 0, 'grad_norm_clip': 1.0, 'optimizer_type': 'AdamW', 'optimizer_kwargs': {'lr': 0.0001, 'weight_decay': 0.1}, 'lr_scheduler': 'cosine', 'use_amp': False, 'autocast_adapter_dtype': True, 'generation_kwargs': {'max_length': 800, 'max_new_tokens': 300}, 'attn_implementation': None}

{'task_type': None, 'peft_type': 'VBLORA', 'auto_mapping': None, 'base_model_name_or_path': 'meta-llama/Llama-3.2-3B', 'revision': None, 'inference_mode': False, 'r': 4, 'num_vectors': 256, 'vector_length': 256, 'topk': 2, 'target_modules': ['v_proj', 'q_proj'], 'exclude_modules': None, 'save_only_topk_weights': False, 'vblora_dropout': 0.0, 'fan_in_fan_out': False, 'bias': 'none', 'modules_to_save': None, 'init_vector_bank_bound': 0.02, 'init_logits_std': 0.1, 'layers_to_transform': None, 'layers_pattern': None}

20000

4198051

0.15.2.dev0

main

4.52.4

3.6.0

2.7.1+cu126

0.46.0

{'transformers-version': '4.52.4', 'transformers-commit-hash': None, 'peft-version': '0.15.2.dev0', 'peft-commit-hash': '5fe7f8f8abe914d313fc3751f2ea92de7718fbaf', 'datasets-version': '3.6.0', 'datasets-commit-hash': None, 'bitsandbytes-version': '0.46.0', 'bitsandbytes-commit-hash': None, 'torch-version': '2.7.1+cu126', 'torch-commit-hash': None}

{'system': 'Linux', 'release': '6.8.0-1029-aws', 'version': '#31-Ubuntu SMP Wed Apr 23 18:42:41 UTC 2025', 'machine': 'x86_64', 'processor': 'x86_64', 'accelerator': 'NVIDIA L40S'}

boft/llama-3.2-3B-default

BOFT

11114

8292

0.3646702047005307

0.7268091850280761

24427626496

20103445872

14814855089

3225360

2025-06-20 00:26:06+00:00

MetaMathQA

meta-llama/Llama-3.2-3B

{'model_id': 'meta-llama/Llama-3.2-3B', 'dtype': 'bfloat16', 'max_seq_length': 768, 'batch_size': 4, 'batch_size_eval': 50, 'max_steps': 5000, 'eval_steps': 250, 'compile': False, 'query_template': 'Question: {query} Think step by step.\nAnswer:', 'seed': 0, 'grad_norm_clip': 1.0, 'optimizer_type': 'AdamW', 'optimizer_kwargs': {'lr': 0.0001, 'weight_decay': 0.1}, 'lr_scheduler': 'cosine', 'use_amp': False, 'autocast_adapter_dtype': True, 'generation_kwargs': {'max_length': 800, 'max_new_tokens': 300}, 'attn_implementation': None}

{'task_type': None, 'peft_type': 'BOFT', 'auto_mapping': None, 'base_model_name_or_path': 'meta-llama/Llama-3.2-3B', 'revision': None, 'inference_mode': False, 'boft_block_size': 4, 'boft_block_num': 0, 'boft_n_butterfly_factor': 1, 'target_modules': ['q_proj', 'v_proj'], 'exclude_modules': None, 'boft_dropout': 0.0, 'fan_in_fan_out': False, 'bias': 'none', 'modules_to_save': None, 'init_weights': True, 'layers_to_transform': None, 'layers_pattern': None}

20000

4198051

0.15.2.dev0

main

4.52.4

3.6.0

2.7.1+cu126

0.46.0

{'transformers-version': '4.52.4', 'transformers-commit-hash': None, 'peft-version': '0.15.2.dev0', 'peft-commit-hash': '5fe7f8f8abe914d313fc3751f2ea92de7718fbaf', 'datasets-version': '3.6.0', 'datasets-commit-hash': None, 'bitsandbytes-version': '0.46.0', 'bitsandbytes-commit-hash': None, 'torch-version': '2.7.1+cu126', 'torch-commit-hash': None}

{'system': 'Linux', 'release': '6.8.0-1029-aws', 'version': '#31-Ubuntu SMP Wed Apr 23 18:42:41 UTC 2025', 'machine': 'x86_64', 'processor': 'x86_64', 'accelerator': 'NVIDIA L40S'}

bone/llama-3.2-3B-bat

BONE

2742

2431

0.5170583775587566

0.57634852206707

25251807232

20472733368

14713983755

29367552

2025-06-20 03:31:24+00:00

MetaMathQA

meta-llama/Llama-3.2-3B

{'model_id': 'meta-llama/Llama-3.2-3B', 'dtype': 'bfloat16', 'max_seq_length': 768, 'batch_size': 4, 'batch_size_eval': 50, 'max_steps': 5000, 'eval_steps': 250, 'compile': False, 'query_template': 'Question: {query} Think step by step.\nAnswer:', 'seed': 0, 'grad_norm_clip': 1.0, 'optimizer_type': 'AdamW', 'optimizer_kwargs': {'lr': 0.0001, 'weight_decay': 0.1}, 'lr_scheduler': 'cosine', 'use_amp': False, 'autocast_adapter_dtype': True, 'generation_kwargs': {'max_length': 800, 'max_new_tokens': 300}, 'attn_implementation': None}

{'task_type': None, 'peft_type': 'BONE', 'auto_mapping': None, 'base_model_name_or_path': 'meta-llama/Llama-3.2-3B', 'revision': None, 'inference_mode': False, 'r': 64, 'target_modules': ['v_proj', 'q_proj'], 'exclude_modules': None, 'init_weights': 'bat', 'layers_to_transform': None, 'layers_pattern': None, 'bias': 'none', 'modules_to_save': None}

20000

4198051

0.15.2.dev0

main

4.52.4

3.6.0

2.7.1+cu126

0.46.0

{'transformers-version': '4.52.4', 'transformers-commit-hash': None, 'peft-version': '0.15.2.dev0', 'peft-commit-hash': '5fe7f8f8abe914d313fc3751f2ea92de7718fbaf', 'datasets-version': '3.6.0', 'datasets-commit-hash': None, 'bitsandbytes-version': '0.46.0', 'bitsandbytes-commit-hash': None, 'torch-version': '2.7.1+cu126', 'torch-commit-hash': None}

{'system': 'Linux', 'release': '6.8.0-1029-aws', 'version': '#31-Ubuntu SMP Wed Apr 23 18:42:41 UTC 2025', 'machine': 'x86_64', 'processor': 'x86_64', 'accelerator': 'NVIDIA L40S'}

bone/llama-3.2-3B-default

BONE

1867

1664

0.5079605761940864

0.5775641392469406

20248002560

16303469363

11170837063

29367496

2025-06-20 04:17:11+00:00

MetaMathQA

meta-llama/Llama-3.2-3B

{'model_id': 'meta-llama/Llama-3.2-3B', 'dtype': 'bfloat16', 'max_seq_length': 768, 'batch_size': 4, 'batch_size_eval': 50, 'max_steps': 5000, 'eval_steps': 250, 'compile': False, 'query_template': 'Question: {query} Think step by step.\nAnswer:', 'seed': 0, 'grad_norm_clip': 1.0, 'optimizer_type': 'AdamW', 'optimizer_kwargs': {'lr': 0.0001, 'weight_decay': 0.1}, 'lr_scheduler': 'cosine', 'use_amp': False, 'autocast_adapter_dtype': True, 'generation_kwargs': {'max_length': 800, 'max_new_tokens': 300}, 'attn_implementation': None}

{'task_type': None, 'peft_type': 'BONE', 'auto_mapping': None, 'base_model_name_or_path': 'meta-llama/Llama-3.2-3B', 'revision': None, 'inference_mode': False, 'r': 64, 'target_modules': ['v_proj', 'q_proj'], 'exclude_modules': None, 'init_weights': True, 'layers_to_transform': None, 'layers_pattern': None, 'bias': 'none', 'modules_to_save': None}

20000

4198051

0.15.2.dev0

main

4.52.4

3.6.0

2.7.1+cu126

0.46.0

{'transformers-version': '4.52.4', 'transformers-commit-hash': None, 'peft-version': '0.15.2.dev0', 'peft-commit-hash': '5fe7f8f8abe914d313fc3751f2ea92de7718fbaf', 'datasets-version': '3.6.0', 'datasets-commit-hash': None, 'bitsandbytes-version': '0.46.0', 'bitsandbytes-commit-hash': None, 'torch-version': '2.7.1+cu126', 'torch-commit-hash': None}

{'system': 'Linux', 'release': '6.8.0-1029-aws', 'version': '#31-Ubuntu SMP Wed Apr 23 18:42:41 UTC 2025', 'machine': 'x86_64', 'processor': 'x86_64', 'accelerator': 'NVIDIA L40S'}

adaptionprompt/llama-3.2-3B-lr_0.0005

ADAPTION_PROMPT

2261

1989

0.22062168309325247

0.8317011270523071

22410166272

17907664814

11893757234

17210384

2025-06-20 04:48:22+00:00

MetaMathQA

meta-llama/Llama-3.2-3B

{'model_id': 'meta-llama/Llama-3.2-3B', 'dtype': 'bfloat16', 'max_seq_length': 768, 'batch_size': 4, 'batch_size_eval': 50, 'max_steps': 5000, 'eval_steps': 250, 'compile': False, 'query_template': 'Question: {query} Think step by step.\nAnswer:', 'seed': 0, 'grad_norm_clip': 1.0, 'optimizer_type': 'AdamW', 'optimizer_kwargs': {'lr': 0.0005}, 'lr_scheduler': 'cosine', 'use_amp': False, 'autocast_adapter_dtype': True, 'generation_kwargs': {'max_length': 800, 'max_new_tokens': 300}, 'attn_implementation': None}

{'task_type': 'CAUSAL_LM', 'peft_type': 'ADAPTION_PROMPT', 'auto_mapping': None, 'base_model_name_or_path': 'meta-llama/Llama-3.2-3B', 'revision': None, 'inference_mode': False, 'target_modules': 'self_attn', 'adapter_len': 100, 'adapter_layers': 28}

20000

4198051

0.15.2.dev0

main

4.52.4

3.6.0

2.7.1+cu126

0.46.0

{'transformers-version': '4.52.4', 'transformers-commit-hash': None, 'peft-version': '0.15.2.dev0', 'peft-commit-hash': '5fe7f8f8abe914d313fc3751f2ea92de7718fbaf', 'datasets-version': '3.6.0', 'datasets-commit-hash': None, 'bitsandbytes-version': '0.46.0', 'bitsandbytes-commit-hash': None, 'torch-version': '2.7.1+cu126', 'torch-commit-hash': None}

{'system': 'Linux', 'release': '6.8.0-1029-aws', 'version': '#31-Ubuntu SMP Wed Apr 23 18:42:41 UTC 2025', 'machine': 'x86_64', 'processor': 'x86_64', 'accelerator': 'NVIDIA L40S'}

oft/llama-3.2-3B-rank32

OFT

6852

5772

0.489764973464746

0.5956899918317795

28913434624

24327110000

18387461314

66533232

2025-06-20 05:26:07+00:00

MetaMathQA

meta-llama/Llama-3.2-3B

{'model_id': 'meta-llama/Llama-3.2-3B', 'dtype': 'bfloat16', 'max_seq_length': 768, 'batch_size': 4, 'batch_size_eval': 50, 'max_steps': 5000, 'eval_steps': 250, 'compile': False, 'query_template': 'Question: {query} Think step by step.\nAnswer:', 'seed': 0, 'grad_norm_clip': 1.0, 'optimizer_type': 'AdamW', 'optimizer_kwargs': {'lr': 0.0001, 'weight_decay': 0.1}, 'lr_scheduler': 'cosine', 'use_amp': False, 'autocast_adapter_dtype': True, 'generation_kwargs': {'max_length': 800, 'max_new_tokens': 300}, 'attn_implementation': None}

{'task_type': None, 'peft_type': 'OFT', 'auto_mapping': None, 'base_model_name_or_path': 'meta-llama/Llama-3.2-3B', 'revision': None, 'inference_mode': False, 'r': 32, 'oft_block_size': 0, 'module_dropout': 0.0, 'target_modules': ['q_proj', 'v_proj'], 'fan_in_fan_out': False, 'bias': 'none', 'exclude_modules': None, 'init_weights': True, 'layers_to_transform': None, 'layers_pattern': None, 'modules_to_save': None, 'coft': False, 'eps': 6e-05, 'block_share': False}

20000

4198051

0.15.2.dev0

main

4.52.4

3.6.0

2.7.1+cu126

0.46.0

{'transformers-version': '4.52.4', 'transformers-commit-hash': None, 'peft-version': '0.15.2.dev0', 'peft-commit-hash': '5fe7f8f8abe914d313fc3751f2ea92de7718fbaf', 'datasets-version': '3.6.0', 'datasets-commit-hash': None, 'bitsandbytes-version': '0.46.0', 'bitsandbytes-commit-hash': None, 'torch-version': '2.7.1+cu126', 'torch-commit-hash': None}

{'system': 'Linux', 'release': '6.8.0-1029-aws', 'version': '#31-Ubuntu SMP Wed Apr 23 18:42:41 UTC 2025', 'machine': 'x86_64', 'processor': 'x86_64', 'accelerator': 'NVIDIA L40S'}

randlora/llama-3.2-3B-default

RANDLORA

2457

2213

0.5072024260803639

0.5784689987897873

22798139392

18436063232

12743670025

2211281240

2025-06-20 07:20:24+00:00

MetaMathQA

meta-llama/Llama-3.2-3B

{'model_id': 'meta-llama/Llama-3.2-3B', 'dtype': 'bfloat16', 'max_seq_length': 768, 'batch_size': 4, 'batch_size_eval': 50, 'max_steps': 5000, 'eval_steps': 250, 'compile': False, 'query_template': 'Question: {query} Think step by step.\nAnswer:', 'seed': 0, 'grad_norm_clip': 1.0, 'optimizer_type': 'AdamW', 'optimizer_kwargs': {'lr': 0.0001, 'weight_decay': 0.1}, 'lr_scheduler': 'cosine', 'use_amp': False, 'autocast_adapter_dtype': True, 'generation_kwargs': {'max_length': 800, 'max_new_tokens': 300}, 'attn_implementation': None}

{'task_type': None, 'peft_type': 'RANDLORA', 'auto_mapping': None, 'base_model_name_or_path': 'meta-llama/Llama-3.2-3B', 'revision': None, 'inference_mode': False, 'r': 32, 'target_modules': ['v_proj', 'q_proj'], 'projection_prng_key': 0, 'save_projection': True, 'sparse': False, 'very_sparse': False, 'randlora_dropout': 0.0, 'fan_in_fan_out': False, 'randlora_alpha': 640, 'bias': 'none', 'modules_to_save': None, 'init_weights': True, 'layers_to_transform': None, 'layers_pattern': None}

20000

4198051

0.15.2.dev0

main

4.52.4

3.6.0

2.7.1+cu126

0.46.0

{'transformers-version': '4.52.4', 'transformers-commit-hash': None, 'peft-version': '0.15.2.dev0', 'peft-commit-hash': '5fe7f8f8abe914d313fc3751f2ea92de7718fbaf', 'datasets-version': '3.6.0', 'datasets-commit-hash': None, 'bitsandbytes-version': '0.46.0', 'bitsandbytes-commit-hash': None, 'torch-version': '2.7.1+cu126', 'torch-commit-hash': None}

{'system': 'Linux', 'release': '6.8.0-1029-aws', 'version': '#31-Ubuntu SMP Wed Apr 23 18:42:41 UTC 2025', 'machine': 'x86_64', 'processor': 'x86_64', 'accelerator': 'NVIDIA L40S'}

prompt_tuning/llama-3.2-3B-lr_0.001

PROMPT_TUNING

2715

2394

0.25246398786959817

0.7790318930149078

24408752128

20650676715

15297364466

2457728

2025-06-20 08:01:25+00:00

MetaMathQA

meta-llama/Llama-3.2-3B

{'model_id': 'meta-llama/Llama-3.2-3B', 'dtype': 'bfloat16', 'max_seq_length': 768, 'batch_size': 4, 'batch_size_eval': 50, 'max_steps': 5000, 'eval_steps': 250, 'compile': False, 'query_template': 'Question: {query} Think step by step.\nAnswer:', 'seed': 0, 'grad_norm_clip': 1.0, 'optimizer_type': 'AdamW', 'optimizer_kwargs': {'lr': 0.001}, 'lr_scheduler': 'cosine', 'use_amp': False, 'autocast_adapter_dtype': True, 'generation_kwargs': {'max_length': 800, 'max_new_tokens': 300}, 'attn_implementation': None}

{'task_type': 'CAUSAL_LM', 'peft_type': 'PROMPT_TUNING', 'auto_mapping': None, 'base_model_name_or_path': 'meta-llama/Llama-3.2-3B', 'revision': None, 'inference_mode': False, 'num_virtual_tokens': 200, 'token_dim': 3072, 'num_transformer_submodules': 1, 'num_attention_heads': 24, 'num_layers': 28, 'prompt_tuning_init': 'RANDOM', 'prompt_tuning_init_text': None, 'tokenizer_name_or_path': None, 'tokenizer_kwargs': None}

20000

4198051

0.15.2.dev0

main

4.52.4

3.6.0

2.7.1+cu126

0.46.0

{'transformers-version': '4.52.4', 'transformers-commit-hash': None, 'peft-version': '0.15.2.dev0', 'peft-commit-hash': '5fe7f8f8abe914d313fc3751f2ea92de7718fbaf', 'datasets-version': '3.6.0', 'datasets-commit-hash': None, 'bitsandbytes-version': '0.46.0', 'bitsandbytes-commit-hash': None, 'torch-version': '2.7.1+cu126', 'torch-commit-hash': None}

{'system': 'Linux', 'release': '6.8.0-1029-aws', 'version': '#31-Ubuntu SMP Wed Apr 23 18:42:41 UTC 2025', 'machine': 'x86_64', 'processor': 'x86_64', 'accelerator': 'NVIDIA L40S'}

prompt_tuning/llama-3.2-3B-default

PROMPT_TUNING

2700

2380

0.050037907505686124

1.0654937489032745

24379392000

20669781770

15297773830

2457728

2025-06-20 08:46:44+00:00

MetaMathQA

meta-llama/Llama-3.2-3B

{'model_id': 'meta-llama/Llama-3.2-3B', 'dtype': 'bfloat16', 'max_seq_length': 768, 'batch_size': 4, 'batch_size_eval': 50, 'max_steps': 5000, 'eval_steps': 250, 'compile': False, 'query_template': 'Question: {query} Think step by step.\nAnswer:', 'seed': 0, 'grad_norm_clip': 1.0, 'optimizer_type': 'AdamW', 'optimizer_kwargs': {'lr': 0.0001, 'weight_decay': 0.1}, 'lr_scheduler': 'cosine', 'use_amp': False, 'autocast_adapter_dtype': True, 'generation_kwargs': {'max_length': 800, 'max_new_tokens': 300}, 'attn_implementation': None}

{'task_type': 'CAUSAL_LM', 'peft_type': 'PROMPT_TUNING', 'auto_mapping': None, 'base_model_name_or_path': 'meta-llama/Llama-3.2-3B', 'revision': None, 'inference_mode': False, 'num_virtual_tokens': 200, 'token_dim': 3072, 'num_transformer_submodules': 1, 'num_attention_heads': 24, 'num_layers': 28, 'prompt_tuning_init': 'RANDOM', 'prompt_tuning_init_text': None, 'tokenizer_name_or_path': None, 'tokenizer_kwargs': None}

20000

4198051

0.15.2.dev0

main

4.52.4

3.6.0

2.7.1+cu126

0.46.0

{'transformers-version': '4.52.4', 'transformers-commit-hash': None, 'peft-version': '0.15.2.dev0', 'peft-commit-hash': '5fe7f8f8abe914d313fc3751f2ea92de7718fbaf', 'datasets-version': '3.6.0', 'datasets-commit-hash': None, 'bitsandbytes-version': '0.46.0', 'bitsandbytes-commit-hash': None, 'torch-version': '2.7.1+cu126', 'torch-commit-hash': None}

{'system': 'Linux', 'release': '6.8.0-1029-aws', 'version': '#31-Ubuntu SMP Wed Apr 23 18:42:41 UTC 2025', 'machine': 'x86_64', 'processor': 'x86_64', 'accelerator': 'NVIDIA L40S'}

fourierft/llama-3.2-3B-n_frequency-5000

FOURIERFT

2824

2422

0.1197877179681577

0.9978675174713135

23681040384

19054869872

13111221498

1127472

2025-06-20 09:31:48+00:00

MetaMathQA

meta-llama/Llama-3.2-3B

{'model_id': 'meta-llama/Llama-3.2-3B', 'dtype': 'bfloat16', 'max_seq_length': 768, 'batch_size': 4, 'batch_size_eval': 50, 'max_steps': 5000, 'eval_steps': 250, 'compile': False, 'query_template': 'Question: {query} Think step by step.\nAnswer:', 'seed': 0, 'grad_norm_clip': 1.0, 'optimizer_type': 'AdamW', 'optimizer_kwargs': {'lr': 0.0001, 'weight_decay': 0.1}, 'lr_scheduler': 'cosine', 'use_amp': False, 'autocast_adapter_dtype': True, 'generation_kwargs': {'max_length': 800, 'max_new_tokens': 300}, 'attn_implementation': None}

{'task_type': None, 'peft_type': 'FOURIERFT', 'auto_mapping': None, 'base_model_name_or_path': 'meta-llama/Llama-3.2-3B', 'revision': None, 'inference_mode': False, 'n_frequency': 5000, 'scaling': 300, 'random_loc_seed': 777, 'fan_in_fan_out': False, 'target_modules': ['v_proj', 'q_proj'], 'exclude_modules': None, 'bias': 'none', 'modules_to_save': None, 'layers_to_transform': None, 'layers_pattern': None, 'n_frequency_pattern': {}, 'init_weights': False}

20000

4198051

0.15.2.dev0

main

4.52.4

3.6.0

2.7.1+cu126

0.46.0

{'transformers-version': '4.52.4', 'transformers-commit-hash': None, 'peft-version': '0.15.2.dev0', 'peft-commit-hash': '5fe7f8f8abe914d313fc3751f2ea92de7718fbaf', 'datasets-version': '3.6.0', 'datasets-commit-hash': None, 'bitsandbytes-version': '0.46.0', 'bitsandbytes-commit-hash': None, 'torch-version': '2.7.1+cu126', 'torch-commit-hash': None}

{'system': 'Linux', 'release': '6.8.0-1029-aws', 'version': '#31-Ubuntu SMP Wed Apr 23 18:42:41 UTC 2025', 'machine': 'x86_64', 'processor': 'x86_64', 'accelerator': 'NVIDIA L40S'}

fourierft/llama-3.2-3B-default

FOURIERFT

2824

2424

0.000758150113722517

1.2480293517112733

23653777408

19017267937

13104129350

231416

2025-06-20 10:18:57+00:00

MetaMathQA

meta-llama/Llama-3.2-3B

{'model_id': 'meta-llama/Llama-3.2-3B', 'dtype': 'bfloat16', 'max_seq_length': 768, 'batch_size': 4, 'batch_size_eval': 50, 'max_steps': 5000, 'eval_steps': 250, 'compile': False, 'query_template': 'Question: {query} Think step by step.\nAnswer:', 'seed': 0, 'grad_norm_clip': 1.0, 'optimizer_type': 'AdamW', 'optimizer_kwargs': {'lr': 0.0001, 'weight_decay': 0.1}, 'lr_scheduler': 'cosine', 'use_amp': False, 'autocast_adapter_dtype': True, 'generation_kwargs': {'max_length': 800, 'max_new_tokens': 300}, 'attn_implementation': None}

{'task_type': None, 'peft_type': 'FOURIERFT', 'auto_mapping': None, 'base_model_name_or_path': 'meta-llama/Llama-3.2-3B', 'revision': None, 'inference_mode': False, 'n_frequency': 1000, 'scaling': 300, 'random_loc_seed': 777, 'fan_in_fan_out': False, 'target_modules': ['q_proj', 'v_proj'], 'exclude_modules': None, 'bias': 'none', 'modules_to_save': None, 'layers_to_transform': None, 'layers_pattern': None, 'n_frequency_pattern': {}, 'init_weights': False}

20000

4198051

0.15.2.dev0

main

4.52.4

3.6.0

2.7.1+cu126

0.46.0

{'transformers-version': '4.52.4', 'transformers-commit-hash': None, 'peft-version': '0.15.2.dev0', 'peft-commit-hash': '5fe7f8f8abe914d313fc3751f2ea92de7718fbaf', 'datasets-version': '3.6.0', 'datasets-commit-hash': None, 'bitsandbytes-version': '0.46.0', 'bitsandbytes-commit-hash': None, 'torch-version': '2.7.1+cu126', 'torch-commit-hash': None}

{'system': 'Linux', 'release': '6.8.0-1029-aws', 'version': '#31-Ubuntu SMP Wed Apr 23 18:42:41 UTC 2025', 'machine': 'x86_64', 'processor': 'x86_64', 'accelerator': 'NVIDIA L40S'}

ln_tuning/llama-3.2-3B-default

LN_TUNING

1870

1657

0

1.2370348122119903

21177040896

16903066091

11385589622

358288

2025-06-20 11:06:05+00:00

MetaMathQA

meta-llama/Llama-3.2-3B

{'model_id': 'meta-llama/Llama-3.2-3B', 'dtype': 'bfloat16', 'max_seq_length': 768, 'batch_size': 4, 'batch_size_eval': 50, 'max_steps': 5000, 'eval_steps': 250, 'compile': False, 'query_template': 'Question: {query} Think step by step.\nAnswer:', 'seed': 0, 'grad_norm_clip': 1.0, 'optimizer_type': 'AdamW', 'optimizer_kwargs': {'lr': 0.0001, 'weight_decay': 0.1}, 'lr_scheduler': 'cosine', 'use_amp': False, 'autocast_adapter_dtype': True, 'generation_kwargs': {'max_length': 800, 'max_new_tokens': 300}, 'attn_implementation': None}

{'task_type': None, 'peft_type': 'LN_TUNING', 'auto_mapping': None, 'base_model_name_or_path': 'meta-llama/Llama-3.2-3B', 'revision': None, 'inference_mode': False, 'target_modules': ['input_layernorm', 'norm', 'post_attention_layernorm'], 'exclude_modules': None, 'modules_to_save': None}

20000

4198051

0.15.2.dev0

main

4.52.4

3.6.0

2.7.1+cu126

0.46.0

{'transformers-version': '4.52.4', 'transformers-commit-hash': None, 'peft-version': '0.15.2.dev0', 'peft-commit-hash': '5fe7f8f8abe914d313fc3751f2ea92de7718fbaf', 'datasets-version': '3.6.0', 'datasets-commit-hash': None, 'bitsandbytes-version': '0.46.0', 'bitsandbytes-commit-hash': None, 'torch-version': '2.7.1+cu126', 'torch-commit-hash': None}

{'system': 'Linux', 'release': '6.8.0-1029-aws', 'version': '#31-Ubuntu SMP Wed Apr 23 18:42:41 UTC 2025', 'machine': 'x86_64', 'processor': 'x86_64', 'accelerator': 'NVIDIA L40S'}

full-finetuning/llama-3.2-3B-lr_0.00001

full-finetuning

3275

3111

0.5003790750568613

0.5987544150352478

37241225216

33573390254

33098872284

6425499648

2025-06-20 18:02:43+00:00

MetaMathQA

meta-llama/Llama-3.2-3B

{'model_id': 'meta-llama/Llama-3.2-3B', 'dtype': 'bfloat16', 'max_seq_length': 768, 'batch_size': 4, 'batch_size_eval': 50, 'max_steps': 5000, 'eval_steps': 250, 'compile': False, 'query_template': 'Question: {query} Think step by step.\nAnswer:', 'seed': 0, 'grad_norm_clip': 1.0, 'optimizer_type': 'AdamW', 'optimizer_kwargs': {'lr': 1e-05}, 'lr_scheduler': 'cosine', 'use_amp': False, 'autocast_adapter_dtype': True, 'generation_kwargs': {'max_length': 800, 'max_new_tokens': 300}, 'attn_implementation': None}

null

20000

4198051

0.15.2.dev0

main

4.52.4

3.6.0

2.7.1+cu126

0.46.0

{'transformers-version': '4.52.4', 'transformers-commit-hash': None, 'peft-version': '0.15.2.dev0', 'peft-commit-hash': '5fe7f8f8abe914d313fc3751f2ea92de7718fbaf', 'datasets-version': '3.6.0', 'datasets-commit-hash': None, 'bitsandbytes-version': '0.46.0', 'bitsandbytes-commit-hash': None, 'torch-version': '2.7.1+cu126', 'torch-commit-hash': None}

{'system': 'Linux', 'release': '6.8.0-1029-aws', 'version': '#31-Ubuntu SMP Wed Apr 23 18:42:41 UTC 2025', 'machine': 'x86_64', 'processor': 'x86_64', 'accelerator': 'NVIDIA L40S'}

PEFT method comparison

Choose the task and base model

Pareto plot

Export data