<!--Copyright 2023 The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.

⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.

-->

# IA3

Infused Adapter by Inhibiting and Amplifying Inner Activations, or [IA3](https://hf.co/papers/2205.05638), is a method
that adds three learned vectors to rescale the keys and values of the self-attention and encoder-decoder attention
layers, and the intermediate activation of the position-wise feed-forward network.

The abstract from the paper is:

*Few-shot in-context learning (ICL) enables pre-trained language models to perform a previously-unseen task without any
gradient-based training by feeding a small number of training examples as part of the input. ICL incurs substantial
computational, memory, and storage costs because it involves processing all of the training examples every time a
prediction is made. Parameter-efficient fine-tuning (PEFT) (e.g. adapter modules, prompt tuning, sparse update methods,
etc.) offers an alternative paradigm where a small set of parameters are trained to enable a model to perform the new
task. In this paper, we rigorously compare few-shot ICL and PEFT and demonstrate that the latter offers better accuracy
as well as dramatically lower computational costs. Along the way, we introduce a new PEFT method called (IA)^3 that
scales activations by learned vectors, attaining stronger performance while only introducing a relatively tiny amount of
new parameters. We also propose a simple recipe based on the T0 model called T-Few that can be applied to new tasks
without task-specific tuning or modifications. We validate the effectiveness of T-Few on completely unseen tasks by
applying it to the RAFT benchmark, attaining super-human performance for the first time and outperforming the
state-of-the-art by 6% absolute. All of the code used in our experiments is publicly available*.

## IA3Config

[[autodoc]] tuners.ia3.config.IA3Config

## IA3Model

[[autodoc]] tuners.ia3.model.IA3Model