The attention mechanism lets a model weight the importance of the other elements of the sequence to compute the representation of one element. Each token produces a query, a key and a value; the weights come from the query-key similarity after softmax. Self-attention directly connects distant positions.