D2L-73-Positional_Encoding

Last updated Apr 27, 2022 Edit Source

• All
• Positional encoding
• Self attention
• Deep learning

Tags: #PositionalEncoding #Self-Attention #DeepLearning

• 为了使用序列的顺序信息，我们通过在输入表示中添加 位置编码（positional encoding）来注入绝对的或相对的位置信息。

• 我觉得D2L讲的很深入很好了: 10.6. Self-Attention and Positional Encoding

Highlights:

• In binary representations, a higher bit has a lower frequency than a lower bit. Similarly, as demonstrated in the heat map below, the positional encoding decreases frequencies along the encoding dimension by using trigonometric functions. Since the outputs are float numbers, such continuous representations are more space-efficient than binary representations.