Synapse
⌘+K
Synapse
PulseExploreClubsResearchersJournals
Instagram
HomeClubsExplore
August 19, 2024Open Access

In-Context Learning with Representations: Contextual Generalization of Trained Transformers

View Full Paper
Ask AI
Bookmark
Share

Authors

TYTong YangHYHuang YuYLYingbin Liang

Discussion

Loading...

Member takes

Overview

This randomized trial investigates contextual generalization in pretrained language models, highlighting their learning dynamics.

Key Points

  • Transformers demonstrate effective contextual generalization through in-context learning, improving task adaptability.
  • The study shows that training loss converges linearly to a minimum for a one-layer transformer model.
  • Assessment through non-linear regression tasks reveals the efficacy of gradient descent in learning template functions effectively for unseen inputs and tasks alike, despite noise levels in data. The significance features the models’ promising potential for broader applications in various practical scenarios.

Cite This Study

Yang et al. (2024) studied this question.

synapsesocial.com/papers/68e5bd3ab6db643587555043https://doi.org/10.48550/arxiv.2408.10147
View Full Paper
Ask AI
Bookmark
Share

Also Consider

Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context:

  1. 1Transformers Don't In-Context Learn Least Squares Regression2025
  2. 2How Do Nonlinear Transformers Learn and Generalize in In-Context Learning?2024 · 1 citations
  3. 3Asymptotic theory of in-context learning by linear attention2024 · 1 citations
  4. 4Formalizing In-Context Learning in Transformers as Implicit Gradient Descent2026
  5. 5MLPs Learn In-Context on Regression and Classification Tasks2024 · 2 citations