×
Well done. You've clicked the tower. This would actually achieve something if you had logged in first. Use the key for that. The name takes you home. This is where all the applicables sit. And you can't apply any changes to my site unless you are logged in.

Our policy is best summarized as "we don't care about _you_, we care about _them_", no emails, so no forgetting your password. You have no rights. It's like you don't even exist. If you publish material, I reserve the right to remove it, or use it myself.

Don't impersonate. Don't name someone involuntarily. You can lose everything if you cross the line, and no, I won't cancel your automatic payments first, so you'll have to do it the hard way. See how serious this sounds? That's how serious you're meant to take these.

×
Register


Required. 150 characters or fewer. Letters, digits and @/./+/-/_ only.
  • Your password can’t be too similar to your other personal information.
  • Your password must contain at least 8 characters.
  • Your password can’t be a commonly used password.
  • Your password can’t be entirely numeric.
Enter the same password as before, for verification.
Login

Grow A Dic
Define A Word
Make Space
Set Task
Mark Post
Apply Votestyle
Create Votes
(From: saved spaces)
Exclude Votes
Apply Dic
Exclude Dic

Finite Sample Analysis and Bounds of Generalization Error of Gradient Descent in In-Context Linear Regression

Click here to flash read.

arXiv:2405.02462v1 Announce Type: new
Abstract: Recent studies show that transformer-based architectures emulate gradient descent during a forward pass, contributing to in-context learning capabilities - an ability where the model adapts to new tasks based on a sequence of prompt examples without being explicitly trained or fine tuned to do so. This work investigates the generalization properties of a single step of gradient descent in the context of linear regression with well-specified models. A random design setting is considered and analytical expressions are derived for the statistical properties of generalization error in a non-asymptotic (finite sample) setting. These expressions are notable for avoiding arbitrary constants, and thus offer robust quantitative information and scaling relationships. These results are contrasted with those from classical least squares regression (for which analogous finite sample bounds are also derived), shedding light on systematic and noise components, as well as optimal step sizes. Additionally, identities involving high-order products of Gaussian random matrices are presented as a byproduct of the analysis.

Click here to read this post out
ID: 840130; Unique Viewers: 0
Unique Voters: 0
Total Votes: 0
Votes:
Latest Change: May 7, 2024, 7:33 a.m. Changes:
/u/anonymous
Dictionaries:
Words:
Spaces:
Views: 6
CC:
No creative common's license
Comments: