Rendered at 19:22:59 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
smokel 56 minutes ago [-]
This refers specifically to "AI Frontiers", a research group within Microsoft Research that focuses on fundamental advances in AI. There is no end in sight to real AI frontiers.
Perhaps "The End of Microsoft Research's AI Frontiers Group"
would be a less ambiguous title.
sheepscreek 42 minutes ago [-]
Yeah, I went down the rabbit hole scratching my head the whole time. This is about the group at Microsoft Research that was disbanded or reorged into what is now Microsoft Research.
To the author: 100% need this kind of thinking. It’s a balancing act. Historically though, dedicated labs haven’t had the best track record at commercializing their tech. The skillset differs vastly from running a product-focused business.
That said, even within this narrow scope - there are areas ripe for research.
Some of my particular interests include:
(1) Latent thinking/reasoning, which could arguably be more efficient but also opaque is one
(2) Separation of “intelligence” from knowledge - ie. keep the ability to reason like a frontier model and make tool calls (calling the ‘oracle’ or searching the web) but remove all parameters associated with niche knowledge/facts.
These are a couple off the top of my head that are very much aligned to scaling.
xlayn 17 minutes ago [-]
I see this playing the same way in the other big companies that have a pressure to deliver (cough *value for shareholders*)
AnotherGoodName 42 minutes ago [-]
This is true in every large company right now.
The transformer based approach is seen as a highway to race down in terms of training and building better models with minimal organizational distractions allowed while every company puts its foot down on this highway. Exploratory research into other AI methods is highly discouraged for now and organizations are encouraging transformer based scale out and iteration as a top priority.
As an example Meta literally fired it's previously head of AI who held views that transformers won't scale. He wanted to veer off the highway and heavily fund exploratory new types of models. They got someone willing to scale the current method out. Essentially the messaging it "Foot down, there's a straight highway ahead don't divert".
I have the view this makes sense too. Current top models are mere (mere in big datacenter terms) trillions of parameters right now. We definitely haven't hit the ceiling and everyone's noticing "Hey this Opus 5.5 with the increased parameters seems much better than the last one". Well that's a trillion parameter model. We'll soon start hitting 10's of trillions and 100's of trillions of parameter models. Unless there's a ceiling which nobody expects they should keep scaling just as they have been. I really do agree that it'd be insane to veer off this highway with exploratory research at this particular point in time. Foot down, get to 10 trillion parameters, see if the scaling holds, if so off to 100 trillion parameters. If the scaling continues to hold we'll be asking the AI to do the exploratory research for us soon enough.
redox99 36 minutes ago [-]
> As an example Meta literally fired it's previously head of AI who held views that transformers won't scale. He wanted to veer off the highway and heavily fund exploratory new types of models.
Almost all statements Yann LeCun made back then (LLMs won't be able to do this or that) turned out to be wrong. They were right to fire him.
wtetzner 28 minutes ago [-]
Even if he was wrong about those statements, exploring other types of models isn't necessarily a bad thing. Especially if we can find ones that perform well with less demanding hardware requirements.
frays 9 minutes ago [-]
The issue isn't the exploration itself, it's the opportunity cost. When a figure of Yann LeCun's influence pushes a specific direction, it pulls key talent and resources away from mainstream LLM work. That kind of organizational distraction has a massive cost, and would've stopped Meta from delivering a polished product like Muse sooner.
esseph 4 minutes ago [-]
[delayed]
AnotherGoodName 27 minutes ago [-]
Absolutely agree fwiw. Scale out over LeCunns desired exploratory research was absolutely the right call. Called out because it seems like MS is making a similar call.
The author states "I consider the scaling-only focus to be both misdirected and not fun.". I get the point but also totally get the focus on scale out for now.
srean 36 minutes ago [-]
When a company pays billions to OpenAI, any activity that does not do glorious dance in praise and support of the investment is an act of disloyalty.
fraywing 51 minutes ago [-]
To me, all signs point to US over investment in hardware capacity to support the "frontier" when in reality the next major breakthroughs are likely radical optimizations that end in increased consumer training and inference efficiency.
I'm not sure how that, combined with open models inching up on frontier intelligence, is compatible with these massive AI IPO valuations?
It seems like closed model AI demand has a lot of contingencies that are ultimately fragile and possibly short sighted.
uptrails 46 minutes ago [-]
The article is about a guy leaving Microsoft and the disbanding of an internal team called "AI Frontiers".
fraywing 35 minutes ago [-]
Yeah, I'm speaking to "scaling vs research" and where investment is going. Which is relevant to the article. Accelerating scale requires demand, closed frontier demand is what I'm questioning.
joe_the_user 41 minutes ago [-]
And gp's comment is relevant to the particular problems the guy (John Langford) cites in his comment about leaving. IE, that research first winds being only about scaling and then winds-up ignored because "just scale" can be an easier answer.
zodwick 5 minutes ago [-]
Not what i expected, but good read nonetheless
ansk 23 minutes ago [-]
If you value the freedom to pursue your own research direction and share your work freely, then choose academia. All the perks of working in an industry research role are provided in exchange for this freedom.
If your research aligns with the incentives of industry, you can get the best of both worlds in industry. This was the case for machine learning researchers in the post-AlexNet, pre-ChatGPT era. The good times can't last forever, though.
srean 9 minutes ago [-]
I largely agree. For US though, academia has not been a good place lately.
redmaple892 25 minutes ago [-]
The Microsoft AI (MAI) part stood out to me. I bet the politics of being in Mustafa's group were not fun.
alpha_trion 40 minutes ago [-]
Lack of focus and long term planning at MS
outside1234 44 minutes ago [-]
Microsoft is a really shitty place to work right now. There are a hundred stories like this all over the company like this.
rwmj 22 minutes ago [-]
They've offered voluntary redundancy to a quite large proportion of the staff (a quarter?) so they'll also soon be losing anyone with employment prospects elsewhere.
Perhaps "The End of Microsoft Research's AI Frontiers Group" would be a less ambiguous title.
To the author: 100% need this kind of thinking. It’s a balancing act. Historically though, dedicated labs haven’t had the best track record at commercializing their tech. The skillset differs vastly from running a product-focused business.
That said, even within this narrow scope - there are areas ripe for research.
Some of my particular interests include:
(1) Latent thinking/reasoning, which could arguably be more efficient but also opaque is one
(2) Separation of “intelligence” from knowledge - ie. keep the ability to reason like a frontier model and make tool calls (calling the ‘oracle’ or searching the web) but remove all parameters associated with niche knowledge/facts.
These are a couple off the top of my head that are very much aligned to scaling.
The transformer based approach is seen as a highway to race down in terms of training and building better models with minimal organizational distractions allowed while every company puts its foot down on this highway. Exploratory research into other AI methods is highly discouraged for now and organizations are encouraging transformer based scale out and iteration as a top priority.
As an example Meta literally fired it's previously head of AI who held views that transformers won't scale. He wanted to veer off the highway and heavily fund exploratory new types of models. They got someone willing to scale the current method out. Essentially the messaging it "Foot down, there's a straight highway ahead don't divert".
I have the view this makes sense too. Current top models are mere (mere in big datacenter terms) trillions of parameters right now. We definitely haven't hit the ceiling and everyone's noticing "Hey this Opus 5.5 with the increased parameters seems much better than the last one". Well that's a trillion parameter model. We'll soon start hitting 10's of trillions and 100's of trillions of parameter models. Unless there's a ceiling which nobody expects they should keep scaling just as they have been. I really do agree that it'd be insane to veer off this highway with exploratory research at this particular point in time. Foot down, get to 10 trillion parameters, see if the scaling holds, if so off to 100 trillion parameters. If the scaling continues to hold we'll be asking the AI to do the exploratory research for us soon enough.
Almost all statements Yann LeCun made back then (LLMs won't be able to do this or that) turned out to be wrong. They were right to fire him.
The author states "I consider the scaling-only focus to be both misdirected and not fun.". I get the point but also totally get the focus on scale out for now.
I'm not sure how that, combined with open models inching up on frontier intelligence, is compatible with these massive AI IPO valuations?
It seems like closed model AI demand has a lot of contingencies that are ultimately fragile and possibly short sighted.
If your research aligns with the incentives of industry, you can get the best of both worlds in industry. This was the case for machine learning researchers in the post-AlexNet, pre-ChatGPT era. The good times can't last forever, though.