Exploring the Growth and Mission of the PyTorch Foundation with Matt White | Open Source Summit NA 20025

Hey everybody. We're at the Open Source Summit in Denver, and we're gonna have a little chat about Pie Torch with Matt White, who's the executive director for the Pie Torch Foundation. Matt, welcome to show.

How you Doing? Hey, very well. Very well, thank you.

PyTorch, the foundation is expanding its mission apparently, and that was some of the conversations that were in the keynote today. So beyond PyTorch itself, what, what are your ambitions for the foundation? Sure.

Yeah. So the foundation's actually grown quite, quite considerably since it started, uh, in 2022 with, um, the Linux Foundation. So prior to that it was with, um, meta.

And so in these last few years, and in particularly in the last about six months, we've grown to become an umbrella foundation, which means that we're able to take on additional projects beyond just the PyTorch framework. Um, and this puts us in a really good position to help steward be good stewards of open source AI and sort of advance the cause and, and provide tooling and open source that embraces open principles and, um, moves the dial for us. And so some of these projects, uh, satisfy really important parts of the AI lifecycle.

And so the ecosystem has grown. And we recent, recently announced that we added additional projects, uh, VLLM and deep speed, and we have a few projects on the back burner that, uh, will be joining shortly as well. So How far does that go across the lifecycle?

Because there are open source AI coding tools. I mean, how, how ambitious can this whole program get? Sure, sure.

So our, our objective really is to equip AI researchers and engineers, and basically scientists and anyone else even hobbyists to, with the tools that they need to be able to do either research or productionalize workloads or even just learn about PyTorch and learn about AI and, and deep learning. And so as far as we're going on, the horizontal is really from data ingestion, so data pre-processing and storage all the way through for, you know, model training and then right to the, you know, tail side of that, which is like inference and serving. And so that really gets us through this like horizontal, where we're also building up vertically, including like AI frameworks, agent frameworks, and all of the tools that are needed by mainly enterprise to build applications on top of AI models as well.

So, does PyTorch the foundation become the primary vehicle for open source AI development? Or will there be other foundations alongside it that need to kind of collaborate and what will be the relationship? Yeah, So I think even within the like specter of like Linux Foundation itself, there are other adjacent projects and adjacent foundations that we, you know, have collaborations with.

And if you look at like the tech stack for, um, you know, let's just say like, like serving side for ai, so serving models to, um, end users on, you know, the very bottom of that. You have the Linux operating system and the kernel and built on that you have Kubernetes and you kind of keep going up. And so there's always gonna be this like, interplay of different foundations working together.

Um, and so we do collaborate with other foundations on AI directed initiatives as well. Mm-hmm. When all of this first started out, there was no shortage of tools and it was kinda, you know, to a lot of folks it kind of looked like gobbly g**k, right?

There was all kinds of different things to do, all kinds of different things. Is that starting to coalesce a little bit? Are there certain tools that are maybe becoming defacto standards that are kind of more widely adopted?

And have we reached some level of maturity there? I, I think the, the, I think the industry has sort of spoke at least on the like training side with PyTorch, which, you know, hugging faces announcement that they're going to be strictly based with PyTorch and they've, you know, decommissioned jacks and, uh, TensorFlow, uh, from their, from their platform. So, and most contemporary models are built on PyTorch.

Uh, most distributions of open models are in, uh, you know, PyTorch ready format. And so I think from the, at least from the training side and model redistribution side, uh, you know, PyTorch has being sort of the, the defacto, um, you know, uh, framework of choice on the serving side. There's still some, you know, discussions, but VLM has kind of skyrocketed up in popularity and, uh, you know, they joined us, uh, earlier this year as one as our initial, um, you know, umbrella foundation project.

Yeah. And VLLM is an open source alternative to Cuda and whatnot, and those types of frameworks. 'cause I don't think everybody knows what V lms.

Yeah, sure, sure. So VLM takes care of like loading a model and serving it and, and doing all the inference. So when you go on, you know, Chachi pt a query, a model that on the side on the back end of that, it's serving.

And so that's what VLM solves, right? Is being able to serve up tokens or serve up bits or whatever the case may be, whatever, um, modality it is. And so it's, you know, torch Serve was a, a project that in the PyTorch ecosystem that did serving, uh, there's others out there like SG Lang.

And so, but the industry's really rallied around VLM and a lot of folks like IBM, red Hat, you know, meta Cisco, others have kind of put their weight behind, um, this serving framework. And the Idea is not to get locked in on the inference side to a particular platform. Right?

Right. Yeah. And is that gonna apply not just the GPUs, but other classes of processors?

It's kind of, I mean, how Open is open, yeah. So like VLM team, just like the PyTorch team and others that are working closer to, to the metal, um, work with those vendors, right? And so work with a MD Nvidia, work with Intel, you know, Qualcomm arm, so forth to make sure that models are performant on their particular silicon, right?

And so this is always like a constant process of evaluating, and as you know, PyTorch evolves as some of these other platforms that work more closely with the Silicon evolve. They work sort of hand in hand, um, to make sure that models are performance on these, uh, these different, uh, systems. Coming back to, uh, PyTorch, um, it's, you know, it has a relationship with Python, which is widely adopted by folks beyond just developers.

So will we get to a point where we'll see more so-called citizen developers working with PyTorch to go build and train models? I mean, how expansive can the target audience get? Yeah, I think you, there's some limitations with like, especially large models, right?

Like we're, when we're talking about smaller models, like for recommender systems or these classifiers that they often use for, you know, identifying whether content is, you know, like for guardrails, like identifying whether content is, you know, has, you know, foul words in it or something like this, right? Um, and so these, these are a little more accessible because there's a smaller, uh, sort of like hardware burden to being able to train those models. But when we talk about large language models, there's significant hardware involved.

And so I think I'm seeing we're, we're sort of seeing the most, like usage is actually on the fine tuning side. So someone builds a foundation model, you know, large organization or lab builds that foundation model, releases it with a permissive license, and then downstream users take that model and fine tune it on data for their particular application and create sort of like something that's very tailored to them, or they, um, experiment with it. And, you know, there's a lot of hobbyists out there that can actually never really need to touch PyTorch because they have all these like web accessible tools, right?

Like hugging face, uh, for example, right? And there's some UNL and some other platforms that really make fine tuning and other ac you know, quantization and these other sort of transformations with models much more accessible. And so I think going back to like, you know, PyTorch, it requires a certain set of skills, you know, to your point on, on Python, um, but also having access to the infrastructure needed to be able to train at scale.

There's been a lot of debate about what open source means in the context of ai, and some people are saying that their models are open and other people say, well, they're not as open as they should be. 'cause they don't have, like, the weights are not included, or I didn't understand how the data was set up. How should we be thinking about open source in the age of ai?

Yeah, That's a pretty, um, that's a good question. Uh, it, it's, you know, there's different folks that have different beliefs about what this, you know, when they take like the open source paradigm and apply it into world of AI where there's, you know, in the, in the conventional space in open source we have software period, right? Um, now we have all these other artifacts that go into the mix of creating a model and creating some, you know, output that can be used by downstream users, right?

And so there's, you know, training code, there's inference codes, there's different types of data sets that are used for pre-training or fine tuning and so forth. And then obviously documentation, research papers, there's a lot of things that can go into that. And when Linux Foundation published a, um, a something it's called the model Openness Framework about a year ago or maybe a year and a half ago, which set forth to kind of look at what, what the definition of open science would be, what would be an open model in this new paradigm, right?

And, and when we're talking about openness, and so we sort of went through these two dimensions of like openness and completeness where the industry had sort of settled on this term openness to kind of cover the licensing cover, how many components are released. We went and looked at, okay, completeness is really like how many components are released. And then openness is, is it released under an open source or open license, right?

And so we try to disentangle that to make it much more clear to the public, like what each is responsible for. And I think conflating those two things makes it very difficult to understand like what open source AI really is and what open models are. But if, you know, within our, our, um, classification system, an open model to us was, you know, a model and it's associated weights but released under a permissive license, right?

And this is what gives you what sort of embraces that spirit of openness because in open source it's about being able to use modify study for any purpose without limitations. And so we were able to replicate that with, with models through that framework. Most of the usage of models, at least the ones we hear about anyway, seem to be tied up around proprietary ones, is the open source models gonna become more dominant as we go along.

'cause history would show us that open source seems to eventually catch up and then it becomes the innovation engine when everybody starts contributing. Yeah. Yeah.

And, and we see this time and time again, right, where things are sort of released and as black boxes are developed in house, and then their open source community replicates them, and then at when the sort of performance differential is so minute, eventually, like folks really rally around the open solutions and then the innovation happens on top, right? And so with, with models, we're seeing that the, like, you know, there were big leaps in open source is way behind black box solutions for models. And then we sort of saw that gap close, and now we're at the point where a lot less companies are releasing these foundation models because the differential isn't so extreme, um, between, you know, the last generation and this generation.

And I think we'll start to see less innovation potentially on the model side, although it's not going away, um, but a lot more interest in innovation on the system side, which is now where we're seeing like people that are looking at like, how do I integrate with this back office system? How do I get data out of this particular database? And, you know, we're seeing the rise of protocols and standards like MCP and A to A and others to try and solve a lot of these problems.

But now that we're more systems focused, we're gonna see a lot more innovation in that space and, you know, continue to grow around innovating on top of these foundation models that may not, um, you know, they'll continue to incrementally get better, but the, you know, the industry may not be as model focused as they are systems focused when we try to, you know, operationalize these platforms. Yeah. Last question.

Sure. Um, and I hear this all the time. Everybody kind of, you know, they hear open source and they nod their head and they have a warm feeling for it, but they don't know where to get started.

They don't know how to join the community and they don't seem to have a good handle on where do I get involved. So if you want folks to, you know, come work with you on the PyTorch Foundation, where should they get started? You know, where, where's the interface?

Sure, sure. Yeah, I think there's, there's a few entry points, right? org, which is our discussion boards where people can glean a lot of information from that.

We're also actually building out, um, you know, we do currently have like tutorials and other artifacts online, but we actually have two programs. The foundation is launching, uh, this summer, which are focused on training and certification. And the other is with, uh, what we call like academic outreach or osbo outreach.

So we're building the packets so that educators can teach their students how to get started with pie torch, how to get off the ground with it. And then on the training side, we're actually building training materials and certifications for both entry level and advanced, uh, you know, PyTorch skills. And then we also have like our Discord server and Slack, which are good forums for people to kind of get involved in, start asking some of the questions about like, how do I get started?

And, and, uh, you know, how can I train a model and experiment and these sort of things. All right, folks, you heard it here. There's a new epicenter starting to emerge around open source ai and it's at the PyTorch Foundation, so check it out.

Matt, thanks for Coming by. Thank you very much. Appreciate it.

All right. Yeah. And we'll be back in a minute.