The AI Factory in Action: Basketball play classification with Hewlett Packard Enterprise

Mark site, their solutions architect for Hewlett Packard Enterprise. And we're gonna look at the AI factory in action. Um, and doing basketball play classification.

So this NBA team, one of the other things they said to me, but aside from we wanna get better with document intelligence 'cause it kills us. The other thing that they told me was, we do interesting things with video because that's one of the biggest assets that we have, right? We've got game film, we've got practice film, we've got a thousand cameras in our stadium, right?

Could we be doing more with this data? Uh, and so the piece that we wanted to do for them, they said, they actually came to me with the idea and they said, could you do something around player fatigue? So like with a camera and using video, could I notice if a player is fatigued?

Okay, what does that mean? Are, did their gait change or are they not fighting through screens? Are they not fighting, you know, like over screens, through pick and roll.

Are they not getting back on defense? Do they, are they gas? Right?

Do they need to be taken out? Could we get insights like that? And so I took that and I went home and I spent some time in the next couple of weeks and I started building something.

And it was a very simple idea. Um, what I wanted to do was take an open source model, right? And it's called Slow fast from Facebook Research.

If anyone's familiar with slow, fast, slow, fast is a classification model, video classification model. It has two pathways, it has a fast pathway and has a slow pathway. Fast pathway is meant for movement.

Lower, higher frame rate, lower resolution. The slow pathway is meant for objects, right? So I think object detection.

So it's a high resolution but a low frame rate, right? So more for like what's going on statically in the image or the series of images. And I wanted to see if I could take something because they currently have a team of analysts that the coach will be like, I wanna see this.

And then their analysts go and they sit down and watch videos to find. They're like, oh yeah, I think I remember that. Like from this game back in the playoffs in 2017.

Lemme go see if I can pull up these clips, right? That's kind of the process. And so I said to myself, I think, and I'm gonna switch over to my, I'm gonna switch over here.

So They labeled the frame, they labeled the videos for you. Is that how I labeled them? By hand myself?

Just because I wanna see what was possible. There's a lot of other interesting pieces and I'll tell you about where I'm taking this and I maybe you'll think it's cool. Um, I think it's fun.

This is the platform itself, that software platform. And I, I wanna show you this before I dive in deeper. This is the platform.

So this is AI essential. So this is the software aspect of PC ai. And it has on it, you can see like this is kind of a getting started page.

There's not like a ton of information on here other than I would say like recent items. The top frameworks. You can see your utilization of all of these different tools and what you're actually doing with them.

The getting started piece is, we've been doing this for a long time and we've got a lot of things that our customers crib from, right? We've built things and showed in these examples how to use all of these different tools, how to connect to different sources of data. And so as part of that, we want customers to take what we've done and use it to stand on, right?

Use it to continue and, and go further and develop with it. So we've provided a lot of documentation around that Slow, fast integrated here. Is that just something you were playing?

Nope. Slow Fast was an external model that I brought in just 'cause I, I wanted to, it was what I thought appropriate, most appropriately matched what I wanted to do. So I brought that in and I can tell you more about that with MLIS.

Remind me, MIS is, I'll talk about it in my flow. I always start this off with the data and I'm gonna breeze through this 'cause the, there's a fair amount of meat to actually get to, um, click on data sources, those connections to external data sources. This is that, right?

How do I allow you to connect to all of your enterprise data, whether it's structured data or objects, the stores or S3 buckets or, you know, actual NFS shares, different volumes that I have. There's also a shared volume on here that you can utilize backed by GreenLake for file. And so this is a place where teams now can collaborate, right?

And develop notebooks for example, together, right? So, and you can share data. This is how I set my, my example up.

Um, so you'll see it. Um, but when I wanna go add a new data source, right? We've defined a number of different data sources that we can go and connect to, right?

And so, like the easiest ones to understand is like great connect to SQL server, A-U-R-I-A-U-R-I, ah, URIA connection, a user and then a password. I can connect to it and no one else gets to see it. This is mine, right?

But anyone else on the platform can't see my data source. It's just mine. You can do service accounts if you want more broad access, but there's r back throughout the entire platform, right?

So role-based access control over all of my data assets that are in the platform. So if I have access to it, then I have access to it on the platform. If they don't, then they still don't.

So that's fairly probably pretty good on that component. And we'll get into the most interesting pieces, the tools and frameworks. So with the data sources, you were using the um, obviously NBA video.

What did you use to suck that in there? Is that something that comes with it or did you have to do something special? So I cheated a little bit and I'm willing to admit that because I was, I'm, I'm a hacker, right?

And I like to do things quick and dirty. And so what I did was, and this will be I think interesting for you, I was like, okay, I need to go find videos that I can take and I can create a data set out of those videos that's labeled data so that I can do training on it. And so what I did was I went out to YouTube and I used a clip grabber, grabbed it from YouTube, used QuickTime, did a screen recording of it, created compact clips that showed examples of pick and rolls examples of isolations.

And then I took that labeled them with an external tool that I brought onto the platform. There's a tool called Label Studio. Label Studio is an open source.

They have an enterprise version, but it's an open source data labeling tool, right? And so I took it and it's actually right here. I took, and you can see that LS app, a ie.

This is actually running on, on the system. And so you can come in here. The one that I actually did it on just 'cause I was doing things fast, was I did it here.

So you can take these and I put, there's a couple of different folders full of them that I have where I have ISOs, I have pick and rolls are here called Basketball plays. But you can go through and you can label them, right? And so I just went through, yeah, I labeled all the tasks.

So, but at the end you can essentially see this checkbox down here. That's how I labeled them. What I did there is it then gave me a, uh, I exported JSON document essentially that had a unique identifier for the, the video clip and it had a label, right?

And then I took that, I took those videos and I put them, I uploaded them directly to my data storage, um, pane that I was just in. So here or here under data sources, in data volumes under shared, you've got, I put it inside of this folder just for this one. This is actually not the system that I developed it on.

Another one that I'm using or borrowing from somebody. But under data you've got videos. And then in videos you've got all of my different PR is pick and roll or pick and roll is pick and roll.

And ISO is iso, right? So two different classes that I created and then I took that into a Jupyter Notebook that has connectivity to this outta the box, didn't have to do anything there. And I used it to train that slow fast model, right?

So what that actually looks like, and I don't how many of you are like deep and heavy coders, then I will then I will, I'll give a little bit of depth, not a lot of, bit of depth because I don't want to put people to sleep. Yeah, they don't deserve that. No, I understand.

I, I really do. So what I did was I took those that op that open source model and I took the dataset that I created. And essentially all I'm doing with all of this is I'm taking in the two different JSON files that are the doc, the labels and the unique identifiers for those videos.

I create one CSV file all out of all of those that maps to the actual video assets that I have. I do some in imports, right? Some installs some packages and some libraries that I need.

And then I come down, I define what my data set actually is. I create my data set. So I created a label map and this is my read that the pandas read the video label CSV and then I create that data set, right?

So that's it. And then load my pre-trained slow fast model, right? So I load it and I'm not running this because training the model actually takes, I put one GPU in it.

It took about an hour and 20 minutes or so. Um, and it's an L 40 SGPU. That number gets better if it's a better GPU with more memory.

I also did this in a a, um, in a memory safe way. So I didn't wanna blast it. And if I have a situation where I ran outta memory, 'cause I was using kind of smaller GPUs for a fairly intensive task.

Um, I did 10 epics of training. Um, and then so I loaded it, trained my model. Like I said, it took approximately an hour and 20 minutes or so.

And you can see like the actual, you know, like going through the batches and the different epics and the accuracy. It actually hit by like the third Epic. It kind of hit like a, a plateau.

Um, but I just did it 'cause I was testing what could be done with it. Um, so I trained it saved that model creates a PT file in my directory, right? I just told it where to save it.

So this slow fast pick and roll PT right there, that is my model, right? My trained model. And then I can take that trained model and I can go and run against it.

And I can say this is a predict clip function. And so when I run my predict clip function, it goes and grabs. And I have actual examples of this, the ones that I did.

And I'll, uh, I'll show you because I think it's worth it. This is the pick and roll that I gave it. If you saw he picked for his EB Beasley, he picked for him and then he rolled to the basket, got the ball.

One of the limitations that I've seen that I'm working on developing a way around is that it doesn't let a pick and roll can be anything that you do a pick and roll on. You don't have to pass the ball. You can do a pick and roll.

The guy can roll to the basket and I can keep the ball and shoot it or I can pass it to somebody else. It's still a pick and roll. Mm-hmm.

Right? The the structure is still there. It likes when they pass the ball.

The accuracy is better when they actually do make the pass rather than when they don't make the pass. Right? So I'm, I'm working in thinking about ways that I can kind of incorporate that, that knowledge into this.

So that's the pick and roll And an artifact of your training data set, perhaps. Yeah. Yeah.

And I actually, I had to work with this a little bit because I did one where I included other videos that were not of NBA games, like high school games. And it was such a different video that it degraded my accuracy fairly significantly, right? So I had to kind of strip out some of that information and just kept it to this view in particular, right?

Which is again, fairly standard, at least in the clips that I've seen from, um, from like actual organizations where like they have that bird's eye view. And that's a lot of times how they break those, those plays down when they're doing analysis. And then this is the, the iso, it's Mitchell, which he does a couple of crossovers, some hesitations and then drains it on him.

Uh, and so these are the two clips that I feed it. They've never, the model has never seen those two clips before. And so when I, I can run it again right now, but when I ran it, it predicted correctly that the first one, chunk 20 was a pick and roll.

And that chunk, uh, 18, that's just what I had 'em called in my local one. I didn't bother renaming them. Chunk 18, um, is an isolation.

I think the isolation is a much more easily recognizable play because it only involves a single player and that movement is fairly constrained. Um, and so it seems to be much better at recognizing an isolation versus a pick and roll. What was the size of the training set?

It was, uh, and I noticed something when I was on the plane out here. Uh, it was supposed to be 10 and 10, which is a small training set I know. Um, but it's actually one of the most time consuming things is just creating the training set.

Yeah. Um, and so the, I noticed on the plane out here though, that one of them I did not label appropriately. And so it actually ended up being nine and seven because I stripped out some of the ones that were making it perform less ideally.

And then it went to nine on one of them because, or no, seven went from eight to seven because, um, I didn't label one of them. So, but a small training size and actually pretty decent accuracy for what it did. Well, I mean, you're in dev dev test so that you just, it's almost like a mini POC or you're proving the case and then you expand the trainings, then now you promote.

And that was kind of my thought. And so now the piece that I'm very excited about with this is, now this is cool and this is more traditional kind of ml, it's not using LLMs or Elms video language models, right? Not using any of those.

It's more traditional than that. Right? But I think the next opportunity that I have with this is I'm going to take this model.

We have a tool on the platform called MLIS, machine learning Inferencing Service, HPEs. This tool under data science gives you the ability to create registries of models and then create deployments of those models, right? So I'm gonna take this and I already did this component, this piece of it, I was trying to get it for this presentation.

I, I ran into a couple of hiccups 'cause I don't, I've never used Bento ML before. Um, but I'm gonna package it up as a Bento file or a Bento ML file or model and then deploy it from an S3 bucket that I stuck it in with MLIS to create an endpoint. And so now when I create that endpoint, I'm gonna create a UI for it.

And what that UI will do is it will accept an MP four, it'll store that MP four on the platform and then it'll run that MP four against the endpoint and then it'll predict what that is for. Now if I combine that with the tool, maybe like airflow, if you're familiar with airflow, great for like workflow scheduling and automation. So like I always use the example of I'm 37 years old and I still am terrible with Celsius.

I'm like, Fahr Fahrenheit just makes sense to me. Right? So if I was reading in Data from a weather API and it was in Celsius and I wanted to trans wanted to do this every hour or every day, and then I want to do a transformation to take it from Celsius to Fahrenheit and then I wanna post it to a database, a great type of job that I can do in airflow, right?

To create that, that workflow so that I don't have to do anything. But if I can use a, a workflow tool like that, I can build a pipeline for all of the video assets that they have to chuck them against my processing endpoint to do all the classification. And there's some data indexing concerns in there that I'll need to address eventually.

The last mile though is I'm gonna take it and I'm gonna integrate it with a video language model to give me more information about the clip. Mm-hmm. So some of these really cool VMs to describe to me what's going on.

So, and the idea being that I'll be able to, in that endpoint have a chat with an LM and say, show me every instance of Steph Curry running a pick and roll in the fourth quarter of a game in 2017. And assuming we do all the data side of it right, and all of the indexing and all of that, then it'll be able to bring up the clips for me. Show me this is all of those clips that you requested.

So was was the goal thing here, uh, with the small data set was to go and train it and then in theory you say Steph Curry doing pick and roll, it would then be able to go through all the data and and start pulling those all out. Yep. Okay.

I wanted to show a position for that. I built it for, for a customer, right? For that NBA team.

Just 'cause I wanted to show them what could be done because it's, it's easy to like think about things kind of like theoretically it's a different thing to have. Like this is a real model that's trained on an open source model that can go pull down right now. And we took it and made it do something that's very interesting, right?

That could actually make a difference to their organization because they have an N-A-W-N-B-A team too. And their biggest goal in the organization is how do we get our WNBA team A championship, right? Using tools like this analytics that maybe they have lower budget for.

If it can be jointly used for both the NBA and the WNBA team can get them a lot closer to having the types of analytics without the giant analytics staff that might be required to build that type of stuff. So you did the 10 and the 10 training and then did you run that uh, on a wide database or is that the next step after this? That's the next step.

I just did some training on, I did the training and then I took some, again, clips that I hand kind of curated myself. I've pitched probably eight to 10 clips that it hasn't seen before. The accuracy on all the ones where they actually do pass the ball pretty good, right?

Isolation's great. It can recognize an isolation. I can do it against others too, right?

So I can do like a screen for example. Um, I haven't created a screen, uh, as a play type in here, but as an other it'll recognize that that is not um, an isolation. So isolation, really good accuracy.

It can roll pretty good accuracy as long as they pass, as long as they pass. So that was kind of, um, my position on how I could use this system. And if you start to take into account what can be done with MLIS and then what can be done with helm charts to create UIs.

'cause I can build a UI in a helm chart pretty simply, right? And use that we've built as part of our platform, we've built this Gen ai, uh, which used to be uh, a little bit different. Um, it used to be called solution accelerators.

And a solution accelerator was a collection of kind of these different microservices that we all stuffed together that, you know, achieved a use case. And so the first one we had was a rag essentials use case. And so that rag essentials use case was we'll give you a space to specify these are the documents that I want to use, right?

This is the model, this is the embedding model that I want to use. This is the in or the, uh, other model that I wanna use. And then you click go and it goes and has, you know, has deviate under the covers as a vector database and it goes and uses that embedding model to create those vector embedding puts it in there and then you can run the similarity search against it and you can ask questions.

And so it's a very simple, quick and easy way where I can go and do something that might actually be meaningful. But it's a package on the platform that you can go and retrieve from Heart Museum, which is a heart repository helm chart repository on the platform. You can retrieve that and you can edit it, right?

So if you want a custom UI right, or you want a different set of resources underneath it or you want whatever, you can do that. So it's a springboard for people to do their own development on top of it. So that's kinda what we did there.

And we're gonna give this a, a whirl knowledge base. We're gonna do. And I, my apologies, I thought this was set up, but this, as I've mentioned, this is not my, uh, I was gonna show the N-B-A-C-B-A, um, this is not my, my environment that I traditionally use.

Six one, sorry for the uh, flight wall in communication. I have a zoom thing in my way that's, there you go. Either way I'm gonna upload a file directly and apologies you're gonna see a little bit into my, uh, into my computer here.

This is the one thing that I didn't do 'cause it was already built on my other system. Search Davis. So this is the collective bargaining agreement for the NBA and I have to move this zoom thing again and I'll hit upload.

I'll select an embedding model model. We use that embedding model, use that Mistral one. These are just templates.

This is part of what comes from, um, we have auto scaling on the platform, which I think is very valuable, especially against like the cloud sometimes. Um, or doing it yourself. The idea being though that with MLS and with how we've deployed these, you have auto scaling here all the way to zero.

So if a model is not being used or an endpoint is not being hit, go ahead and scale it to zero when it retreat, when it receives another request, it'll go and scale itself back up and you can set scaling parameters, right? So how much it can scale to before it needs more. So this is just, and there's other pieces inside of here around chunk size overlap, right?

So temperature is another in LLMs, right? That's kind of how much freedom you give it in its response, right? So is it hard line, this is specifically exactly show me just the facts or do I give it more creative freedom, right?

So there's other configuration parameters that you can give their system prompts that you can tell it, this is how I want you to act. I'm gonna minimize this. I hope that doesn't mess anything up 'cause it's been in my way for quite a while.

And then yeah, that is all I have to do to go and set up this knowledge base, which will is essentially is a RAG implementation, right? And ideally in the next eight minutes this will spin up and I can show you how we can go and query against the collective bargaining agreement for the NBA. That's all it takes.

So When you talked about you're not seeing value and people like actually being able to realize value, I've done this with customers, they're like, that's all we want. That would be so valuable for us because it would save our team members so much time and having to go and find information or it would save us a ton of time and a ton of costs and having to hire a team of lawyers to answer questions that we may very well be able to answer ourselves. It's just we kind of hate legalese and the wording of these things 'cause it is a headache to read.

So back to the, the hurdles to getting there. Those, those are the ones on the right of the step chart because as you say, there are some real concrete benefits or, or use cases and yet they're not proliferating and those are the reasons that you listed before. I think a lot of times people just don't know that, you know, like unless you have someone who's really intimate with the space, you're not as familiar with what is out there tooling wise, right?

Especially things that are low code, no code. That's huge. Right?

And this is part of the experience that we've built here, particularly with these solution accelerators, particularly with the NVIDIA blueprints. We wanted to provide quick wins for people, and this is one of my favorite components of this platforms is that every customer that I've talked to has been so interested in this because they have a use case for it. A lot of times they are able to kind of concretely say, I think this will be save us X, right?

They're fairly able to communicate what they think this could do for their business. If they can see a win in this and then defer, use it to gain value and then defer cost for some of their later more stretch development efforts, they start to get really excited about it because everybody's got the long shots, right? The type of stuff that I'm doing with that, you know, NBA classification kind of a long shot, it's more development, it's, you know, kind of an unknown space, but it, it could be really impactful for them.

And most customers that I'm talking to are like that. They're like, yeah, we see some quick wins. If we can get value outta the quick wins, maybe we can hit some of our stretch goals without having to, you know, come out of pocket for it essentially because we've gained so much value upfront with the quick wins.

How can you gain, uh, gauge the accuracy of it? Is there, are there hallucinations that you have and say, show me all the click So you can This and it pulls up things that aren't there? Yeah, you can, and I think it's actually, we've done it in a a a you, it's a parameter that you can kind of set on the back.

I think we've actually put it in the ui. I'm not sure about this version, but you can make it site for you, right? I can make it site the paragraph and page for me.

So like in the N-B-A-C-B-A when if this pulls up in the next five minutes and eight seconds, I always ask it the same question because I know exactly where it is in the document. Um, it's how many, uh, appearances are players required to make during the season and it answers like six. And then I ask it one further question.

That further question is how many of those have to be with season ticket holders? The answer is two, that's on page 48 in section eight of that 600, 700 page PDF, right? Can You actually hyperlink to it like the way that GPL will do to It?

Yeah, you can. I it's not implemented in this, but yes you could. So gonna make a liar outta me.

'cause maybe should have assigned more GPUs to it because you know, you'll, I'll spin it down in just a second here. But that's the, I think the purpose of it. I'd love to take some more questions.

We have about four minutes left. What happened to the fatigue, uh, pro problem? Did you just using that as an example of something they do want to do that would work well with this?

Or did? No, That was just an example that they thought of that got me thinking about things that could be done with video, right? And I think fatigue is, uh, probably if I really want to approach that problem, I think I really have to work with them because they're the ones who have all of the video assets, right?

And they do have some kind of like data management, data indexing. Like they, they have an idea of where some of their assets are and so it would be a lot easier for them to turn me onto a larger set of data that actually would show that information. My ultimate goal though is I want to gamify this because I think that would be like the ultimate cool thing where it's like, if I can take this and say, great, based on these players on the floor in this given situation, what is the expected value of running this play in this scenario?

Right? If I can take that, and that's part of why I started setting up the play classification because I wanted to say, okay, this play, and part of this that I also set up that I didn't show, there's a video chunker. Now if I can go through an extended clip that I, and I built a video chunker and it does it by, um, it does it by like two different ways by play, right?

So by possession change or by, um, scenery change, right? So like clips, if you have a highlight clip reel, right? I can do, is it a scene change or not?

And I can create chunks of that video. So every time it sees a scene change or a possession change, it clips it and creates, you know, an atomic clip, right? So that gives me kind of the start of I've taken all of those little, those long running videos and I've put them into manageable clips that I can now classify.

Did it result in a score or not, right? And then I can go further with who was on the floor ultimately, hopefully leading towards that gamification effort where I can say, great, what's the expected value with if you bring your sixth man in and sit your, you know, your two guard, you know, what's the expected value given these conditions? That's ultimately what I'd like to move.

Wouldn't the most value from that actually be able to analyze it real time during a game. Mm-hmm. For sure.

Yeah. Well, yeah, the, the next thing is a headset for Jordan Ott that, uh, you can just, you know, yeah. Tell him what to do next or just take his place.

You could take yeah, take Moneyball to, uh, put on steroids as far as who do you wanna put together the less best amount of value that you can put players together that would work better as a team than individual stars and what are their salaries and be some pretty cool things you could do with it. Yeah. We're, we're gonna, This is Actually their AI's final contribution will be to revolutionize the game of basketball.

This was actually one of their ideas. They wanted to be able to understand who they should go out and acquire to make the most compelling lineup, right? So that was definitely one of the ideas that they were thinking about.

But it is, it's like Moneyball just with basketball. It's very interesting to me because like when you see, uh, like the NFL, if you watch the NFL, they always have, like, that type of pass hasn't been completed in 24 years in that given situation powered by AWS, right? And I would've thought that all sports teams would've had kind of a similar level of analytics, but what I've kind of found in beginning this effort was that it, it, it really, some of them don't, there's still a ton of really interesting opportunity and being that AI is on the rise, um, trying to put some cool stuff together for that space.

And as new rules are introduced, who, which players are gonna do better and which players are gonna do worse? And how do you reshape your team to take advantage of it? Possibilities are endless, fascinating.