Introduction to Amazon Bedrock (feat. Ramya Ganesh)

Amazon Bedrock is a new foundational AI product from AWS. Bedrock offers a catalog of models for various applications.

In this episode, host Krish Palaniappan welcomes back Ramya Ganesh to discuss Amazon Bedrock and its applications in AI and cloud computing. Ramya shares her extensive experience with AWS, particularly in cybersecurity and AI, and explains the differences between Bedrock and SageMaker. The conversation delves into practical use cases, such as code generation and architectural diagrams, while also addressing the challenges and considerations when integrating Bedrock into existing applications. The episode concludes with insights on prototyping with AWS AI tools and the future of AI development. In this conversation, Krish Palaniappan and Ramya Ganesh delve into the intricacies of using AWS Bedrock for model selection and application development. They explore the open-source nature of certain applications, the importance of selecting the right model for specific problems, and the nuances of model configurations. The discussion also covers how to compare different models and the next steps for integrating these models into applications.

Takeaways

• Amazon Bedrock is a new foundational AI product from AWS.

• Bedrock offers a catalog of models for various applications.

• Integrating Bedrock with existing applications is straightforward.

• Use cases for Bedrock include code generation and architectural diagrams.

• Diagrams provide better understanding than textual code alone.

• AWS offers various services for AI, including SageMaker and Lex.

• Choosing the right model in Bedrock requires research and understanding.

• Prototyping with AWS tools can lead to production-ready applications.

• AWS's Party Rock allows for quick AI application development.

• Collaboration between different AWS services enhances functionality. Open source applications can be created easily with prompts.

• AWS Bedrock allows for model selection based on specific use cases.

• Finding the right model involves research and understanding of available options.

• Model configurations can significantly affect output and creativity.

• Temperature settings in models influence the randomness of responses.

• Comparing model outputs helps in selecting the best fit for a problem.

• Integration of APIs is crucial for deploying models in applications.

• Understanding the limitations of AWS Bedrock's model search is important.

• Practical experience is more valuable than theoretical knowledge in AI.

• The next steps involve deploying the selected model and integrating it into applications.

Podcast

Transcript

Krish Palaniappan (00:01.259)
Hey, folks, welcome back to Snowpal Polyglot Software Development Podcast. Our guest today is Ramya Ganesh. Ramya, happy to see you back on the podcast. Thank you so much for taking the time.

Ramya Ganesh (00:12.792)
Thank you and so glad to be back here again, Krish.

Krish Palaniappan (00:15.895)
Yeah, I know we have a bit of a time crunch, so I'm just going to get right in. But before we do that, in the event that people haven't watched or listened to the previous one just as yet, if you could give a quick brief intro about yourself, that would be awesome.

Ramya Ganesh (00:30.03)
Sure, so yes, so I work in Cisco. I've been working in cybersecurity, AWS cloud for the past 10 years. I've been exploring AI in AWS for the past one year. I've done a couple of hackathon projects using AWS and AI.

so that's, that's where like, you know, exploring bedrock and SageMaker and everything started. And I really have been liking, the way, you know, AWS has been used to spin up, applications. It's it's, think it's pretty good. So today I'm here to talk about, SageMaker and, bedrock. And I've been exploring a little bit of agents too. Maybe I can just talk about a touch about touch upon agents as well. you have time a little bit. Yeah.

Krish Palaniappan (01:19.575)
Okay, cool. That's amazing. I'm gonna need a lot more time to be honest from you to go through all of these items. You can do that as you have more time. But I'm glad that we kicked this off, right? The first of our tech podcast. I know there was a bit of scheduling. We tried to get this to go and I'm happy that it's happening more than anything else. So much appreciate you sort of squeezing this in your busy week. I know you have to go drop off your daughter in like 45 minutes. I wanna...

Stick to that and Ramya, if you are able to pop in after you, know, sort of you're waiting in the car, feel free to jump back into the podcast. I most likely I'll still be here trying to figure some of these things out. Okay. And folks by way of a quick introduction to Mike Niren's about Bedrock. If you remember a little while ago, let me do a quick screen share here.

Krish Palaniappan (02:16.455)
And this is the screen share. And I think if I go really quickly, it was maybe about a month ago.

Krish Palaniappan (02:27.467)
Yep, the AI Explorer series, as I called it, one, two, and three. I think it's about three hours long. haven't watched it and you have nothing better to do and you have three hours. You may want to check that out. It's me walking through every AWS AI product with me having absolutely no idea about any of them. So I did not even know the names of those products and services before I hit record. So the whole idea behind those three hours was me as an ignorant AWS AI developer trying to figure out, just navigate, maneuver.

and see what's out there, right? And then make a lot of speculations and assumptions about each of those products. So that in itself took like three hours. And I think I spent like five, seven minutes per product just trying to read and figuring out what it possibly might be. Only as we dig deeper into it, I'll know how far off I may have been from the reality of what the product offering was. So this is a continuation to that.

So it's one, two, and three. I think this list has a bunch of those. And number three, says Bedrock is a foundational AI product. And I probably spent five minutes on it literally, right? So let's jump right in. I'm going to do just that. Let me pull this to a different window.

this to a different window so I can share something else. Perfect. So let me have the first question, as somebody who's not used this before. There are a number of different AWS AI products. And let me make sure you can see all of my screen.

Krish Palaniappan (04:02.709)
AWS products. If I go to AI. Okay, there are a number of things here.

Krish Palaniappan (04:14.167)
to that page.

Okay, I'm just gonna stay here. Yeah, so this is a list of products. And like I said, in the first series of three quick podcasts, I just went through the list to see what it was. And trying to understand the difference, AWS has just like every other hyperscaler, they have a large catalog of every product. A lot of times they have redundant products when it comes to like databases or anything that you wanna do. I've seen that you have seven different options. Some AWS may have created themselves. Others, they probably acquired one way or the other.

But there's more than one option. And one of the initial challenges is to figure out which one should I go with, right? Even after you know you sort of have an idea. With that as the context, let's kick this off here, right? What is Bedrock at a very high level? And how is it different? And I'll tell you what my understanding was. And it's from a month and a half ago, and could be very wrong to begin with.

Ramya Ganesh (04:55.352)
Mm-hmm.

Krish Palaniappan (05:06.665)
As I went through each of these products, I realized that some of them are sort of vertical offerings. They had a specific purpose. Like there were some AWS AI products for the healthcare industry and some for some other industries specifically, right? Whereas Bedrock seemed like one of the other ones like SageMaker, which I believe is now called SageMaker AI after the rebranding. Some of those came across as a bit more generic. They were foundational horizontal types of products, if you will.

Is that right to begin with? as part of the question is, what is Bedrock and when might one want to use Amazon Bedrock?

Ramya Ganesh (05:41.42)
Yeah, so Amazon Bedrock is something actually fairly new, like SageMaker has been there for quite some time. It's been there since 2017, since its inception. But Bedrock was there in the market at 2022. So it's very fairly new. But the key difference between SageMaker and Bedrock. what is actually Bedrock? Bedrock is when, so you have, it's kind of like a.

Say, when you log into Amazon Prime, and if you want to watch some movies, you have a catalog of movies. So similarly, Bedrock has a catalog of models. It has like hundreds of models and different LLMs which are offered by Anthropic. Amazon actually has its own LLM which is called Titan. So it offers loads of LLMs, foundation models that you can see.

So, and this one is like, if you have, it's very easy to integrate a model with your other applications that are running in cloud. And because it uses a very flexible API. So if you see like, there's a certain use case that you're trying to address, and you would see here, you would go to the catalog and you will see like what each of these models are used for. So there's a basic research that is required.

Krish Palaniappan (07:07.361)
Hey, Ramya, sorry, I'm gonna interrupt as we go, is that all right? Yeah, that way, you know, I wanna learn as much from in the next 40 minutes, right, as anybody else watching this as well. So again, like I told you, I've not built anything with either Amazon Bedrock, right? So I'm a beginner from when comes to a consuming Amazon Bedrock standpoint. So my first question is, what are some of the use cases? In other words, from my very high level read of it,

Ramya Ganesh (07:07.938)
to see if that.

Yeah, go ahead, go ahead.

Ramya Ganesh (07:19.446)
Mm-hmm.

Krish Palaniappan (07:35.831)
And you mentioned the catalog, Prime Video as an example as well. Text generation, image generation, and those types of things are potential solutions. But outside of that, one of the things I want to look at was just to step back. What does it mean to create a full stack web app using Ambedra? I even know it was possible. I just Googled this before I hopped on this call here. And I stumbled into it. And I figured I'll keep this open and probably it'll guide our conversation. So in other words,

As somebody running our startup, I'll ask you, it's not a manufactured question. It's just a real question. So I may or not have socialized some of what we do, but let's say we have a mobile app, we have a web app, we have APIs, and we have a bunch of different things. I want to see that as a company that's been doing this, but without using say Bedrock, how could we potentially integrate Bedrock? A simple example is a web application that we have, which is a project management platform.

We have a mobile app, is essentially a project management platform with a mobile interface to the same set of functionality. We have a rich set of back-end APIs built in a number of different languages. They're deployed on the API gateway, and they're deployed on AWS Marketplace. People can subscribe to it and consume and start building their applications however they want to be able to build it. This is just a quick background of what we do, and the answer could be, Krish, it's not related to any of that, which is totally fine, too. I just want to know, as somebody who's not used it,

Ramya Ganesh (09:00.45)
Mm-hmm. Mm-hmm.

Krish Palaniappan (09:02.763)
Where would this kind of fit in in this equation of what I've mentioned here?

Ramya Ganesh (09:07.34)
Yeah, so I can talk about how what the application that I have built Krish and maybe you can fit there because you have like other integrations in place, right? So for the application that I used with Bedrock was it was like code generation. So basically, say for example, you have code in different repositories and so the use case was pulling the code from different repositories, different files.

Krish Palaniappan (09:11.991)
Sure.

Ramya Ganesh (09:35.646)
different PDFs and converting them to an architectural diagram. So that was what the use case was.

Krish Palaniappan (09:41.426)
Say that again sorry Ramy can you repeat the use case one more time

Ramya Ganesh (09:44.14)
Yeah, so it was like, for example, you're a developer and you're new to the team and you have no idea about the code base. So there's tons of code base that you will have to see, you'll have to check different PDF files and everything. So this application would basically pull the code from different repositories. It will pull, if you have like Kubernetes files and different PDFs, so it will pull all the documents.

and it will basically convert that to an architectural diagram. Basically, it will tell the user what the flow is. Like you have each number of microservices.

Krish Palaniappan (10:21.079)
Okay, that's lovely, right? So I have questions, Arun. Perfect. So say at Snowpal, we have a number of different repos on GitHub. And now those repos span across different languages and different frameworks within the same language as well. So let's take just two. We are a polyglot shop, but let's just take two. We have a number of Ruby repositories and we have a number of repositories built on Golang. Now what you're saying is a new developer joins a team or a product manager, doesn't matter. And they sort of want to...

Ramya Ganesh (10:25.294)
Thank

Krish Palaniappan (10:49.225)
get a feel for the environment, if you will. Just trying to understand what this company does. It's almost like analogous looking at your tickets. When I pick up a new project, even if nobody told us anything, if you just go look up all the open tickets, issues that are open in whatever bug tracking software that you use, it gives you a good sense of what the team is doing, where the problems are, what features might be, where there might be bugs, and so on and so forth.

Ramya Ganesh (10:54.744)
Yes.

Mm-hmm.

Krish Palaniappan (11:19.433)
In this case, you're saying, let's take, you said multiple repositories, but let's take one repository. does, but does this use case only make sense for multiple repos or does it also make sense for a single repo?

Ramya Ganesh (11:29.614)
makes sense with multiple reports or if you have a single report that's fine too, but it will apply to multiple reports as well.

Krish Palaniappan (11:36.073)
Okay, let's take a single repo, right? Let's say I have like one of the APIs is built in Golang, for instance, it's got Golang files, it's got some read me and whatnot. Now I have this repo, what you mentioned the architecture diagram. So let's say in this use case, a new developer joins the team. Is the idea here to have them get more acclimated to the code base in their repository?

Ramya Ganesh (11:46.648)
Mm-hmm.

Ramya Ganesh (11:59.2)
Right, so we have like hundreds of files there, right, in the repo. So it is very difficult for the developer to understand, okay, so this is how the flow is, and this is what the code looks like. So...

Imagine if the whole code base can be converted to architectural diagram where he gets a better understanding of how this microservice is like, this is how the code is written in this microservice. That's how the code is written that one and how it interacts with the different microservices and Kubernetes, AWS and all. So that's what it is about. you have different LLMs, which actually does code summarization. this actually enhances a little bit where you...

It does code summarization, also pulls it from the repo, and also we convert it into a diagram basically. Just make the, it will have the whole flow from start to end to see like, okay, how the various components connect to each other.

Krish Palaniappan (12:56.375)
I think that's a good use case, right? That's a good starting point. Now, once we figure that that is what we want to be able to do, sort of generate a diagram of a single repo. Let's keep it small, right? Let's say one repo and I want to create a visual representation, an illustration of the code. Before we go through what it entails and we end up going through the process, I want to kind of understand the end result. You know what I mean? Just so that way I know what to imagine. So the end result.

Ramya Ganesh (12:59.95)
Mm-hmm.

Ramya Ganesh (13:12.6)
Mm-hmm.

Krish Palaniappan (13:22.387)
If we were not using Bedrock to do this, let's say if a human were to do it, what would the end result be? You mentioned an architecture diagram. So it could be an application architecture, could be a database architecture, could be security architecture. It doesn't matter what it is, right? So it's a diagram with a bunch of boxes, circles. And if you're a developer like me who doesn't draw well, but likes to see visually, every diagram has only like three or four different shapes, right? I only squares rhombus rectangle in a circle because I don't

my creativity sort of stops there.

Ramya Ganesh (13:52.714)
Everything actually gets a better understanding with diagrams than just looking at a code, right, which is actually the all textual base. So.

Krish Palaniappan (14:00.221)
Absolutely, right? Totally. So the picture is certainly worth a thousand words, right? So when we create this, let's say we took a single repo in your example, the use case that you had mentioned, what did that diagram end up looking that? I mean, it's a very high level abstract question. How detailed was that diagram? In other words, was it along the lines of what a human would have generated? Like, you know, I'm just trying to see, maybe I'll begin to pull up an example here. I want to kind of imagine what the complexity or the...

magnitude of the diagram was, how specific was the diagram?

Ramya Ganesh (14:31.63)
So for our use case, because it was a hackathon, it was done in like a 24 hours ride, so this was prompt engineered. So basically we took a single repo use case and it was prompt engineered such that you have the input files already pulled from a specific repo and it actually will produce a smaller diagram, which is actually, will just show the entire flow.

So just imagine like in a repo you have like two, three files, have your Golang code, two, three files, and then it will just generate a diagram which will show like, okay, this is what the code base is, this is what the components, this is how the various functions or modules are linked to each other. So that's how the diagram gets generated at the end.

Krish Palaniappan (15:15.542)
Okay.

Ramya Ganesh (15:19.374)
And this was prompt engineered because fine tuning a model would actually be a little bit. But since we are already doing, since we are using a repo, which is like, you know, more like an in-house repo, we have to create a rag and a vector database for it. So, so, but it was really easy to integrate everything in AWS because AWS was, it's much faster because to integrate them in AWS because

Krish Palaniappan (15:36.779)
Okay.

Ramya Ganesh (15:48.45)
We use a hugging phase function for the rag and vector DB. And for bedrock, was already lots of LLMs, which actually does the code summarization and all. But it's the research that took a longer time. Because if you see, even in hugging phase, if you see, there are so many models out there.

Krish Palaniappan (16:11.351)
Yeah, it's funny you should mention a hacking phase. actually seen it one time and I want to sort of let me pause you there, right? So just to, from what you've said so far, this is the use case. We have a repository, one or multiple repos, doesn't matter. We want to be able to generate a diagrammatic visual illustration of this. That's our requirement. That's all we aim to do here. Now let's go just at the high level. Let's see what this entails. I think that's small enough a use case to sort of digest the problem. So we go to bedrock is able to do

But just one question before I go there, which is, the repository has, you mentioned PDFs, like different kinds of files, right? It has actual source code. It has configuration files. has PDFs and maybe we have diagrams or a number of different kinds of assets that the repo comprises of. The idea is for this bedrock, this exercise to.

go across that, right? Not just text, not just visual, not just images, go across all of them and essentially come up with a diagram that is also sort of inspected the images that were in the repository too, correct? It's not just the text, right? Okay, perfect. Now, first question is, not first, one of the fifth or seventh question, whatever it is, is I've seen some of the AWS AI products that actually have to do specifically with image generation or image analysis and text analysis.

I don't remember the names, but there is that list there that has a number of them. But when we say we are using Bedrock, does it mean are we just using Amazon Bedrock and that is it? Or would we actually be near exercise? Did you use Bedrock alongside a number of other AWS AI products to actually get what you needed done? That's my basic.

Ramya Ganesh (17:56.11)
Yeah, so it's basically S3 where you would store, you know, it's everything is like AWS related. So it would be S3 where you would store your documents, your reports, you would pull the reports from there. And that's pretty much it. And then hugging phase integration with AWS. So everything was actually inside the AWS.

Krish Palaniappan (18:15.799)
Okay, but what I'm asking is, which means, you know, forgetting S3 for a second, if I go to this list here, yeah.

Ramya Ganesh (18:18.286)
Mm-hmm.

Krish Palaniappan (18:28.789)
Is this maybe this is the list looking for that?

Ramya Ganesh (18:41.23)
So are you talking about comp?

Krish Palaniappan (18:42.585)
Yeah, I just want to know that, you in this list here, I ran into some that were specifically text or image related. My first, my question was, did you use any other AI product in this list alongside Bedrock or was Bedrock the only thing that you had used in your exercise here?

Ramya Ganesh (18:45.442)
Okay.

Mm-hmm.

Ramya Ganesh (18:59.45)
is like so powerful, like we just use the bedrock, but nothing not like the Amazon queue or whatever, but it was just mainly S3.

Krish Palaniappan (19:10.103)
But that's outside this list here, right? which is very, I think I answered my question. But the reason I asked that is one of the challenges that almost everybody has when you're picking up something from a catalog I mentioned, right? Like I mentioned the example of database when you want to use an AWS database, do you use document DB, do you use DynamoDB, do you use one of the other things? Because there's like seven different variations that are all similar in some ways and different in other ways. So navigating the catalog for picking and choosing what you want to use is one of those first items.

Ramya Ganesh (19:16.898)
Okay.

Krish Palaniappan (19:39.071)
So if I wanted to do what you just mentioned, generate a diagram from a piece of code, a repository is bedrock the only option or are there other AWS AI products that I could possibly be using that's not bedrock.

Ramya Ganesh (19:54.616)
So you could also use SageMaker, but SageMaker is like a total ecosystem, right? Where you would be like a building models and you can deploy models and everything. But in this use case where you have already LLMs in place that are very good with code generation, anything to do with code, right? Code summarization and all. So in that case, this specific use case, Bidrock would be more easier to use than use SageMaker.

Krish Palaniappan (20:20.339)
Okay. Okay, cool. maybe if I remember before our session ends, I'll come up. I thought I'd seen some of the image generation AI product here. And that's the one I'm thinking of what I just don't know which one it is, but that's all right. So we go to Bedrock and this is our use case. Sorry, go ahead.

Ramya Ganesh (20:35.406)
But AWS does offer lots of services, like functions that would convert text to image. don't know if that's what you're referring to, but those are called Amazon Lex.

Krish Palaniappan (20:51.351)
yeah, see there's a, yeah, exactly. And then there's a text tract here that say extract text and data from documents, which means, so one more time, the same challenge does exist, right? I don't know what the history behind text tract and say bedrock is. Is it?

Ramya Ganesh (20:56.91)
Practical, yeah.

Ramya Ganesh (21:04.204)
Yes. So text is basically a function that would convert, it's like text to text. But if you have like a document, right, or that would convert to speech, like I think Lex converts text to speech. And there is something called poly that would convert, you know, text to image. So AWS does have a lot of functions which does that.

Krish Palaniappan (21:28.927)
Okay, so in other words, we could either go with Bedrock or we could pick one or the other. The same challenge still does exist here, right? So for the use case that we're talking about here, clearly what I'm understanding is Bedrock is one way to go about from an AWS catalog standpoint, but certainly not the only way to do this, right?

Ramya Ganesh (21:46.062)
But that is for if you're going to use a model or deploy a model obviously will need a model for this for sure right for this use case people definitely need a model. So we will be using by drop for example, let's say in this use case right where you are generating text from image from text you're generating an image basically code to a diagram right so now I have

Krish Palaniappan (21:53.142)
Right?

Ramya Ganesh (22:08.418)
chosen a model and which can get the code from the repos and the model is deployed. I'm able to infer the model. I'm prompt engineering it. So I'm putting a prompt and it's giving me in a text form. But the next challenge for me is how do I convert this text to image? That's where I will use the AWS function, which is the Lex. So I would integrate Lex into the output of this model. So it converts whatever text it gives.

to an image if that makes sense.

Krish Palaniappan (22:38.497)
So which means just to be clear, even when you're using bedrock, let me go pick up his legs in this list here.

Ramya Ganesh (22:47.822)
Yeah, if you're still for likes in the search, I you should be able to pop it. Yeah, I'll leave it at that.

Krish Palaniappan (22:48.052)
Maybe.

Krish Palaniappan (22:54.281)
Is it legs like LEX?

Krish Palaniappan (23:00.831)
Okay, so in our use case, you built it, you used Bedrock and Amazon legs specifically? Like you intentionally used that or was it kind of behind the scenes that Bedrock was leveraging legs or something like that?

Ramya Ganesh (23:02.798)
Mm-hmm.

Ramya Ganesh (23:14.37)
Yeah, so for this use case, we kind of stopped at a point where we got from the repo, we got an output from the model. But the next step was to integrate AWS function, which converts the text to an image. Basically, you can also do it in ChatGPT as well. if you're ChatGPT, if you just put a output.

Krish Palaniappan (23:36.789)
No no forget Jayaji, would bedrock let me do that without using legs or did you have to use legs to accomplish that?

Ramya Ganesh (23:43.436)
have to use a function, a different function, which will actually convert the output to an image. There are models which will allow you to do it, which will convert, you know, there are many models actually which converts text to image, but remember that those models will have a lot of GPU conception. Okay, so the resource utilization for those would be really high because it is, you know, processing an image.

Krish Palaniappan (24:04.791)
Great.

Ramya Ganesh (24:11.278)
So that is also something that needs to be considered. So if you're just doing a home project and you'll not be able to spin a model that is like an extra, extra large GPU, right? Because of the price constraint. So in that case, I think it's better to go with a model that will do like just a text generation and just use a function to convert the text into an image.

Krish Palaniappan (24:21.397)
Right? Right.

Krish Palaniappan (24:31.711)
Okay, makes sense, but the thing I'm also trying to understand here is, tell me if my understanding here is more or less accurate. If I wanted to be specific about which, like if I go to Amazon Bedrock, one of the first things I see is a model catalog that you mentioned as well. But if I were to be not very specific about the models that I was interested in, if I went to something like say Amazon Text Rack, my again, reading this two minutes,

I'm assuming and tell me how much of this assumption is correct and how much is way off. That this is also using some LLM, but perhaps I'm not able to pick and choose which model to use when I'm going with Textract because Textract is implemented with certain models, perhaps behind the scenes. Whereas when I use Bedrock, I'm able to sort of hand pick which LLM I want to start off with. Is that correct or wrong to begin with?

Ramya Ganesh (25:21.896)
Right, It is correct. So in Bedrock, when you are in Bedrock, will be like, you know, you will have, there is a research that is needed to choose, to pick the correct model for your specific use case. Because even if you see like that a lot of models that Bedrock offers, which is the Cloud Sonnet, most of the Cloud Sonnets will do code processing, code summarization.

Krish Palaniappan (25:45.975)
Okay, so I think you kind of answered my question. So what we are seeing here then Ramya is if I wanted to say extract, I don't know, text from any document, seems like, I mean, when we do this hands-on and we get to that point, we could probably compare and contrast as well. You could use text track, which means it probably is going to be a much faster exercise, meaning it probably has some fundamental assumptions and it uses some models and it is going to probably not ask you a whole lot of questions. Whereas when you're doing, going with something like Bedrock, then you kind of...

you're doing a lot yourself, meaning you're first figuring out which model should I be using as opposed to using some other AWS product where the model is perhaps embedded or transparent to me as a developer. I mean, is that correct?

Ramya Ganesh (26:28.278)
Yeah, correct. But I just want to add something for more clarification. So all the ones that we just talked about, the Textract, Lex and everything, it's kind of a function that AWS offers that you could use to process your model outputs. So if there is any output that, you know, like you have.

Krish Palaniappan (26:46.113)
But when you say function, it's not function as I'm thinking of a Lambda function, right? That's different. This is still its own product, right? So just want to make sure when you say function, what do you mean exactly in the context of if I think AWS Lambda, it's a function in that sense. So what do you mean when you say this is also a function?

Ramya Ganesh (27:05.538)
It's the same like Lambda. it basically, it is a service that AWS offers that will actually create, like if you have a use case where you have to convert like text to speech or speech to image or say anything like that. There's also contact center. There's also different functions that, not functions, I would say models that AWS offers that will actually create a contact center for you actually. So, right, so it will ask questions. will like all the fine-tune models for contact center.

So all these are actually services that AWS offers. So yes, so you can either use those services or you could come to this model and you can either fine tune the existing models or prompt engineer it.

Krish Palaniappan (27:46.583)
Okay, perfect. Right. Okay. We can chug along here. I'm sorry, that took a little while. I just want to digest the first question, which is, should I be picking this? Why should I pick this? if, cause you know, a lot of times if, you know, if you're a startup, at least in my experience, the idea is you try to do a prototype as quickly as possible, and then you see whether it makes sense or not. But as it turns out in reality, if you get something working in a prototype and you've made enough progress, I've hardly ever seen the case where you've gone back and

analyze something else and try to do and compare and contrast because you just don't have the time and money. So if you did a decent enough prototype with the first product, nine or 10 times, we've seen that that is what ends up going to production. So it's less to say that this is a tool we use for prototyping versus you know what, if you did a decent enough job that is literally going to show up in production. So you better spend those initial hours and minutes sensibly, which is kind of why before I spend the first three hours on anything, whether it's AWS or outside of it, personally speaking, I...

I wanna make sure that it's worthwhile, if not for any other reason, because sometimes you get it working as an engineer and then you don't have the heart to go back or the time or the money. you're like, you know what, if DocumentDB does what I wanted for it to do, even if it's not perfect, I'm gonna just go with it, right? You just end up going with it because it's a prototype and then move on, right? I think you've answered those questions. So.

Ramya Ganesh (29:00.686)
Yeah. And AWS does offer, I don't know if you have heard of Party Rock. Have you heard of Party Rock? So that is a very cool app actually that AWS offers. And if you can actually put it in a browser, it's called Party Rock. I think it's partyrock.aws. Yeah, it's a very cool application. Like you can just put, you can just build an AI application just in like few minutes. I just explored that and I could just build an application in 10 minutes. So.

Krish Palaniappan (29:07.786)
No.

Krish Palaniappan (29:18.198)
this one here.

Krish Palaniappan (29:29.653)
Wow, what is this?

Ramya Ganesh (29:31.15)
And it is also all open source. You can just create applications and you can just put in there. it's a list of, you like you can just, it basically works as a prompt. So you would give a prompt there. Say for example, you want to, you're in mood for South Indian food in an area like 10 minutes around and you want to get the best, restaurants, the top three restaurants based on ratings. And you can just put that as a prompt.

to this application and it will create an application for you. So each time you run that application, it will generate the output based on your prompt.

Krish Palaniappan (30:07.713)
So hang on, this is interesting. I had no idea. How does this sit alongside some of the AWS products we saw in that list? I'm guessing it sits separately. I when do you use Party Rock versus something else?

Ramya Ganesh (30:17.806)
Party Rock is just separated, just a web application that you can just create based on certain prompts that you put into the model.

Krish Palaniappan (30:27.287)
So after I create this, when I generate an app using some of this, I can then download the code that it generates and then make it a repo of its own.

Ramya Ganesh (30:38.31)
Yeah, I don't think you can generate, you can download the code. don't think you can do that. Yeah, but I just use that as an, I felt very interesting because it's just like, you you have like a web application, like what 10, 15 minutes, you just feed in the prompts and then it will also create an image for you. this

Krish Palaniappan (30:42.711)
Okay, we can play with this. There's already another thing that you've shown me. like, it seems, it looks like eye candy here. It looks pretty cool actually.

Krish Palaniappan (31:00.247)
Okay, you know what, that's why, you know, I'm going to have to convince you do a one day session, probably convince you, just all that I can buy is just lunch for Ramiya and then we can go through a number of these things because it's going to take that time because you know, I want to understand, I just don't want to jump to doing something till I got a high level understanding and overview of this, which is awesome. So back to this, we have like another 14 minutes before you have to pop out and then maybe you can join back if you want, if you can from the car as well. It's okay. The first thing I need is someone needs to do when they go to Bedrock, Ramiya is picking a model, the first...

action item or is there something else we can or should do before we worry about the model selection.

Ramya Ganesh (31:35.672)
first thing we have to do is finding the model. So you're in the model catalog and you have got. Yes, this is the first place where you.

Krish Palaniappan (31:40.311)
So this is the first, so that was my question. So is this the very first thing that you typically do? Okay, you have a problem in mind and then you have a number of these models available. And now there is this challenge that almost everybody has, which is okay, how in the world am I gonna find a model that makes sense? But two parts to the problem, right? One is a model that solves my problem and then the best model that solves my problem. And the third part of that puzzle is,

the most affordable model that solves the problem in the best way possible. So I broke that into three parts. The first part is probably the easiest, which is the model that actually can solve my problem. So in our case, you said, okay, I want to generate some diagram out of a repository. How do I, that's my problem domain. Somebody told me to do it. I have to get started. How do I start here? Before I pick and choose or filter these models, how can I search within AWS Bedrock and the model?

Ramya Ganesh (32:10.433)
Yes, yes.

Ramya Ganesh (32:29.678)
Mm-hmm.

Krish Palaniappan (32:35.051)
to say you know what give me the model that makes the most sense for my problem.

Ramya Ganesh (32:38.446)
So for the research, had to do extra like, you know, Google, Chachi Piti, different kind of, you know,

Krish Palaniappan (32:44.917)
hang on. So Bedrock does not have a place for me to type into a prompt that gives me the model that I'm looking for from this catalog.

Ramya Ganesh (32:54.156)
Yeah, so you can use the chat. If you go to the Playground chat, there is a chat, in Playground. So if you see on the left, yeah. And there you can select a model. And you can also, say for example, at Tropic. Let's pick at Tropic. So there you can, yeah. Do you have any models? Or you can pick one. So what I would do is I just put a use case here. So this is my use case. So what would suggest a best?

Krish Palaniappan (33:00.993)
Here, this one here, okay.

Krish Palaniappan (33:11.667)
Okay, yep.

Ramya Ganesh (33:21.678)
model for this use case in Anthropic. So it's going to suggest that. If this is the use case, I have code, and then I'm going to pull it from different repos, and I'm going to convert it into an architectural diagram.

Krish Palaniappan (33:34.231)
You know what I mean? Tell me what should it be? This actually goes one step further, but it's all right. See here, I've made a selection. I've already picked a model, so it didn't suggest the best model to me. But from what you're telling me, sounds like that is not something you can do within the console of Bedrock. You have to do that research outside of this is what I take.

Ramya Ganesh (33:53.062)
It is a little bit research outside using Google, chat, you know, and I would, I'd also searched in hugging phase as well because there are different models in hugging phase and it also you hugging phase also offers a very good integration with Bedrock and SageMaker. I would search there as well. So I would just see, okay, so in hugging phase, which model is best?

Krish Palaniappan (34:08.619)
Okay.

Krish Palaniappan (34:13.601)
Because you have to not as well, I'm saying you have to because AWS does not support a search here to help me find the model I'm looking for.

Ramya Ganesh (34:19.402)
Yes, but that is a good one that AWS offers. It's something called a model comparison that you can do. Can you click on cancel and I can show you what.

Krish Palaniappan (34:31.775)
Okay, actually, I'm give me one second. Let me, I'm curious here. What should, let's type something to have it generate, right? Whatever, what a decent system prompt for what we're trying to achieve here?

Ramya Ganesh (34:35.395)
Mm-hmm.

Ramya Ganesh (34:42.744)
So write a prompt and choose run to generate a response. Or you can say like, okay, me a sample API or give me example API.

Krish Palaniappan (34:50.571)
But not the prompt, right? First I have to enter something into the system prompt, I reckon.

Ramya Ganesh (34:54.38)
You can write in the prompt itself.

Krish Palaniappan (34:56.663)
Okay, without providing any context, I thought the system prompt would provide context to our high level problems, so to speak. No, we don't need a system prompt.

Ramya Ganesh (35:04.686)
They don't need it actually, so I'll just use this one. you can say, okay, give me your example API of a model that converts.

Krish Palaniappan (35:07.071)
Okay.

Krish Palaniappan (35:13.94)
Okay. Should I say, what are we looking for?

Ramya Ganesh (35:19.34)
So, provide an example API of

Krish Palaniappan (35:27.381)
Or should I just say, let me say the dumbest way I would say it, right? Want to generate an architecture diagram from a code repository. mean, does that, yeah.

Ramya Ganesh (35:45.238)
Yeah, that's fine. So, or yeah, suggest a model for this use case, something like that.

Krish Palaniappan (35:51.063)
Can you suggest an appropriate?

model of other ear skins.

Ramya Ganesh (36:08.174)
Yeah, so let's see where it comes from.

Ramya Ganesh (36:13.55)
So static code analysis, C4 model tools, language, PyCharm.

Ramya Ganesh (36:22.414)
Okay, diagram migration, convert the into visual representation. Yeah, would you like more detailed information about any specific tool or, okay, so it doesn't, yeah, it doesn't say the model, right? So yeah, so the way, so yeah, it just requires some Google, know, hugging face, searching, hugging face, JackGPD to see.

Krish Palaniappan (36:23.959)
We make it.

Ramya Ganesh (36:46.786)
But this one does not, this is basically providing you an entire, you know, what if you use this model because you actually use, you selected the model, right?

Krish Palaniappan (36:54.913)
So Ramesh, sorry, tell me what we did here. It sounds like to me like if I put the same question in ChatGPT, it would have given me an answer. This is the answer generated by this particular model, correct? There is no other difference. It didn't do anything specifically. Then we go to ChatGPT.com and plug in this question and using the OpenAI LLMs, this was using the cloud, how you pronounce that, LLM essentially.

Ramya Ganesh (37:16.854)
Right, because you selected the specific model here, it's like giving you an answer that, for this use case, this is the tons of steps that you would do if you use this model, basically. So if you use this model to convert a code base to a diagram, so these are the different steps that you would take.

Krish Palaniappan (37:36.715)
Correct, so this is basically, in other words, if I took the exact same question for what it's worth, and if I go to say, let me go back here to chat playground.

I select a model. Let's say I select a DeepSeq model, right? I don't have access. Do I have access to... Maybe I have access to an Amazon model.

Ramya Ganesh (38:00.814)
think you have mis-roll AI access. No.

Krish Palaniappan (38:04.343)
I thought access, where is the list of models that I do have access to? These were two within Anthropic, but I thought, I thought there were more model providers. I'm not sure why. Okay, let's see. I don't know how instantaneous this is. Okay, now, I can, but this is also an exercise, right? Just for people to know. So let's say, I do have access to Titan text. I don't know why I didn't pull that one up.

Ramya Ganesh (38:18.112)
Yeah. But I can share my school.

Ramya Ganesh (38:24.803)
Right.

Krish Palaniappan (38:33.215)
Or maybe it's, you know what mean? I do have access here, right? This one says access granted.

Ramya Ganesh (38:38.444)
Yeah, I don't think you have access Krish because it's all grayed out, right?

Krish Palaniappan (38:42.133)
No, no, it's not. See this one here. This one sees access granted.

Ramya Ganesh (38:46.19)
Where is it?

Krish Palaniappan (38:47.603)
wait, hang on, actually, let me make sure I'm actually sharing.

Ramya Ganesh (38:50.882)
So what I see is request access, they're models without access.

Krish Palaniappan (38:55.171)
wait, hang on. Am I on my screen? I sharing? Okay. no, think I'll, maybe it's not refreshing. Tell me if I scroll, if I go to that.

Ramya Ganesh (39:05.518)
Because I think you have access to Anthropic Cloud. So it's a select Anthropic and then.

Krish Palaniappan (39:10.359)
No, my bad. thought I was sharing the entire window. Give me a second. It looks like it was only sharing. Oh, you know what? My apologies. All along, I've probably been switching between tabs. I think I picked the share. OK, so my bad. You can see it now, right? You see Titan text G1 Lite access granted. So I do have access to that one. So if I go back here, I was expecting to actually see that, but I don't. What is it called? Titan text G1 Lite.

Ramya Ganesh (39:26.146)
Yes, yes, yes, you do have.

Ramya Ganesh (39:39.47)
Okay.

Krish Palaniappan (39:40.009)
And this doesn't even show text.

Ramya Ganesh (39:42.754)
So just, could you put it in search bar and see if it comes up? It's the light, right? Okay, so here model without access. G shows the G1 Express. It doesn't show the one.

Krish Palaniappan (39:47.489)
to.

Krish Palaniappan (39:55.255)
so maybe in this playground, it doesn't have the other one, right? Maybe that's probably what it is. And let's see, even if somebody wanted, okay, which one did it actually have? Let's go there. It was like what? No, let's say if I pick Titan, it's what was that? Titan text G1 express, Titan text G1 is available to request.

Ramya Ganesh (39:58.926)
Yeah.

Ramya Ganesh (40:04.654)
the cloud one.

Krish Palaniappan (40:15.743)
First of all, know, Rami, this is silly question, what does it deal with requesting access? I mean, why do they expect you to request access before it's granted? Do know what the logic, I what the rational there is?

Ramya Ganesh (40:26.088)
Yeah, because most of them are actually free. It comes in the free tier. But it takes a second to get the...

Krish Palaniappan (40:36.989)
no, I this one is, I think some of them took a while. Okay, this one looks like it's instantaneous, so.

Ramya Ganesh (40:40.246)
Yeah, The moment you request access, it takes just like one minute to get the access.

Krish Palaniappan (40:46.167)
Okay, it still doesn't, maybe I should refresh the page. Maybe it didn't pick it up. So let's see if that solves the problem. Yep, now it picks up. So all I had to do was I request access and then refresh this page for it to actually recognize.

Ramya Ganesh (40:52.526)
Yeah

Ramya Ganesh (40:57.998)
But even the same model, I thought you had light. Did you have the express or the light?

Krish Palaniappan (41:03.517)
I had light but it didn't pick pull that one up so I requested access for the other one that this actually showed in this drop down.

Ramya Ganesh (41:10.179)
I think light could be something that is like, doesn't have all the features, maybe like the chat.

Krish Palaniappan (41:15.571)
Okay, so now we went to this other one. One of my initial challenges is each model surely is different in its metadata as well. How do we understand this? Like say the previous one didn't have randomness and diversity, it didn't have length, it didn't have guardrails. So what are some of these things? What do you call the configuration?

Ramya Ganesh (41:31.854)
Mm-hmm.

So one thing, yeah, that is like temperatures basically, you know, like for example, you have the output, right? So what is the probability that you will find a specific word in the prompt output? So that is temperature. So that's top P actually refers to the probability.

Krish Palaniappan (41:57.45)
Okay.

Ramya Ganesh (41:57.544)
of a specific, like anything, could be word or it could be something, a bunch of words. So what is the probability of that specific word coming up in the output? So if you want like less number of probability, you would decrease it. So you would decrease, that's something that you could, it's 0.9. So you can decrease it to like 0.1 or something.

Krish Palaniappan (42:19.967)
Okay, so which means long story short, each of these models could have a slightly different configuration and you have to sort of go into the model to understand what this means to make it pick the right set of values.

Ramya Ganesh (42:29.248)
Right, right. And if you see there's a temperature, temperature means like if you want the model to give you the same answer for like different questions, you don't want the model to be very creative, then you would go with a lesser temperature. Okay, so that's why it's zero by.

Krish Palaniappan (42:44.527)
But the notion of temperature is that generic to models in LLMs in general or are some of these specific to a certain model?

Ramya Ganesh (42:49.87)
It's LLMs in general. So all LLMs will have temperature and you'll have the top P, top K and everything. But temperature is actually related to hallucination. So the more the temperature, in this case it's zero, but if the temperature is more, it would result to the model hallucinating.

Krish Palaniappan (43:08.691)
Okay, so the actual configuration, some of these params are generated across models, except that when we pick the other one, Bedrock didn't give us an option to play tweak or play with these numbers, right? With this model, is actually able to give us, it's giving us more control over what those numbers can potentially be.

Ramya Ganesh (43:24.398)
Yeah, it's basically temperatures like say I'm giving you prompt input, but I want the model to be more creative with output. So in that case, you would increase the temperatures. These are just the properties that you want the model to customize. So I want to get the output.

Krish Palaniappan (43:39.263)
And okay, no, think, okay, that makes sense. And then when I plugged in the exact same question, the answer is quite different, right? This model's response is very, very different from that model's response, for instance.

Ramya Ganesh (43:44.462)
Mm-hmm.

Ramya Ganesh (43:50.902)
Yes, yes. So there is something called compare mode. If you see that there is a toggle switch on the left on the top left. So if you the compare mode, you see compare mode over there.

Krish Palaniappan (44:01.228)
Yep.

Ramya Ganesh (44:04.514)
compare, yeah, so here you can compare different models. you first say for example, based on your research, you think that okay, Titan could also serve your use case and Claude can also serve your use case. Now if you want to compare these two models and see how, you can just use a prompt and you can just prompt these two models and see how it's responding to the same prompt. So you would see like that would be a different answer, the length of the answer would be different.

and all that. So based on the response, you can figure out, which model would fit your use case better.

Krish Palaniappan (44:37.747)
OK, I'm still a little bit lost here. Probably I know we have to wrap up in a minute, but I have to probably when we circle back, can ask. Because let's say we ask these questions. But this is me ask, maybe the question here was not the right way, the best way I could have asked this. Because I said, to generate an architecture diagram from a code repository? Can you suggest an appropriate model? Probably that doesn't make sense because I'm in the context of a model already. So perhaps that question doesn't make sense here. The question should perhaps be,

whatever the next logical thing to do there is in the interest of generating. And I'm trying to understand what we want to accomplish by asking this, like forgetting what the prompt should have been. What would have been a good end result as an answer to that question here?

Ramya Ganesh (45:23.828)
What would be the end result for the different?

Krish Palaniappan (45:25.941)
Yeah, like what in other words, when I type this and I generated this response, right? I'm just trying to think aloud here as to what response are we even expecting for it? What would be a good response that tells us what to do as step number two?

Ramya Ganesh (45:40.142)
So if you see that there'll be different response based on the different models. So I think you already asked the question, right? The model can be useful. I mean, if you see the Titan text, right? It doesn't give you most of the answers there. So it just gives you a bunch of the models are designed to generate architectural. It doesn't say what you need to do for your specific use case. Whereas the Cloud 3.7 Sonnet, it basically tells you like, these are the tools that you could use. These are solutions you could use,

Krish Palaniappan (46:06.025)
Okay. And then basically if you like one or the other, let's say we would pick cloud. How do you, mean, I, I mean, maybe it's cloud or cloud. I mispronounce every possible English word. So I'm just going to say cloud. that, is that cloud? I don't know. Maybe it's cloud. Okay. So let's say we pick that. So where we need to pick this up Ramya is once we basically said, okay, you know what, for whatever we are doing, we for some reason want to go with, but

Even within that, there are so many different ones. But let's say we pick one or the other, and then we can go through with the next step. Let's say I pick this one. And now, as we continue this conversation, we can say, OK, what do I need to do next to generate this? Let's say we go through the exact same use case with what you mentioned, creating a diagram out of this repo. We could just take one of the Snowpals repos and try to do this exercise and just see what steps I Because so far, just to do a quick recap.

What we've done here is we went and I asked you questions about which other product we could potentially use. And you mentioned a couple of other products, Lex and Textract. Maybe they have their own embedded models, but if you wanted more, find granular control over your models and the choice of LLM, then you go with something like Bedrock. You have a number of models. I mentioned three ways you could, know, does it solve my problem? Does it do it? Is that the best model that can solve my problem? And is that the model that I can afford? Maybe there's a better one, but maybe I cannot afford that.

So there's a pricing that comes into play as well. And we made that much progress given the 50,000 questions I've thrown at you. Maybe when we pick this back up, could be, we've picked a model, we have a use case, we know of a problem, we kind of have an idea as to what Bedrock is, and then I can continue to showcase my ignorance, and then we can pick and choose whatever the logical next step is. I have lot of sort of unanswered, high-level questions, but as a developer, this is what I recommend to people.

Ramya Ganesh (47:53.763)
Yes.

Krish Palaniappan (47:58.805)
Don't try to understand too much of something. It's just my two cents. If you have some hunch, you have some level of understanding, chug along because sometimes you can become an expert in theory, but it's not going to be particularly useful. in my opinion, it's okay to be sloppy as long as you get something done at the end of the three hour window that you've blocked and allocated to yourself. And real quick, anything you want to summarize before you have to pop out.

Ramya Ganesh (48:25.324)
Nothing much. we have selected, we have gone through all these, how the model can be selected for the specific use case and all that. So the next thing would be like getting the API from that model and integrate it with your application and deploy a model and inference. So that would be the next stage in Bedrock.

Krish Palaniappan (48:43.543)
Perfect, right? you have to, your daughter's probably waiting for you, but let's say if you're in the car, you feel like you want to continue, you're bored, you don't want to listen to music, just ping me and we'll just pick up from where we left off. And folks, again, before we end this, thanks a lot to Ramya Ganesh. I was a cybersecurity XDR leader. This is conversation number two. The first one we were talking technically, the other one was very exciting, just not technical. This is the first of our technical conversations and I know we had to sort of schedule this.

Ramya Ganesh (48:50.35)
you

Krish Palaniappan (49:13.311)
I'm glad it happened. Well begun is half done. I feel like I can pretend to be less ignorant about bedrock after 45 minutes of picking Ramya's brain here. Thank you so much, Ramya. Until next time, folks, talk to you soon.

Ramya Ganesh (49:29.57)
Thank you, bye.