Video: The Golden Image Paved Road: How to Reduce Dev Toil & Speed Product Velocity | Duration: 2847s | Summary: The Golden Image Paved Road: How to Reduce Dev Toil & Speed Product Velocity | Chapters: Welcome and Introduction (0s), Introducing the Speakers (50.94676130724412s), Golden Image Challenges (161.75676130724412s), Golden Image Pillars (536.246761307244s), Building Trust Securely (894.506861307244s), Secure Image Automation (1163.666761307244s), FIPS and Ecosystem Compatibility (1716.186761307244s), Distroless Operating System (2039.5017613072441s), Vulnerability Scanning Explained (2167.471761307244s), Building Golden Images (2366.7567613072442s), Inventory and Adaptation (2471.221761307244s), Conclusion: STIG Management (2586.421761307244s)
Transcript for "The Golden Image Paved Road: How to Reduce Dev Toil & Speed Product Velocity": everyone, and welcome to our webinar, The Golden Image Paved Road, How to Reduce Dev Toil and Speed Product Velocity. We're so excited you decided to join us today as this has proved to be a hot topic amongst many organizations that we've spoken with. My name is Molly, and I work over here at Chainguard. And today, you'll be hearing from two of our very best solutions engineers, Natalie Summersall and Steven Seibold. During the webinar, if you have any questions, please ask them in the q and a section on the right hand side of your screen. We'll get to those at the end of the webinar. In the chat area, you'll find a link here that I'll be posting that'll lead you to our website where you can find all of our golden image program resources. And we'll also have a QR code on one of the last slides that will lead you there as well. Now I'm gonna pass the torch to Natalie who will dive into the webinar. Enjoy. Absolutely. So hi. I'm Natalie. I'm a principal engineer. I'm our technical lead for public sector here at Chainguard, and I'm a field facing engineer. So I get to help people implement these golden images programs. Having done the thing myself, so I led a built and led a software factory at one of the large prime systems integrators for about seven years. And with me, I have Spencer Seebel. Spencer, you wanna introduce yourself? Absolutely. Thanks, Natalie. Good morning, everyone. Nice to talk to you. Again, my name is Spencer Sebald. Just like Natalie, I'm also a field engineer here at Chainguard. But prior to joining Chainguard, I helped manage platform engineering team, large video game publisher, for several years. And before that, also worked at Puppet where I did a lot of DevOps work, configuration management work in the space. So excited to be here today and, talk to you guys more about Chainguard. Yeah. So what we've got in store for you. So we're gonna talk a little bit about golden images programs and why they still matter. And then we're gonna kinda, like, take it the dirty laundry a little bit and go why they're such a pain in the butt still. Both of us have done the thing before and are guiding people through this process as well. So from just thinking about golden images to having, like, really mature and running programs. So we'll talk about some of the patterns that we can see work and and why they work so well. And then talk a little bit about how Chainguard can help. And finally, we'll end up on q and a. So please send us all your spicy questions. Like, that is what we're here for. Especially the spicy questions. Right? Yes. Especially the spicy questions. Alright. Cool. That's why Absolutely. So let's talk about golden image programs and why why they still matter. So in broad sweeping terms, what is a golden image program? So really, it's a preapproved kind of standardized software development foundation, for developers to build their software on top of. Right? It's kind of an organization's attempt at blessing kind of, infrastructure that developers can use. Really, the idea behind this is kind of a standardized preapproved approach that tries to ensure consistency, across the board. If everyone's leveraging the same base, you know, the idea is that it should make it easier for platform teams, to kinda build things up, apply security policies, and all that sort of stuff. And everything is just fine. Right? Super easy to do. But in actuality, that's not what we see at all. You know, really, as we kind of, you know, as as open source software becomes more prevalent in the world, we're seeing more and more, vulnerabilities kinda spring up. Right? Standardized foundations were great, for a lot of things. But, you know, the second that we bless a piece of infrastructure, automatically, it starts accumulating tech debt. Right? And as we can see from the graph here, the number of malicious and vulnerable packages that we're seeing out in the ecosystem is just, you know, it's it's exponential at this point. So by adopting, a golden images approach, we're trying to solve for a lot of these vulnerabilities, you know, rather than, you know, having a whole bunch of disparate bases that we're having to maintain. So kinda talking about why they still matter. So it's not just about, you know, it's not just about, you know, streamlining compliance and everything, but it's also about developer velocity. If everyone kinda sticks to the same, you know, kind of blessed images and everything like that, ideally, our, our developers can go faster, the business can go faster, and we're we're working from a more secure, foundation. But that's not really what we always see. Is that right, Natalie? Yeah. So this is how I've seen a lot of these golden images programs kinda go off the rails a little bit. And that is you pave a path, that's not really where people wanted to walk. So when we talk to these programs, we we typically hear, well, it's static. It's rarely updated. I see a lot of I I'll call them kitchen sink golden images where they have a lot of things in them. So this one size fits all doesn't really help these teams so much as it does kinda hurt them. And then manual governance. So I work with public sector, so I'll hear well, I have one image, but I have to manually go through and run a STIG scanner on it. I have to manually go through and do this, that, or the other. So from your dev teams, like, even if they're not saying it out loud, they're going, this is stale. This is gonna slow us down. I can't meet my mission, and I can't build with this thing. I need to be able to do to go where I need to go in a much shorter path. So rather than doing this lovely 90 degree corner, you know, maybe we should have paved the path that people are already trying to walk toward to begin with. Because that's why we hear, they're such a pain in the butt. And here, it's kinda like unpacking that a little more. What we want and what we get aren't always the same thing. So centralization should free people up to build. You're not having to have each different team decide, oh, well, I need this version of this. I need this version of that. And a few base images should meet every use case, and that shouldn't require frequent updates, and devs should have no reason to look elsewhere. Like, that is not that's the failure cascade that is circling the drain to go down the toilet. So how we what we end up with is what we want is we get these teams that aren't equipped to service these cases, and the perfect image doesn't exist. You're going to end up with more smaller images, and that's actually a lot more easy to secure. They don't accrue vulnerabilities as quickly, and they're not outdated as quickly. So devs aren't trying to sneak things in, I'm gonna say on thumb drives, but, I have definitely been a part of burning things to CD. Yeah. And that's not this is what's driving shadow IT. This is what's driving risk that your business can't quantify. Because this cycle, this doom loop has real business impacts. So each CVE added per day to the the CVE database, 108 a day. So there is a never ending cycle of maintenance. And then per developer, that's, you know, fully loaded cost of a developer. We're thinking 28 k or so per year on security related tasks. Now this is not, like, doing really cool security work. This is not doing fun things. This is hatching. This is, this is chores. This is toil. This is the things that your developers go, maybe I can get a better job elsewhere. Because six hours a week is a lot. That's most of a business day. That, actually, that is basically my whole business day minus some meetings that I might have to attend. And that is, you know, sourcing, selecting, and updating the things that aren't critical to the mission that we're trying to build for, but are instead the things that I need in order to meet that mission. So here's what we've seen work and why. Alright. So thanks, Natalie. I appreciate that. And, like, I think the shadow IT stuff really resonates with me quite a bit. I mean, there's there's nothing that, that kept me up late at night or gave me those late night pages more than, than a shadow IT something or other that was deviating from this, you know, perfect golden image program that we had created or attempted to create. I I know. I, I have definitely gotten paged on, hey. Our, our, you know, system scanner, our infrastructure scanner detected this entire new environment. Right. It's like, the what? What? And it turns out that it's just, like, some server under some dude's desk. Yep. Right. It's Exactly. 2025, and we still got servers under desks, everybody. What's going on? Absolutely. So let's talk about what makes a good golden image program. Really, it's a combination of three major pillars. It's people, processes, and technology. So just to kinda dig into all of these different things a little bit more, you know, all these, all these kinda different pillars inside of an organization, they all kind of want different things out of a golden image program. Right? Everybody's a little bit different, and so everybody has different goals that they wanna get out of it. So, really, you know, when it comes to people, a lot of it is kinda understanding exactly what the teams are trying to get out of a golden image program and what success looks like to them. So for a lot of times for, you know, for folks on the platform side of things, they just want things efficient you know, to run efficiently. You know, I myself, I'm kind of fanatical about automation and, you know, making things run as efficiently as possible. It drives my wife crazy. But, you know, it it's just something that that I really personally like to do. And then we've got folks on the security side. Right? Security is responsible for setting the policies for the organization. They're kind of on the hook for, you know, anything, you know, any breaches or anything like that. Then you got developers. Right? And developers are just kinda out there. They wanna they wanna build cool things. Right? Like, they've they've got all this, knowledge and programming experience and things, and all they wanna do is go fast. So, with, knowing that every organization's a little bit different, you know, kind of getting that leadership buy in is key. You know, a lot of times, your KPIs are huge. One of the things that we really tried to emphasize, when we were managing our golden image program is if you can't measure it and you can't prove it, then nobody really knows what you're doing. Right? So being able to prove things through KPIs, check ins, sharing, you know, kind of cross collaboration. You know, the things that we do through a golden image program or or things that we can provide mean a lot, and they can really provide a lot of benefits, to a company that's, that's adhering to them. So things to watch out for. You know, one of the things that, I think we're all familiar with this is is really just kind of golden images can kind of lend themselves to a siloed approach too. I see a lot of teams that kind of want to take control over the entire pipeline, and they wanna do that in a silo. Right? And, really, that is kind of counter to what we just talked about with, you know, it's gotta be the people. It's gotta be the processes. It's gotta be all of the things. It's gotta be the DevOps things. Right? We gotta do the DevOps in order to make this work. So making sure that we have that visibility, we're aligned on all the, timelines and outcomes, and we have all the leadership buy in that we need in order to make it successful. Alright. So, shifting things from, you know, people, we're gonna talk a little bit about processes too. So, see, a lot of the times with, a lot of these different organizations, you know, they're we're wanting to curate images, you know, in a certain way, but we need to be able to update them in a, update them at a cadence that doesn't break dev workflows. You know, we need to be able to, not only not break dev workflows, but also keep up to date with all the CVEs and vulnerabilities that we're seeing, in the environment. So we have to have some sort of process around updating and then rolling those updates out. You know, one of the things, you know, at Chainguard that we talk about a lot of a lot of times, and Natalie, you probably see this a lot too, is, you know, Chainguard builds from source every day. And a lot of the customers that we talk to, say, well, you know, we can't take updates daily. Like, that's that's crazy. You know, so being able to kind of understand, you know, what's the intersection between when the business needs the update and, you know, and and what that's gonna do for us without kind of getting in each other's way is super important. Yeah. I've got a really great story to tell there. So, the day after, so Sunday night, about two or three months ago now, Wiz came out with a critical remote code execution CVE on NGINX ingress controller. Why they popped that public at Sunday night, Not my decision. Have having carried a pager for many years, I would not have appreciated that, but understand the the need to get information out. Monday morning, I was with a customer going through an ATO review. And for my private sector folks, this is when all of the security folks associated with running a a system in in the federal government are all up in your business. I I can't talk about how deep they wanna go on each and every CVE because they wanna know that. But they also wanna know all of your process controls and all of that. And being able to roll into that meeting to support my my customer, on Monday with them saying, oh, yeah. That popped last night. It's already been out. It's in production as of, like, four hours ago. Surprise. Surprise. Like, we did our testing. We did our our all of our, you know, backwards compatibility stuff. We've confirmed that that fix is there and not you know, we're not vulnerable. Was phenomenal. Like, that's the sort of thing that, like, as someone who carried the pager for mission critical systems for for years, you know, being able to go like, that big sigh was amazing. So that's awesome. Yep. Cool. So kind of building on that a little bit, you know, one of the, I think one of the biggest things with with getting a golden image program kinda off the ground at any organization, a lot of it goes back to building trust. Right? And we can definitely build trust through all of our processes, but just to kind of, you know, just to kind of reemphasize, you know, how we get that trust through, through technology. One thing is, you know, being very, very deliberate in terms of what images that we're curating, what environments that we're curating, and specifically how we're building these things, to support the business. Also, making sure that we're not breaking any existing workflow. So a lot of times, this can be pretty challenging. Right? Not all of us are always blessed to have a a, you know, brand new greenfield application that we can deploy, you know, in Kubernetes in the cloud that's gonna be totally, you know, totally cloud native and all that sort of stuff. A lot of times, we're trying to bring over, or kind of fit a, you know, a square peg kinda legacy avenue into the round hole of the, of the new golden image program. And sometimes there's a mismatch there. Right? And so it's really important to be mindful of, of the workflows that are or how they're currently working because we don't wanna break that trust, with our end users by by forcing them into an unnatural motion. That's that's gonna be the the number one way of getting your devs to say, I've got the company credit card. I'm gonna go on Amazon and have some fun instead of using this golden image program instead. And, and just to kinda emphasize, you know, things to watch out for, in terms of a golden image program, undefined image life cycles. We kinda touched on this a little bit earlier, but, like, what is your image updates update cadence? Right? How fast do you need to go, one, to meet the needs of the business, but also not break any existing environments? You know, making sure that your end users have the ability to request custom things from you. This was actually something that, we didn't put in our process for a really long time. And when we finally did it, it was kind of like a, oh, yeah. Duh. Right? Like, we're serving customers here. Our customers just happen to be internal to the to the business, but they're still customers. And so giving them a mechanism to request things, feeling like they're part of the process itself, and that they have the ability to customize or or change what they need, is super important. And then, of course, this one, I think, bites, all infrastructure and platform folks at at, at a point in their career. Big updates and hard cut overs, are always just a recipe for disaster. So the longer that you wait in between update cycles, the more stuff that kinda builds up, it makes your update even more critical. And, you know, it's just, I'm already I'm already having a PTSD PTSD back thinking about my pager again, Natalie. Thank you for that. You're welcome. I every time I say pager, I'm, like, kind of, like, touching the phone going, is it is it is your duty pinging me or or, you know, can I relax again? Totally. Right? So I have it on do not disturb, and it's been trying to get through. Yes. Yeah. Exactly. I have absolutely had dinners, weekends, kids events, like, all the fun things ruined because I was autopager duty. But let's talk a little bit about the technology that drives these workflows because there's kinda three key pillars. And the first is a trusted image and a centralized repo. So a central I I like to call it a single source of truth because if you don't have a single source of truth, you have multiple sources of lies. Like, that's the, like, the first part of of any security reporting. Like, what are we really running type program is, you know, having those those single source of truth in a central location means that you know what's running, where it's running, how long it's been there, when it needs to get updated, all of that good things. And having minimal purpose built containers means that you're only having a web server, only having a database, or only having, you know, a middleware client in that container. So there's not 800 other things causing CVEs, but also just complicating your system. I'm a huge fan of, of automation as well. Again, also drives the family insane. But my time as an engineer is important, but it's also kind of expensive. All software developers in some way are kind of expensive. And being able to repeatedly do the exact same thing over and over and over again is why we automate to begin with. So verifying all of the that ingestion process. So what I I typically see are we're gonna pull it. We're gonna scan it. We're going to verify that it is what we think it is, and then we're going to bring that into that central source of truth. And all of that builds on your CICD process, all of that builds on your security tooling, and that also bleeds into testing. So testing those use cases and testing the compliance frameworks against what your organization expects and then confirming the reproducibility of that image. Like, is this image the exact same thing every single time? And what to watch out for here, I'm actually gonna add a fifth thing here, because I just found this last week and and spent a a lot of time having fun with it. The first one is images that are simply debloated without true transparency. I worked with a team a a few months ago that, their security engineers thought they were actually quite clever. They were removing the RPM database. It, it saved space on the image quite a bit. And it also had a reported CVE reduction, but all of that software that is actually vulnerable was still in that image. So, you can deblote and you can break your scanners. And a lot of times your scanners will just say, oh, I I don't see anything. I as someone who owned a a Volkswagen turbo diesel, I think if if you know anything about cars, they falsified the emissions reporting onboard. That is fundamentally what a deep loader is doing to your security scanner. The other thing to talk about is, like, no SLAs or guarantees of updates. So, you know, at Chainguard, we like to say we have an SLA. We actually beat it by a very substantial margin all the time, but we do guarantee updates. So, you know, it's one of those things to think out all of the projects that you're trying to to build and all of the software that you're trying to build off of is is there are there people around this? Is there a vendor backing this? Like, do you have support? Lack of compatibility across your existing tooling and scanning. I've seen a lot of teams try to build out a golden image program and go, oh, hey. We're only going to use Postgres as our database without maybe talking to the teams that are way, way elbows deep in MySQL. That's a big database migration in and of itself that's not necessarily helpful to the goal of securing your your images. That's a whole another project and should be treated as such. Limited regression testing. So I am a huge fan of testing. I hate writing tests personally as a developer. And every time it saves my butt, I'm like, wow. Good job past me. Like, pat myself on the back. I have never regretted spending time building tests. I feel like every time that I do that, it, it helps the systems that I'm responsible for running. And we have very, very detailed regression testing here at at Chainguard. And the fifth thing I'm going to, kinda add to this list is, for my my public sector customers, but also for all of the folks that are going, oh, hey. We should think about FedRAMP, or I've heard CMMC is a thing, and I really need to think this through, is there is a huge difference between FIPS compliant and actually a FIPS validated module. So I say this because I, I saw a team that, saw the documentation. Okay. Well, this is an encryption algorithm. Like, obviously, I'm I'm a I'm a smart person. I know how to do this. And they wrote their own encryption algorithm. And every time I hear someone say this, I'm like, okay. Well, tell me more about that. And they ended up using a prime number of forty ninety six, thinking that's, oh, hey. This is forty ninety six bit cryptography. That is not, and that's also not a prime number. So, like, there's all manner of broken things there that they didn't really understand were broken, but they said, oh, well, I saw the doc. I wrote a thing to a spec. I'm good. Right? No. You're not. So let's talk a little bit about how we can help. So the old way the the ways that we've seen, you know, problems happen on on these teams, manual governments, they're stale, one size fits all, so the big kitchen sink image. The way that we approach golden images are they are purpose built minimal images. So for that database, the only thing that's in that container is exactly what is needed for that database to run. And that's the same across all of our 1,400 images in the catalog. It is customizable, though, so you can you can change this to meet the needs of your business. We build everything every single day. That doesn't mean that everything changes every day, but it does mean that we know that if there were any updates on both that project and any of its dependencies, that we're continuously able to address those. Our images preserve process immutability, meaning that, as an attacker, I cannot, you know, have a shell in that container and change that process at runtime. And they're reproducible. So we know that the same bits in are the same bits out every single time. And it's compatible with the ecosystem of artifact repository. So it's compatible with your JFrog, your Nexus, your Artifactory. There is no magic to what we do at Chainguard at all. There is none. I mean, I'd love to be the the feel good person and all the friends we've made along the way, but it's really hard work done well. All of our dependencies are explicitly declared in code, so those all go undergo code review. However, going back to that automation point, we are continuously opening proposed changes and building and testing and regression testing all of those changes, as those updates are released. So we call them bumps because they're version bumps, but, it's the same process no matter what. And we optimize for those minimal dependencies. So, again, everything that goes into any one of those images is all defined as code. All of that means that we can have cutting edge automation. So there's roughly 10,000 open source projects that maintain all of those 1,400 images, so we need to update the slide on on the number. I I love constantly bumping that number up. It all go into rigorous testing before they're actually pushed to a customer. So unit tests, acceptance tests, most projects have their own testing harnesses. So, you know, being able to say that this database is working if we add these tables. We can run these queries, have these results. All of these edge cases are all kinda covered in that testing, and that ensures that they're drop in compatible. The actual implementation is dreadfully boring. Actually, no. It's delightfully boring. Having having carried a pager. And we do this because we build everything from source every day. And it turns out when you do things the right way, you end up having a much lower CVE count and a much smaller image. So our Python image in this case does not have a shell, does not have a package manager. It genuinely has only the things that are needed for that thing to run. And you can, of course, customize this image. You can add Python modules. You can add other packages, all that good stuff. And, yes, that does add to the size, but these are designed to be developed on. They're not designed to sit statically for forever. And there are two callouts that I really want to to make on this. And the first is that, in this example, this is a FIPS validated module. So all of our images or almost all, not all, most images in that 1,400 catalog have a FIPS validated variant. And what that means is we actually do all the hard work of taking out the existing cryptographic methods, and we put in a missed validated source of cryptography and of entropy into that image. And what that means is you can actually run a FIPS validated workload on the edge without a FIPS validated kernel. Super duper cool. And if we wanna go, like, super deep into cryptography, please use the q and a. The next thing is by looking at this graph, like, yes, that CDE counts over the course of a day. However, working with public sector and working with disconnected missions, you know, someone had asked me, at this point, okay. That's great. But I have a system that'll be disconnected for an unknown amount of time. What does that vulnerability posture look like six months? So I ended up taking an image and and looking at a six month differential. The chain card image accumulated eight CVEs in that six months time span. Like CVEs happen. Like, no no one is ever gonna say that this thing that I built will never ever be vulnerable to anything unknowable in the future. However, had they used the the upstream image for that, they would have an additional 200 CVEs that if this is a fully disconnected system, you also wouldn't know about because your scanner is not getting CVE updates in that disconnected time span as well. So when you think about risk and risk over time, like, this really gets to be quite fun. Spencer? Oh, oh, I need I need to hear about that. There you go. You're on mute. No worries. Alright. Perfect. Yeah. So let's talk about, you know, kind of Chainguard's broad ecosystem, compatibility. I think, you know, for a lot of folks, as you're looking to bring any sort of tool, into your environment, it's gotta be able to plug into what you've already got. Right? So, currently, all of Chainguard's, container images that we curate are all GLIBC compatible. So they're dropping replacements for a lot of existing workloads, that you might have today. Kinda like Natalie mentioned, you know, our our demo, I guess, is very, is very uninteresting. Right? It's like we're gonna change a from line in the Dockerfile. Everybody buckle up. But it is very cool, and the magic, the magic is real there. But in addition to that, you know, Chainguard is gonna integrate with all, or most existing scanners out there today. So being able to not only, you know, prove what you've got out there, but also verify through these existing integrations that, yes, you know, what we've got out there, is what we say it is, and it's a secure, you know, as we as we think it is. And like Natalie mentioned, we've got over 1,400 images. That's also one of my my favorite stats as well. I think that grows by, like, about a 100 a month ish. I'm not entirely sure, but, it's growing like crazy. You know, and one of the things that, I think for me is, like, a, you know, as a platform engineer too is, like, thinking about what Chainguard offers, like, what if we could just turn off the spout to, like, different, you know, different sources of truth and only pull from, you know, a single source. Right? I think that's, I think that's a big thing here where we can, like, really lean into, lean into what we offer because our ecosystem compatibility is so broad. Alright. So, where can we learn more? Let's kinda talk about, so we've we've got, a golden images best practices white paper, that's available, here via the QR code, where folks can download, go a little bit deeper into the session that we had, today via the webinar. And, you know, obviously, we're we're also here to help, in terms of, you know, if you've got additional questions or, you know, looking for, some guidance in terms of next steps or whatnot. Lots of folks here at Chainguard are, we've got the scars to prove our, our platform engineer, experience. So, yeah, please leverage us, as much as you can or need to. We'll be happy to help. And with that, I think we're ready for some q and a. Awesome. So I'll read out the first one. What are the differences between an organization using Distrullist versus Chainguard curated golden images? What difference will it make in the long term? Spencer, you wanna take that one? Yeah. Sure. Happy to. So Chainguard is also, uses what we would call a distroless, operating system, if you will, inside of our container images. It's one of the things that sets us apart from, a lot of the competition or what a lot of folks might be more familiar with, which is maybe like a like an Alpine or, you know, a Red Hat Slim or something like that. So when we talk about what Chainguard uses on the back end, we use, something called Chainguard OS. Historically, it's been called Wolfie. It's essentially a kernel less or an undistro Linux, kind of container optimized operating system. What that allows Chainguard to do is it gives us the ability to manage and, and update all the different dependencies that go inside of our images when we need to. So that's kind of how we maintain that zero CVE baseline by rebuilding from source daily is we're able to control all the dependency chains that go into what's in our images and then update as we need. We're not beholden to, like, these long term, you know, kind of LTS releases that we're that we're used to seeing from a lot of traditional, operating system vendors. Instead, we control that whole supply chain. So in terms of what differences it'll make long term, I think going to distroless approach right away has some pretty major advantages. The first one being your images are shipping, from a place of minimal. Right? Like, they're not shipping with all that bloat, that we were talking about. A lot of times, like, you know, when I'm debugging a container, I'm guilty of it. I'll I'll admit to it. Like, the first thing I wanna do is jump into it and, like, get on a shell and and start hacking away and trying to figure out what's going on. But in actuality, for a lot of our prod workloads, like, they don't need a shell. Right? Like, they don't need a lot of these other creature comforts that we need, you know, for troubleshooting. And so by building from a place of minimal, by using that distroless, methodology, we're able to keep our images as small as possible. And then like Natalie showed, they accumulate CVEs way less over time. Anything you want to add to that, Natalie? Nope. That sounds perfect, actually. Nice. Next question is, how do you solve the problem with vulnerability scanners and enabling them to detect vulnerabilities within hardened images, especially with the lack of a package manager in most of our images? And then how do you ensure the SBOMs are accurate? So I'm actually gonna bump back a slide here. So we don't have a problem with vulnerability scanners. We create a vulnerability feed the same as any other distro. We also ingest all of the current NVD feeds. So not just NVD, but also, you know, Python has their own feed, Ruby has their own feed, etcetera. So we ingest all of those, and then we publish what is and isn't vulnerable. And all of these scanners here ingest that feed. So the scanners know how to read our images, and they know how to understand what is inside them, what is and isn't vulnerable, all that good stuff. Important to note, we are not and will never be a scanner. We think it is a conflict of interest to say, we have no CVEs. We have low CVEs, but only if you use our scanner. Like, it's a it's a check your own homework type thing. But, you know, especially when we're talking about, you know, business important missions or, you know, for me, you know, public sector missions. Like, it it's funny to make a joke about grading your own homework, but, you know, for me, people's lives can be on the line there. Like, that's that's not something we joke about. So enabling them to detect vulnerabilities is really straightforward. We we publish that feed same as every other feed that gets brought in by Ancor, by Wiz, by, you know, all of these other logos on here. We're also very open to partnering with new scanners. So if you're interested and there's not a scanner you see on this list, like, we're happy to reach out and and kinda foster that relationship. And then the other one is how do you ensure the f bombs are accurate? And that is a fantastic question. So because we build from source, like, recursively all the way down, we know very definitively what goes in that image. So the software bill of materials is basically the recipe of, hey. This is our Python image, and we put all of these things inside of it. So we generate that at build time. We sign that in a test with that in the build attestation and the artifact with sigstore. So you know that these three things go together and that they haven't been altered from the time that we've built them to the time that you're consuming them. And then on the accuracy on the, like, second part of that, we make sure before we one of those regression tests before we publish that the s bam that we generated is valid and that it matches what's actually inside that container. So we, we're we're kind of a two phase process there. Does that answer your question? I'm gonna assume quiet is, is it okay? Yes. Quiet. Oh, there's a chat. Yes. Perfect. Alright. And there's one more question in here. Yeah. I can use Yeah. Yeah. Absolutely. Do you wanna read it first? Yeah. Sure. Can you elaborate, with the use case of a financial org who started building the golden image program with keeping end in mind, how would we start the program? So this is a this is a pretty open ended one, but I wanna, I wanna take this from at least how I would think of it, and then, Natalie, I'd love to get your thoughts on it too. But, like, I think starting in a golden image program is actually the most fun part to be at. Right? Because you really get to define what your golden road looks like. You know, you're not, you're not taking on a whole lot of, you know, tech debt or anything like that. But it's also probably the the riskiest part to be at too. When, when we were first starting to build our golden image program, the first thing that we tried to do was try to distill down the requirements of the business into something that we could manage from a technology side. So what does that mean? So the first thing that we that we tried to do was understand what kind of infrastructure our devs needed to use. And then we tried to put that into buckets. So we, you know, we kind of we did the t shirt size approach where, you know, small, medium, and large sizes. For us, when we first started, obviously, our workloads were all VMs, just to kinda show you, like, how long ago this was. Right? We're not even talking containers at this point. We're talking VMs. But a lot of the same principles, you know, kind of apply here. But, really, it's kind of, I would recommend just distilling things into set buckets, and starting from there. Really don't try to boil the ocean because then you'll never get anything off the ground. I think, you know, perfect is always gonna be the enemy of of just getting something out the door. And so, you know, you never wanna be in that that place where you're just kind of constantly searching for that perfect image because you're never gonna find it. You know, the goalposts are always moving. Things are always changing. So I would recommend trying to distill things into a couple buckets and then just going forward with something and then just, you know, trying to, you know, kinda whittle things down and and get it more dialed in over time. Natalie, any any other thoughts there? Yeah. I I didn't get the the benefit of of getting greenfield projects. For the most part, all of my projects were existing. So that has upsides and downsides. The on the upsides of that, like, I didn't have to think through further architectural, like, future state decisions of, like, hey. Should we use NGINX or Apache? Should we use Postgres or MySQL or something else? I knew that I had all of those things. And I've it wasn't my spot from a a golden image architect type thing to to really say, like, yes. Everyone needs to be using this one. Not only is that cost expensive to switch applications, but it's not really related to my mission of improving the security of these projects. So that was kind of the the big one. So gathering that inventory, even if it's even if it's not complete, like, especially if it's not complete. Just kinda understand, like, hey. I, you know, I know that we have a couple of these types of databases. I know that we have a couple of these middlewares, in getting parts of that. Because even if you're a 10% across, like, all of the 100 projects in your your organization, that's an that's a beachhead on all of these projects to improve the the posture of your organization as a whole. Totally. And just I I love that you, I love that you touched on that. Because it yeah. That that's the like, inventory, it's it's so funny. Right? Like, it sounds like something that everybody knows, but as I think as as we've seen in the field, Natalie, like, very few people know, like, what their inventory actually is. Right? So having any sort of, like, handle on that is is huge. Yeah. It is the least fun and most important part of, I think, any cybersecurity program. Yep. Alright. There's another question in here. How do y'all manage STIGs in the context of containers and providing STIG and Scat Scans? I'll take that one, since I'm more of public sector, I guess. So we harden all of our images to the general purpose operating system SRG. NIST and and DISA specific guidance here is if there is not an existing STIG that is approved, you should use the most appropriate SRG. And that is what we do. So we ship an OSCEP profile for you to run through, STIG viewer, etcetera. And that's really, like, the most straightforward part. We use general purpose operating system because that gives us both the broadest coverage and the most specific help that we can do as a I would call it a golden image provider. A good example of why we don't do things like application STIGs is so much of that depends on how your application is going to to apply to that STIG. So, like, we cannot validate that you're using row level cryptography. That is how you are managing Postgres. We can't validate that you have not done anything terrible in a configuration file that you're giving to NGINX When we don't have that configuration file, we only have the application that we've given you. And, apparently, my Internet connection is unstable. So, hopefully, that all came across. Yeah. You're not breaking up or anything at all. We're good. Oh, oh, okay. I just got this, like, pop up that says your Internet connection is disabled from the, the webinar software. Oh, it's exciting. Awesome. I think that's all of the questions we've got. Any, anyone else have any any other burning questions they wanna ask? We still got a little bit of time left. Uh-huh. Alright. Well, I guess if there's no other questions, I'm gonna kind of splash this page up one more time just to kinda advertise our, our golden images white paper that we've got here. Lots of great tidbits of information inside of there. You know, lots of ways to, you know, kinda learn a little bit more. Also want to, just kinda throw out there for anyone who's, who's kinda newer to Chainguard or hasn't really gotten started yet. In addition to the 1,400 images that we have available in our image registry, I think Natalie, what is it? Like, maybe 30 or 40 of those are are actually free images for folks to try out. Yeah. It's about four 40 or so images, and they typically are our most popular images. So think Ruby, Python, several several varieties of of GCC, but also application images. I believe Postgres is on there and NGINX and some of the other most popular apps. Yep. So there's ton tons of good stuff to get, get started. Yeah. It's absolutely a great way to kinda get your feet wet with, what does it look like to to move our application over to Chainguard or, you know, what is what is Chainguard's distroless approach to to containers actually look like or or feel like in, you know, in comparison to maybe maybe an Alpine or a a different workload you might be running today. So highly encourage folks to check that out. And I think with that, I think we're probably at the end. K. Thank you all so much.