A Field CTO’s Insights on Secure Software Supply Chain

Name: A Field CTO’s Insights on Secure Software Supply Chain
Uploaded: 2025-05-22T17:16:09.373Z
Duration: 54 min 44 s
Description: A Field CTO’s Insights on Secure Software Supply Chain

Transcript for "A Field CTO’s Insights on Secure Software Supply Chain": Hey, everybody. Thanks for joining our webinar today. Excited to welcome George Kichukov, from GitLab. He's GitLab field CTO. And, we're gonna be talking about his insights on secure software supply chain from working with, GitLab customers. So just a couple, just a point of order here before we begin. We'll answer questions along the way. So, feel free to drop them in the q and a, and we'll be monitoring the q and a and answer your questions. A little bit of just some quick intros before we get started. I'm Ed Sawma. I lead product marketing at Chainguard where we're the safe source for open source. And, I've been in the space in in DevOps and in infrastructure for quite a bit of time. I spent a lot of time in identity management at at Okta, and I'm pretty excited about what we're doing here at Chainguard in in helping organizations secure their supply chain, software supply chain, and and open source. Yeah. George, why don't I go ahead and let you, give a little bit more of an intro to yourself as well? Hello, everyone. And, Ed, thank you so much for having me on the webinar. A little bit about myself, been in the industry for over twenty years. I started as a software developer at a start up. We were doing very unique niche name matching and identity matching software, and I did a few years of, consulting across software development best practices, code analysis, application best practices. After that, I spent twelve years at a large financial institution where I did a number of different roles, always focused on developer services, developer experience. As part of my role there, I also ran application security for a couple of years, so very familiar with that space near and dear to my heart. Really, that combination of DevOps, developer experience, developer services, and security, application security. In my most recent role, before GitLab, I also, spent about two or three years in the infrastructure space focusing on infrastructure automation, where we did a lot of work in, trying to bring DevOps practices, self-service automation, and developer experience improvements in the self-service provisioning of infrastructure, both for private cloud and public cloud. So I'm very familiar working at a large, enterprise, thousands of developers, you know, I'd say up to 30 or 40,000 developers. In fact, 200,000 employees and a very large group of people. So, I joined GitLab about a year ago. And, in GitLab, my role is, field CTO. A lot of people ask what does that mean. So field CTO is, externally facing senior role, which usually interacts with technical leadership at, clients and prospects. We have our internal CTO that runs our engineering organization. That's Sabrina Farmer. And we have a team of field CTO that specialize in different areas. My particular, areas of area of focus is on financial services. That's my background. Really, dev ops, dev sec ops, application security, software supply chain security in financial services in regulated industries. And, I had a chance to get to know a Chainguard even before joining GitLab. And now at GitLab where we are both a customer and a partner, really excited to get together with you guys. Yeah. It's great to have you. Certainly a wealth of experience, George, and and we'll get into some of the learnings that you've had in, you know, in your career across the picture here. If we start with the bigger picture, you know, what are we hearing from customers as you as you meet with them, GitLab customers, about what they think about software supply chain security today, what they think about the supply chain risk, and has that changed in the last, you know, year or two? I mean, we've all heard about the large scale attacks in the news in the software supply chain. So, certainly, people are starting to pay attention. I've been part of multiple conversation at a CISO level or or a level below where they have programs, sometimes different programs in place, mostly focused around application security and cloud security. And folks are starting to realize that, software supply chain security is is another type of program that can work alongside those. And, I'm starting to see some more mature organizations really establishing a dedicated software supply chain security program, and that usually includes a number of different areas and topics. One big aspect of it is obviously consumption of software components, if it's operating systems, container images, open source libraries. That's a big part of that. But then it's also the security of the entire SDLC process. Starting to like a secure SDLC and expanding that into secure software supply chain. So in terms of what I'm seeing is that companies are really start starting to pay more attention to it. Some already have established programs for this, and, really, those are maturing. But some don't have that yet, and I think it's time for everyone to realize that a dedicated effort around software supply chain security is required. It is not something that will happen organically. It's something that requires proper governance, proper program structure, proper resourcing, and it's really multifaceted. There are so many different aspects of software supply chain security. I think we covered some of them, but if we look at it from a definition perspective, it's as simple as who is allowed to contribute code into your repository. How do you even govern repositories to projects, to application, to services that you manage? Then you start to think about the build process within your application. Especially relevant is the, inclusion of external dependencies. If these are operating system binaries or container images or even third party middleware runtimes, and even more so, the open source libraries that you embed in applications. Some numbers that I hear is 80 to 90% of the code of applications these days is in fact through, third party libraries, third party dependencies. In some cases, we don't have a good understanding of what these are, where they come from. Are they secure? Are they safe? Are they properly maintained? So this, dependency management is a big aspect of that. But it's really that overall traceability from codes to build, to deploy, to scan results, test results, that entire, supply chain. And when we think about a factory or a manufacturing supply chain, that's a well well established term. It's an assembly line. It's repeatable. Multiple parts come out of it every day. From a software supply chain, there's still, a lot of work to do to really create a robust, scalable, repeatable process across the entire life cycle. Thanks for that intro and overview. I mean, I think it sounds to me from what you're saying that, the industry has has awareness around this. Some organizations have moved to get more proactive, have made more dedicated investments. A lot of folks trying to bring this further in you know, shifting further left in the SDLC. Do you think that we're reaching an inflection point where, you know, more organizations are accelerating their efforts to treat this as a first class security concern? So I would say that, there there are some people who are doing it, and more and more organizations are starting a dedicated effort around software supply chain security. I think we're, you know, slowly climbing up and reaching that inflection point. I don't feel like we're there yet. I think we'll feel I'll feel when we're there yet. Every when I see that every organization has a dedicated group or team or program around it, we're probably about 50% there, but it's definitely growing year after year. Sometimes I hear that, hey. We have an application security program. That's reached a level of maturity. Everybody has it. It it's rare to see organizations that don't have that. And, you know, oftentimes, this, software supply chain security is an expansion on the application security angle. And if application security includes things like dependency management and software composition analysis, supply chain security builds on top of that even more. So, we're getting there. We're growing. I don't think we're at the inflection point where everybody has it. I think we're at a point in the industry where everybody knows they need to think about it seriously. It's, it it's something that we hear in the news every day. It's something that, people know they need to do. It's it's become an issue of priority. Is it the highest priority item? Not yet. Hopefully, there isn't a compelling event that makes it a a a a high priority item because that would not be a good event typically. So what we wanna do is have a sort of organic growth, dedicated executive effort, dedicated education even within an organization's developer community as to why software supply chain is important and what are some basic steps that people can do to, make it easier for them. Yeah. It does it does seem like one of the pressures out there that, you know, we've we've found is that, a lot of organizations that are are b to b that sell to other organizations are are feeling pressure from their customers, around this to to to be more secure. That seems to be one point. But, as you kinda think about some of the aspects around that organizations make to, you know, address this, issue, you know, there are some things that seem addressable on the surface that seem, like, straightforward, like patching, for example. But then you start digging into it, and it it could become overwhelming, with the scope and the breadth of what you've gotta do to keep all of your open source patch and up to date. What do you see people struggling with as they go in from, like, awareness into implementation here and they're trying to actually take action? Action? Like, what are the hard problems that that they face? I think one of the problems that I see is that the software supply chain as it exists in most organizations today includes, you know, five plus or sometimes upwards of 10 different tools and processes. You may have a dedicated source code management system, a continuous integration CI system, a separate platform that does deployment. Then you're doing your planning work, in something like Jira, for example. And, then you have a number of different application security scanners. You may have a SaaS, static application security scanning software, compositional analysis, container scanning. When you end all of this up, you can easily see 10 plus different products. And there is some consolidation happening, but people get overwhelmed as to how do I secure the supply chain across all of these different tools that have different APIs, different, data models, different entitlements model. The entitlements model is is a big part of software supply chain security. So I think that's one of the biggest challenges. Where do I start when you have five different tools in the in in in the tool chain? Another aspect is, like you said, a lot of people know they have to patch. They view a number of process around patching at the operating system level or middleware level. Even scaling that is challenging when you have, 10,000 or a hundred thousand servers or container environments, and you have to update them oftentimes on a weekly or daily basis depending as new, items come up. This is a lot of work. Optimizing this process is not easy, and some people are doing it well. Some people are still struggling with the scale, with the automation required, with the validation required at each step, the dependencies. So, typically, if you have, let's say, a new container image that you have to, push, And people most organizations have a library of best base containers. So let's say there is a change in the operating system layer. Well, now all the layers above it, if it's your middleware layer, something like, Tomcat or WebSphere or your language runtimes like Java or Node. Js. All of these layers have to be tested on top of the core layer. And it takes time, and it's usually different teams that are doing it. And, it's just it's just a lot of hard work across the different tools and the different teams. So in terms of, where do people struggle with just to kind of recap. Hard to do across multiple teams, hard to do across multiple teams responsible for a piece of the supply chain, a lot of cross dependencies. And then on the consumption side of it, as an application team, you're oftentimes, asked to do things sort of, on, out of your regular release cycle. Hey. I've got a big release coming up at the end of the week, but there is a vulnerability that requires me to patch and fix now. So now I need to kind of stop working on that release, get the new, base image, properly integrated, tested, deployed, validated, etcetera. So it just it's just hard to do. It's not easy to do. And, there there really isn't a simple answer here, typically. Yeah. It's a lot of work required. I mean, you described that whole toolset that needs to be managed. And and then as you said, these these kind of last minute, patches or fixes that need to be made to ship a product. I mean, overall, this sort of begs the question of, like, how organizations balancing the the need to implement security with the need to still ship faster and and the pressure to innovate. You know, there's always been this tension between between security and engineering. How are you seeing customers balance that velocity with the need for deeper security guardrails? Yeah. I mean, we've been talking about shifting security left for years. It's almost become, one of those things that people talk about. But the reality in many enterprises that today, security vulnerability, security patching is still outside of your typical developer and release workflow or it comes in too late in that process. So one of the first things, obviously, to do is build it into the workflow, do it earlier, write in your pull request, in your code review process. You get the results, you know, from the various security scanners. Really optimize the process such that a security patch should not feel like an incident. It should feel like a regular, incremental release. The other aspect of it is is that collaboration between the security teams and the development teams. Many times, I've seen security teams act a little bit as a reporting layer. Hey. You have all these things that we found. Here is a report of maybe 10 items. But when it comes down to how many of those actually need to be prioritized, how many of those need to be fixed first, and really guidance as to how exactly to address a specific item. Let's say you you happen to inject a secret in your, application. You nobody should be doing that. There are tools that prevent you from doing that. But in some use cases, there really just isn't a better, easier way to do it. And you end up doing it, and the it get it's getting flagged. Security is going to block it. But instead of guiding you, well, here is how you do it differently, they sometimes don't have the necessary expertise at the tech stack level or the integration level to make it seamless and easy to do. So that collaboration, that effort, that focus on education, that off focus on early detection, integration into the developer workflows, those are the items that I think still cause friction and still are important to address. All all really great points. We do have a question that's come in from the audience. Where do you recommend someone start to address their supply chain? If you're kinda just getting started with this, where do you get the most bang for the buck and and where do you get started? Well, I think the first thing you want to think is what are a lot of the issues stem from various access tokens or elevated privileges that exist in the supply chain. So you really need to think about the different tools in the supply chain. How are they interacting with each other? Are they exposing, elevated tokens when they talk to each other? If your CICD pipeline reads from the source code repository and it's automated, And that token is is a long term, high, highly privileged token. Or if somebody if that token leaks, it's a big problem. Right? So just looking at the secret management and the integration between the different tools, simplifying the two the tools and the process, that's one area. The second one comes back to dependency management. Integrating your, dependencies, your software bill of materials, your third party open source, and other and vendor components that you embed in your application and having a good visibility of what these are right within your development environment. Oftentimes, you have to log in to a different tool that's only generated after you release to production. But, from a starting point of view, look to simplify the stack, look to see how different, tools interact with each other from a token perspective, privileges, and then create that, bill of materials right within the developer workflow so that they are aware of what's there, what's outdated, what's risky, and they may choose to address it as they're, going about working on their business related features. Good. That's great. Yeah. That's really tangible ways to get started. Let's let's zero in a little bit more when we talk about software supply chain. I wanna I wanna zero in specifically on the problem of open source. And we know, you know, open source code itself is pretty secure. It's actually, you know, arguably more secure than than closed source. But, you know, the problem when you bring open source together with the with the supply chain problems is is sort of where you get into into trouble. But when, you know, GitLab plays an important role in the open source ecosystem. I'm sure you you have a lot of viewpoints in terms of how organizations use open source and and whatnot. From your vantage point, where do you see teams struggling the most with open source security specifically? Yeah. It's a complicated topic. Well, first of all, with GitLab, love open source. The entire platform is open source, open core. We have, a lot more, code contributors to our platform than we have engineering, team members on staff. So it's really a a a large community. It takes investments to maintain a community like that. It doesn't happen on its own. We have dedicated teams that are working with the community to help with contributions to ensure those are safe, and there is a process around contribution that makes sense in the overall direction of the product and the company. When it comes to open source security, I I definitely feel like open source is secure. We want this to be out there as a mechanism to get more transparency. There is an element of who the maintain maintainers are and how do we ensure that, a, they get proper recognition, but also there is more transparency around the community behind a particular product or project. So, just to go back, open source open source definitely will continue to be used. Open source will continue to remain secure. So it's really around transparency. What exactly are you using? Really deep understanding of that con that, interdependencies. You may include one component that's going to bring in another 10 or 15 components. So that deep dependency tree needs to be fully transparently, displayed, shown, understood. The different versions around that needs to be there. And, really get into the habit of updating all the dependencies. It's a process. It's a step. It's work. But, the more you do it, the the better you get at this. So, just, in terms of open source community, what the advice from an open source community is, oftentimes, we get started with a really good valuable piece of functionality, but we have to think long term. How is this going to be maintained, secured, patched, and used by users? So just like every software project you start, open source or not, the initial creation is almost the easy part of it. It's the ongoing maintenance and life cycle that is that is important to keep an eye on. Yeah. Because, I mean, as you said, I think, you know, secure open source projects are often the ones that have a very healthy community of people that are contributing and and and and updating. And so and and and the project owner, like, in this case, I mean, GitLab does a lot to to maintain that active community. So it sounds like that's a real important thing to worry about to sort of think about. But then you bring up dependencies, which is a whole other area. So if you're if you're if you're running an open source software, it's dependent on a whole bunch of other projects. And I know one consistent pain that organizations deal with that, you know, Chainguard plays a role in is is CVE remediation. How do you see teams handling that today out in the field if they're trying to deploy open source? It's dependent on a whole bunch of other libraries and projects and OS layer packages and things like that. How are they handling CVE remediation, and are they keeping up with it? Yeah. It is hard to do. And some when there is some, projects do a much better job than others. It is not easy to do, and that is why companies like Chainguard exist in this space and really bring that mature practices of building from source, full traceability every time, full validation of every step in the process. It it really is not easy to do. And when it comes to CVs, I almost seem see, like, several different sort of, of things that happen. One is alright. Well, it's as simple as just moving to the new version. That's almost like the easiest path. In some cases, that move to a new version would change functionality, would result in additional testing, would result in additional work to, to incorporate changes. That needs to be planned and allocated and, and really considered. In other cases, it's a simple version upgrade that's usually done faster, ideally, with less impact. In other cases, there is a known vulnerability. There really isn't a solution or a fix yet. So how do you deal with that? Do you try to do the fixes yourself? I mean, it's in the open source. You're using a specific version. Community is not fixing it. There really are a complicated set of challenges here. That also appears in this sort of inner sourcing community is a variation of open source. In an organization where some there is reuse that's created, this inner sourcing community is only, this only effective if the community is active and actively addresses items. If it doesn't, it becomes a challenge, and maybe it's, it's a question of bringing it as part of your core source code or taking another component that's better maintained, that's more proactive. Yeah. That's that's really helpful to kind of understand those different options that organizations have. We got another question that's come in through the q and a here, from the audience. Do you consider package repositories such as NPN, PyPI, Maven Central? Are they living up to the current needs for modern software with security, kind of geopolitical considerations, transparency, abuse prevention, and and things like that? Yeah. This is a really good question, and it's not an easy answer because most of these communities are more or less not getting paid to be the communities for that. There is the, idea of low barrier to entry. You want a wider community and more contribution, and that's been successful over the years. But on the other end, these are important building blocks of your enterprise mission critical financial system software that it requires an additional level of, scrutiny, an additional level of of, I would say, due diligence. I wouldn't want I wouldn't say that I I mean, I guess some of these organizations are doing a better job than others, and, I think the bar needs to be raised. I I I would expect and hope to see more, and I'm not sure what's the right commercial model around it. But, certainly, better due due due diligence and more, preventative controls in place would be beneficial for the community. Yeah. Yeah. I've seen some statistics of, you know, these, repositories. You know, they have some protections in place, and I've seen some statistics that they, you know, sort of are able to prevent, like, somewhere around 50 to 70% of malicious software from from entering those repositories. But then then there's some leakage there, and and I know organizations will do things to curate libraries internally. They will use, you know, art you know, things like, artifact repositories internally to kind of handle different policies around which libraries can be used and not. Kind of on this topic of broader topic around, like, potential for malware to be injected into libraries like this, it's a tougher problem to solve, I think, than just patching and updating software, something that's maliciously injected into software. Do you see organizations do anything to mitigate these kinds of threats? Are they are they doing you know, are there any kind of common patterns you see out there? So in many times, we talked about this common open source repositories like Maven Central, PyPI, and NPM. I think the first practice is you really cannot provide open access to those repositories to developers. There has to be a gated process through which, some of these external dependencies come into an organization. It's a government process. It it it involves, malware detection, CVE detection. So as as you're running, a controlled build, you cannot just go to the external repository. You really need to have, if you call it, a local mirror where the things are so so called known good. And maintaining this known good is a real difficult problem. When I was doing this, in my previous role, it was almost easy to say, hey. Why can't we just point to a known good repository? Because it's known good now, but it may not be known good in an hour. So this constant maintenance of what's known good is really hard to do. And when it comes to malware, I think one of the things to think about is that there's almost you have to introduce a delay. If there is a new version that's published and, maybe it hasn't appeared that, you know, people haven't identified an issue with it yet, maybe there has to be a waiting period. But it's hard because sometimes you need to go to a new version to address another already known CV. So, it's a really hard problem, and, I think it starts with proper governance and steps as you're bringing in new versions, new components, and making sure that there is kind of a little bit of a, you know, process. It shouldn't be super heavyweight. Right? It's not like a manual review that may take thirty plus days, but at least take advantage of industry data, industry scanners. And as this comes in, kind of create this, sometimes people call it the dependency firewall. Right? As you're bringing in new dependencies, stop them if they're known bad. And if you're not certain, allow for some time before, you consider them known good. Yeah. That's a that's a great, a great approach, is to kind of monitor and and keep track of that. I I wanna make sure we leave time for q and a here, and, I've got few more questions for George. For those of you, you know, listening, if you wanna go ahead and and start entering in your questions that you want George to answer, we've got a few come in. But definitely go ahead and enter those into the q and a. We can answer them along the way, or or or if there's a lot that come in, we'll we'll, have q and a at the end. But kinda just, you know, starting to starting to kinda wrap up here and draw sort of some conclusions and and looking at both, you know, GitLab and Chainguard and what we're doing in the in the industry. You know, GitLab itself is a customer of Chainguard, has spent a lot of time thinking about these kind of supply chain risks. What are the what are some of the things you've heard from your own engineering and security teams when it comes to deploying Chainguard? Yeah. And then this is a very timely webinar. We weren't planning it that way, but just earlier this week, three days ago, we, at GitLab announced our FedRAMP authorization approval process. And this is this is a really big step, and, it's, Chainguard played a role here. I want to, give credit to our internal engineering teams and SRE teams that we spend a lot of time, building our infrastructure. So the way GitLab, deploys is we have three or maybe I would say even four different deployment models. Fully self managed on premise, self managed air gap, multi tenant SaaS on gitlab.com, and then what we call single tenant SaaS or dedicated. So GitLab dedicated for government is the one that we really focus on when doing the FedRAMP authorization, and that is where, we use Chainguard extensively. Again, great work by our internal engineering team. So but this is a situation where our teams at GitLab, the experts on the product, run GitLab on behalf of our customers at scale in a single tenant environment. Mhmm. And that environment has a number of controls in place. It also uses a number of, open source tools for monitoring. I'm not going to give a specific example, but there is an upwards of 20 open source tools, part of this dedicated stack, to be able to run and monitor GitLab at scale. And, that's where we use the chain guard images. Some of this, tools that we use, the vendors or the community does a good job. But even when they do, they're not as fast. They don't have as strong SLAs around providing patching. And when we're talking about really, high security and compliance, government organizations or financial service organizations, providing that degree of additional hardening and security validations and SLAs for CV that we get through Chainguard really helps us here. K. And is there anything, that you're looking forward to seeing from Chainguard? You know, as what you know in terms of what GitLab needs or on behalf of the businesses you speak with regularly, something that you think could really help the industry. Yeah. I would say, most of what we use today is, container images, but it's not everything we run is containerized. We still have, operating system images that we have to deal with or other third party packages that we have to deal with. So I would love it if and I think Chainguard is already doing some of that, expand into more generic types of artifacts and packages and building them securely from source with full traceability and full set of mature steps, both, operating system packages as well as other packages that you'll install on your service stack that may happen to be outside of the container environment. Yeah. That's I mean, that's definitely along the lines of of where we're heading, so it's good to see. From GitLab perspective, you know, GitLab is a key part of the the ecosystem here as well in terms of security software with over a hundred million developers using the platform. Can you talk about how GitLab helps with supply chain security as well and and users of the GitLab platform? So especially organizations who are mature on their journey, we provide all the capabilities within a single platform. So first of all, you have your, source code management and your CICD working together closely. Second of all, you have all the different security scanners. There's over eight embedded security scanners right in that process at scale in the developer workflow. And then we have the right integrations and capabilities around, cryptographic signing and validation, provenance and attestations. All of these steps build it in within the same platform with the right level of access control, the right level of secrets management, really really provides for a real for a strong foundation. And, a lot of customers use the capabilities today, and those that are looking to use those capabilities, it's a lot easier once you're within that unified platform versus working across 10 plus different tools that you have to integrate with each other. So I think that's one of the key differentiators. You basically get the supply chain security, benefits just by being on a unified DevSecOps platform. Yeah. That's definitely really powerful to have everything in one place. So organizations can build build software that that is secure, as well. Questions oh, we'll open it up to q and a here now for, for kind of the last ten, fifteen minutes, or however long we go. So another question from the audience. If you can disclose, what is one technical problem you are currently tackling in GitLab? Oh, one technical problem that we're tackling in GitLab. Okay. I think I will look at it from an overall product positioning perspective. Right? We are, we are, looking in a world where AI and agentic AI is is doing a lot more. So it seems from a technical perspective, we're trying to decide where would developers work in the future. Are they really going to is is IDE still going to continue to be the developer day to day working area? Or, as we see developers kind of evolving into maybe potentially more of a orchestration engineers across a number of self-service agents. Maybe you have an agent that's helping with a code review, an agent that's helping address a security vulnerability or a patch, or an agent that's doing a UI work, or an agent that's doing back end database optimization. Are you still going to continue to work in your ID, or are you going to work more in a remote workspace where, you know, instead of having one ID in your single threaded within that, you have maybe 10 different remote workspaces where the agents operate then come back to sort of this review process. So is the ID then in the next five years? I don't think it's that yet. Certainly, a lot of tools are focusing on doing working within the ID. But is, are more organizations starting to look at remote workspaces? GitLab does offer a remote workspace environment that that gets you up and running quickly. It's very well suited for this sort of agentic type of workloads. And we're seeing an uptick in that, and we're trying to decide in terms of an investment team how much investments continue at the ID level, which is definitely still the place where developers work, and how much of that moves to the remote workspace or the platform level. Alright. Well, thanks for thanks for addressing that one. Another question for you. Do you think open source AppSec tools are nowadays on par with commercial tools, especially for large organizations? So, I mean, I I would mean that for some of our scanners, we have a different scanners. We rely on open source tools. It's not all proprietary GitLab developed. And, and, yes, some of them do a pretty good job. And a lot of times, people get really sort of stuck on, you know, what is the best tool for the job, and that's just part of the problem. A lot a big part of this of the solution here is how can I ensure that whatever results I get back from my tools, open source or commercial, I get, I have the right process? I get them early enough. I have the right training to be able to address them, quickly. A lot of times, these scanners turn into a reporting mechanism versus a risk reduction remediation mechanism where you really want to fix them. So there are open source tools that are pretty good. In some cases, enterprise vendor supported, tools, also do a good job or a better job. So it's not like a single answer. As I mentioned in some of our monitoring facts, we do a combination. We have vendor tools that we use. We also use out on those tools, and sometimes we develop our own capabilities. So we try to make the best of our world. Yeah. I've seen that I've seen that pattern before too, especially with, like, larger and more sophisticated organizations are using a combination of multiple tools. Another question for you and and, curious how close you are to the the the deployment here where, you know, somebody somebody asking, like, what your process at GitLab was with container images before starting to use Chainguard, and how much was the full time employee investment in just managing container images. Are are you were you, you know, are you sort of close enough to how engineering deployed Chaingard internally to to Yeah. Yeah. I'm very familiar with how the change how Chaingard was deployed specifically related to our, dedicated for government environment. And, I would say that, we do apply some of the similar practices in the other environment, and it is time consuming. The SLAs are not as good. They have to rely on external open source communities or vendor packages. Some vendors do a decent job, some not as much. Right? Chainguard is a leader in the space. They bring this level of maturity in this process. And when we can use it, it it is beneficial and it saves time. We don't rely a % on that for everything. We do rely on certain vendors who are doing a a good job in the space. We do rely on certain open source community images as well. It's that combination bringing it all together in a consistent reliable way. I mean, that also has its set of challenges. Right? Sometimes if you have a vendor image that was built by Chainguard, potentially, maybe from a support perspective, it may result in in in in a challenge at times. I would say that, you know, it now you have sort of three three parties in the mix. You have the Chainguard, the the provider of the mature build practice and process. You have the vendor that's the original creator of the packages, and then you have the customer. And sometimes you end up in this triangle whose problem it really is and who needs to address it. And, as this process is mature and, organizations get better at that, there is, you know, plus and pluses and minuses in all these different approaches like every other technical decision. Yeah. That that makes a lot of sense. There's always trade offs. And I can imagine, I mean, at GitLab, given the expertise of the team there, the fact that your product is a, you know, build pipeline that GitLab could build some of this stuff. And from what I understand, you know, it was really about acceleration. I think I I think adopting Chainguard, especially in that government environment with our FIPS images, was was about getting there faster without having to use, you know, GitLab engineers that have a million other priorities and other things that they could be innovating around, I could imagine. Yeah. I'll definitely say that, we spend a lot of time on that software supply chain for our own products and images and services. And, just learning from Chainguard as well, that partnership has been beneficial, enabled to see how quickly Chainguard can do it and what are the components of ensuring the images are hard and insecure. And, we can take these learnings and apply it as well elsewhere, and it's an acceleration. Right? Instead of building that expertise from the ground up, we are evolving that and and building on top of it. Another interesting question, in terms of FedRAMP in particular, and actually have not heard this, acronym before. But for your FedRAMP package, did OSCAL, o s c a l, play a role in the process in any way? Oh, I I'm afraid I'm not able to answer that either. I'm not Yeah. Okay. I wanted to follow-up on that one with with that person. I I, I I I've dove into FedRAMP requirements a decent amount and and and, ConMon processes and POAMS and all of that kind of stuff, but, I wasn't familiar with OSCAL. I I would say that the FEDRA process is pretty, extensive. We spend a lot of time and have built a number of hardening processes for that. Now we're applying not just to our FedRAMP environment, but to other environments as well. So, it's definitely, I think, been beneficial, but it is lengthy. It is expensive, and it's just not easy to do, and Changan has been a help on the way. And really, you know, big shout out again to our internal engineering teams who, recently, were able to get that final approval on the FedRAMP, and, it was years of work, honestly. Another interesting question, little bit of a change of gears, but, talking about distroless in general, you know, Chainguard's approach is is distroless images, where it's actually distroless with our own distro, Chainguard OS or, if you're using our our our free tier and starter images off of, our Wolfie OS, which is our our our free open source version. Do you find have you found downsides or challenges around distroless images? Are you know, where are organizations in their adoption of distroless and and what, you know, what are the challenges there? Well, it involves a real good understanding of all the components you need for your, run time stack and your, you know, things like even observability or troubleshooting or debugging. It's a little bit harder to do. If you're a very mature team that has a real good understanding of what's required to both run but also debug and troubleshoot at scale, it's easier. It it's certainly from a CVE mapping perspective. It makes it a lot easier. Even from purely sizing and scalability perspective, the size of the images is is a big difference. Moving images over the network has been beneficial. I'm a big believer in distillate images, but there are challenges, and it really comes down to full understanding of all the components in your run time stack and all the components in your observability troubleshooting stack. Got it. Got it. Yeah. That makes a lot of sense. Zooming out here a little. So question around, in different industries and how they are maturing. So, you know, do you see them maturing at different rates? Do you think, for example, financial services firms might be further along when it comes to supply chain security or health care firms? Have you seen any trends there by industry? So, certainly, I would say there is there is a difference. The regulated industries are further along, but they're not all the way there yet either. I'm surprised sometimes that large financial institutions who still don't have a dedicated software supply chain effort. I'm still able to see seeing some organizations hiring, expanding in the space, and just really ramping up that program, which basically means that it's not all in place yet. And, there is a difference. Some organizations, that's not even on the radar. So I would say, at least in financial services, pretty much everybody I talk to either have a team in place or looking to stand up one and create a proper program around it. So the awareness is there. Is the program, like, ready to produce, results and show improvement? That's where some of the challenges we talked about earlier. If you have a unified platform, that's almost, much easier to do. You have to do it across 10 different tools. It's going to take time, alignment across 10 different teams, and how do you even secure different data models, APIs, and then entitlement mechanisms. Right. Great. Great. That makes sense. And then a couple of maybe final questions here around the developer, and developer experience. So I'm sure we've got some some folks, some developers and SREs on the call here today. You know, what role do you see developer experience playing in securing the supply chain? Is developer experience at odds with a more secure supply chain, or, you know, are there ways to make a developer's life actually easier with some better security practices? This is I I love this question because I'm super passionate about developer experience. Developers spend so much time building critical capabilities for the business, and, their experience is not always optimal. Many enterprise organizations, even very mature tech leaders, struggle with the right developer experience. I do not believe it's it's at odds. The only way the reason why it's it's even we're talking about it is because of the way security gets in incorporated into this developer workflow. People talk about a lot of the security tools being developer friendly, but you have to look at it holistically for the entire workflow from code to deploy and how this gets integrated. So to me, if this is done right, if it's built into the process, if you get the data at the right point of time, all the developers wanna do the right things, and security should not be at odds with the developer experience. Developers are are very smart and capable. They understand that they need to, worry about the signals at the right time and the right, I guess, guidance on what they need to do, they'll they'll easily do it and without an impact to developer experience, and it will improve their overall satisfaction. And if you're a developer or, you know, on on platform team at an organization and and you're you're watching this webinar today, you're seeing this call, and you feel like this should be a a higher priority initiative in your organization. Who's the who who do you see as, like, the ultimate stakeholder often in organizations? Like, what senior executive? Who is it that they should try to lobby and and drive this initiative? In most organizations, there is kind of two or three key, types of stakeholders. The fun the first one would be sort of the central developer services, developer experience, DevOps team that is responsible for your, source code management, CI CD infrastructure. Then you also have the, application security, information security, CISO stakeholders that are responsible for the security aspect. And then you also have the, business aligned CIOs or tech leadership. So those are kind of the three groups that we wanna look at. One of the questions earlier on was, what is actually most successful? I've seen organizations, including in my own experience, when you have the DevOps developer services function working closely with the application security software supply chain function within the same team, that really results in the most benefits and the most progress. And if you're a developer, I think typically you would go to sort of your up your leadership chain and say, hey. Here are some ways we can improve that. They will, ideally, channel you into this central, I would call it, dev tech ops team, which includes the developer services, dev ops systems, and application security, scanning, supply chain, all in one place. And those teams are going to be proactively engaging with the developers on the ground to remove friction, to improve that workflow, and to really lead to, processes and tools and and mechanisms that are designed to help the developers do the right thing. Great. That makes a ton of sense. I'll take one final question here. I got one from me. Somebody asking how is Chainguard involving themselves in open compliance frameworks like SALSA? We are absolutely involved in a lot of the key frameworks, and actually to the point where our our cofounders, were cocreators of some of these frameworks. So SALSA, Kim Lewandowski, who's, who's our our chief product officer, is, was a cocreator of Salsa while at Google. And, and our CEO, our CEO, Dan Lawrence, helped create, sigstore. So there's there's a lot of of open source security, kind of innovation and knowledge, in in sort of these, compliant in these frameworks, at the company, which is, which is exciting. Great. Well, I think that kinda covers all of, the, questions, and thanks a lot for your time, George. It was a really informative discussion. Hope everybody who joined, enjoyed the enjoyed the hour here. I can't believe the hour flew by here. I wasn't sure if we we could talk for an hour, but I guess I guess there was enough to talk about, which is awesome. Yeah. No. Thank you for having me. And, one the key takeaway here is if you're not already thinking about doing software supply chain security in your organization, please start to do that and reach out to GitLab and Cheynga, and we've got a lot of acceleration we can provide in this space. Alright. Great. Yep. Absolutely. Okay. Thanks a lot, everybody. Take care.