tag:blogger.com,1999:blog-4091205522790099892024-03-18T15:10:16.875+02:00Ahmed AbouZaid!Unknownnoreply@blogger.comBlogger100125tag:blogger.com,1999:blog-409120552279009989.post-5948111967146690662023-12-31T01:30:00.034+02:002024-01-04T03:00:14.853+02:002023 Highlights<div class="separator">
<div style="clear: right; float: right; margin-bottom: 1em; margin-left: 2em; text-align: center; width: 30%">
<img border="0" src="https://pics.craiyon.com/2023-09-14/2bbc6d1fb6e846e98acdeec8f9144f68.webp" />
<br/>
Image generated with <a target="_blank" href="https://www.craiyon.com/image/grBNsXALQgCTAqxfwDi0fA">Craiyon</a>.
</div>
</div>
<p>
Finally, 2023 is over! What a year! One more crazy year, but it was
the culmination of many actions I've started previously.
</p>
<p>
Starting with the fun facts ... this is the post no. 100!
</p>
<p>
After 9 years of tech blogging in English (I had another 10 in Arabic
before that). It should be 108 posts since the plan was to post 1 per
month, but I missed some, still not bad!
</p>
<h2 id="top-5-highlights-in-2023">Top 5 highlights in 2023</h2>
<h3 id="1-career">1. Career</h3>
<p align="center">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh4KQa5Bev8V62Ho9CE-BhOe7TU6XJtkLEDRp7Z9wBqsKxlJd9EgAUMS42dlIb-kUiKIsaVzF9Nf2SXDcVqMTegjy-tWtTGZLhNJ9Mj9CeEj84zBDgZMoS6a-Y5wBfXLyDMK8-2PCJGL5jARAD9mLAU2hTt2K-eoMzS76LH3wdFx1nwUjd5Its-hWugaAz2/s1024/career-directions.jpg" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" width="80%" data-original-height="634" data-original-width="1024" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh4KQa5Bev8V62Ho9CE-BhOe7TU6XJtkLEDRp7Z9wBqsKxlJd9EgAUMS42dlIb-kUiKIsaVzF9Nf2SXDcVqMTegjy-tWtTGZLhNJ9Mj9CeEj84zBDgZMoS6a-Y5wBfXLyDMK8-2PCJGL5jARAD9mLAU2hTt2K-eoMzS76LH3wdFx1nwUjd5Its-hWugaAz2/s1024/career-directions.jpg"/>
</a>
Image by <a href="https://www.freepik.com/free-vector/decision-making-abstract-concept_12084832.htm">vectorjuice</a> on Freepik
</p>
<p>
In 2022, I formed the <tt>Distribution team</tt> at <a
href="https://camunda.com/">Camunda</a> which is responsible for
building and deploying the <a
href="https://camunda.com/platform/">Camunda Platform 8</a> Self-Managed
which is an umbrella <a
href="https://github.com/camunda/camunda-platform-helm">Helm chart</a>
with 10+ systems. In 2023, the biggest challenge was increasing the team
headcount to 4 and building the workflow to handle that!
</p>
<h3 id="2-academia">2. Academia</h3>
<p align="center">
<iframe width="80%" height="315" src="https://www.youtube.com/embed/VDiGHaa68yQ?si=IwShOTk_p9VC_MBE" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
</p>
<p>
I've always been into data! So in 2020, I started a part-time
master's in Data Engineering at <a
href="https://www.napier.ac.uk/">Edinburgh Napier University</a>. After
almost 3 years, I was awarded a
<tt>Master of Science with Distinction in Data Engineering</tt> π.
In June, I got the result after I successfully defended my dissertation
titled <a
href="https://tech.aabouzaid.com/2023/07/my-master-dissertation-modern-data-platform-with-dataops-kubernetes-and-cloud-native-ecosystem.html">Modern
Data Platform with DataOps, Kubernetes, and Cloud-Native Ecosystem</a>.
Then in October, I traveled to Scotland to attend the graduation
ceremony. All I can say today is that it was a great experience by all
means! (and that's actually why I had some unusual gaps in blog posts in
2023 because I was working on the master's thesis).
</p>
<h3 id="3-mentorship">3. Mentorship</h3>
<p align="center">
<a target="_blank" href="https://tech.aabouzaid.com/2023/12/become-devops-engineer-with-dynamic-devops-roadmap-career.html" style="display: block; padding: 1em 0;">
<img alt="Dynamic DevOps Roadmap" border="0" width="80%" src="https://raw.githubusercontent.com/DevOpsHiveHQ/dynamic-devops-roadmap/main/img/dynamic-devops-roadmap.png"/>
</a>
</p>
<p>
In the last 5 years, I mentored many people in different career
stages (starting their first job, career shift, moving to another work
style or company). Almost every day, I see people struggling on their
way to the DevOps field. I already wrote <a
href="https://tech.aabouzaid.com/2023/06/your-devops-learning-roadmap-is-broken.html">why
the DevOps linear roadmaps are broken by default</a>! So I decided to
fix that and launched a <a
href="https://github.com/DevOpsHiveHQ/dynamic-devops-roadmap">Dynamic
DevOps Roadmap to become a DevOps Engineer</a> which under the <a
href="https://devopshive.net/">DevOps Hive</a> identity! The nice thing
that I saw many people already like the idea and want to do it that
way!
</p>
<p align="center">
<a href="https://github.com/DevOpsHiveHQ/dynamic-devops-roadmap" imageanchor="1">
βΉοΈ Check out the Dynamic Roadmap content βΉοΈ
</a>
</p>
<h3 id="4-public-speaking">4. Public Speaking</h3>
<p>
<a target="_blank" href="https://jobstack.talentsarena.net/">Jobstack</a> is one of
my favorite tech events all the time! And as usual, this year was
awesome! As a speaker, I had 2 sessions, and as an attendee, I enjoyed
many sessions on different topics.
</p>
<ul>
<li><a
href="https://tech.aabouzaid.com/2023/11/platform-engineering-manage-your-infrastructure-using-kubernetes-and-crossplane.html">Platform
Engineering: Manage your infrastructure using Kubernetes and
Crossplane</a></li>
<li><a
href="https://tech.aabouzaid.com/2023/11/devops-is-not-only-a-culture.html">DevOps
is not only a culture</a>.</li>
</ul>
<h3 id="5-activities">5. Activities</h3>
<p>
Besides these highlights, I had some nice stuff during the year. For
example:
</p>
<ul>
<li>Developed 2 Kustomize plugins in <a
href="https://tech.aabouzaid.com/search/label/Golang">Golang</a> which
are <a
href="https://github.com/DevOpsHiveHQ/kustomize-plugin-merger">Merger</a>
(a schemaless strategic merge plugin) and <a
href="https://github.com/DevOpsHiveHQ/kustomize-plugin-kubeconform/tree/main">KubeconformValidator</a>
(A plugin built around Kubeconform to validate manifests schema within
Kusomize).</li>
<li>Developed <a
href="https://github.com/asdf-community/asdf-plugin-manager">ASDF Plugin
Manager</a>, a plugin manager for the <tt>asdf</tt> version
manager!</li>
<li>Like <a
href="https://github.com/DevOpsHiveHQ/awesome-kustomize">Kustomize
Awesome list</a>, I've created <a
href="https://github.com/DevOpsHiveHQ/awesome-crossplane">Crossplane
Awesome list</a> too!</li>
</ul>
<p>
And since we are on this topic, here are the top 5 visited blog posts
in 2023!
</p>
<h2 id="top-5-posts-in-2023">Top 5 posts in 2023</h2>
<ol type="1">
<li><a
href="https://tech.aabouzaid.com/2022/08/2-ways-to-route-ingress-traffic-across-namespaces.html">2
ways to route Ingress traffic across namespaces - Kubernetes</a></li>
<li><a
href="https://tech.aabouzaid.com/2020/04/validate-format-lint-and-test-terraform-iac-ci.html">Validate,
format, lint, secure, and test Terraform IaC - CI/CD</a></li>
<li><a
href="https://tech.aabouzaid.com/2023/06/your-devops-learning-roadmap-is-broken.html">Your
DevOps learning roadmap is broken! - Career</a></li>
<li><a
href="https://tech.aabouzaid.com/2016/01/continuous-delivery-and-maturity-model.html">Delete
a manifest from Kustomize base - Kubernetes</a></li>
<li><a
href="https://tech.aabouzaid.com/2020/09/3-ways-to-customize-off-the-shelf-helm-charts-with-kustomize-kubernetes.html">3
ways to customize off-the-shelf Helm charts with Kustomize -
Kubernetes</a></li>
</ol>
<p>
The same as last year, <a
href="https://tech.aabouzaid.com/2022/12/2022-highlights.html#top-5-posts-in-2022">2022</a>,
<tt>Kustomize</tt> posts are still in the top; that's because there
is not much content about it even though it's built-in kubectl now!
(since v1.14). That's why I created a <a
href="https://github.com/DevOpsHiveHQ/awesome-kustomize">Kustomize
Awesome list</a>, which is a curated and collaborative list of awesome
Kustomize resources.
</p>
<p>
But this year, <a target="_blank" href="https://tech.aabouzaid.com/2023/">2023</a>,
for the first time, a blog post posted in the same year appears in the
top 5 posts! Which is discussing how the linear DevOps learning roadmap
is broken by default! That's why I had a follow-up post showing the
solution that suggests a better way to <a
href="https://tech.aabouzaid.com/2023/12/become-devops-engineer-with-dynamic-devops-roadmap-career.html">Become
a DevOps Engineer with the Dynamic DevOps Roadmap</a>. β
</p>
<h2 id="whats-next">What's next?</h2>
<p>
As usual, I don't plan the whole year in advance, I just put some
high-level directions then work on them as I go in Agile iterative style
(yes, I use Agile for personal goals too).
</p>
<p>
What I know already is that I need to reward myself and take a good
break after the master π
</p>
<p>
Also, I want to put more effort to grow the <a
href="https://devopshive.net/">DevOps Hive</a> community to help more
people to land their first DevOps Engineer role!
</p>
<hr />
<p>
Enjoy, and looking forward to 2024!
</p>
Ahmed AbouZaidhttp://www.blogger.com/profile/06013456198042893076noreply@blogger.com0tag:blogger.com,1999:blog-409120552279009989.post-82757878224142937802023-12-12T01:30:00.007+02:002024-01-11T08:54:55.112+02:00Become a DevOps Engineer with the Dynamic DevOps Roadmap - Career<p align="center">
<a target="_blank" href="https://github.com/DevOpsHiveHQ/dynamic-devops-roadmap/blob/main/img/dynamic-devops-roadmap.png" style="display: block; padding: 1em 0;">
<img alt="Dynamic DevOps Roadmap" border="0" width="80%" src="https://raw.githubusercontent.com/DevOpsHiveHQ/dynamic-devops-roadmap/main/img/dynamic-devops-roadmap.png"/>
</a>
</p>
<p>
A couple of months ago, I discussed <a
href="https://tech.aabouzaid.com/2023/06/your-devops-learning-roadmap-is-broken.html">why
all linear DevOps learning roadmaps are broken by default</a> (like
roadmap.sh/devops)! In that post, I've discussed the issue, where it
comes from, and the solution I tried for over 5 years!
</p>
<p>
The solution is simply switching to an <tt>MVP-style learning
roadmap</tt> where you learn in iterations and touch multiple parts
simultaneously (not necessarily equally). Each iteration should focus on
a primary topic while exploring related side topics. So after a month,
you already know about each topic in the roadmap, and after the 2nd
month, you have the basics, and after the 3rd month, you have a good
base, and so on.
</p>
<p>
Based on my mentorship experience in the last 5 years, I've concluded
that any "tool-based" approach to deal with DevOps will fail miserably.
The Cloud-Native landscape is getting bigger and bigger every day, and
there is no way to deal with it that way.
</p>
<p>
For that reason, I've reviewed all docs from the previous mentorship
docs I held in the past (I worked with people starting their first job,
career shift, or moving to another work style or company) and built the
ultimate missing solution!
</p>
<p align="center">
<a target="_blank" href="https://github.com/DevOpsHiveHQ/dynamic-devops-roadmap" imageanchor="1">
β Check out the Dynamic Roadmap content β
</a>
</p>
<p>
This roadmap is <tt>polymorphic</tt>, which means it's designed
to work in different modes. It depends on how fast you want to go.
</p>
<ol type="1">
<li><strong>Self-Learning Course:</strong> In this mode, you are not
expected to have DevOps experience, and you want to go from zero to
hero, transforming your knowledge to land your first job as a DevOps
Engineer.</li>
<li><strong>Hands-on Project:</strong> In this mode, you have some
experience with DevOps (usually between 1-2 years of work experience),
but you want to step up your skills with real hands-on industry-grad
projects to learn DevOps in a pragmatic manner.</li>
<li><strong>Mentorship Program:</strong> In this mode, the previous two
modes are covered (that means it could be only for the project or the
whole roadmap) but with support from a mentor! The project will provide
you with a DevOps expert who will guide you in following up on your
progress and personalizing your learning plan.</li>
</ol>
<p>
The whole idea is about the <tt>Learning by Doing</tt> method
(aka Problem-based Learning), which is done in iterative phases where
you learn as you go and cover the whole DevOps cycle like Code,
Containers, Testing, Continuous Integration, Continuous Delivery,
Observability, and Infrastructure.
</p>
<p>
The project is a work in progress; more details are in the <a
href="https://github.com/DevOpsHiveHQ/dynamic-devops-roadmap#status">status
section</a>.
</p>
<p>
I hope that the project helps more people start their DevOps
engineering careers! The market is thirsty for it already!
</p>
<p align="center">
<a target="_blank" href="https://github.com/DevOpsHiveHQ/dynamic-devops-roadmap" imageanchor="1">
β Check out the Dynamic Roadmap content β
</a>
</p>
<p>
Happy DevOps-ing :-)
</p>
Ahmed AbouZaidhttp://www.blogger.com/profile/06013456198042893076noreply@blogger.com0tag:blogger.com,1999:blog-409120552279009989.post-20634466170118256712023-11-12T23:30:00.023+02:002023-11-17T03:12:39.985+02:00DevOps is not only a culture - Discussion Panel
<p>
Today is my second session <a
href="https://jobstack.talentsarena.net/">JobStack 2023</a> after my
previous one yesterday titled "<a
href="https://tech.aabouzaid.com/2023/11/platform-engineering-manage-your-infrastructure-using-kubernetes-and-crossplane.html">Platform
Engineering: Manage your infrastructure using Kubernetes and
Crossplane</a>", but this time it was a discussion panel with <a
href="https://www.linkedin.com/in/ahmadaabed">Ahmad Aabed</a> (CTO of <a
href="https://www.linkedin.com/company/zyda-inc/">Zyda</a>).
</p>
<div class="separator" style="clear: both;">
<a target="_blank" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhfZqz47ZeYyY90Ri0vlZQemOWpunzsPG8b8ORi7DXUq875kycfZmmLztFJIUmwLAsW3LLwpCyjZTD88m2W3SXl77sEyxFIDA2K_jkWfykEVNe1CFOZRy3uCGA5TMeEq9W8bgssyQacMTH_lY4_4sQ7dYwF_Rzo5JeTvSND-WMclftrp-98v0HIpg016Yj5/s2048/devops-is-not-only-a-culture-cover.jpg" style="display: block; padding: 1em 0px; text-align: center;" target="_blank">
<img alt="" border="0" data-original-height="2048" data-original-width="2048" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhfZqz47ZeYyY90Ri0vlZQemOWpunzsPG8b8ORi7DXUq875kycfZmmLztFJIUmwLAsW3LLwpCyjZTD88m2W3SXl77sEyxFIDA2K_jkWfykEVNe1CFOZRy3uCGA5TMeEq9W8bgssyQacMTH_lY4_4sQ7dYwF_Rzo5JeTvSND-WMclftrp-98v0HIpg016Yj5/w400-h400/devops-is-not-only-a-culture-cover.jpg" width="400" />
</a>
</div>
<p>
For a long time, DevOps methodologies have been the driving force
behind innovation and efficiency in modern software development. You've
likely encountered the popular jargon: "DevOps is not a role; it is a
culture"! However, the truth is that DevOps has much more!
</p>
<p>
This session dives deep into the pillars world of DevOps, going
beyond the cultural aspect to explore practices, technology stacks,
mindset, and more. This is an open discussion where we will share (and
encourage you to do so) the experiences of implementing DevOps. In other
words, what practical steps have you taken to embrace the power of
DevOps in your organization?
</p>
<div class="separator" style="clear: both;">
<a target="_blank" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhHtUAn5GaGvKWLN8PhyphenhypheniKyptnvV5rY7qakKcoX8z4iWzCA_6aRwFvfGE8xhhNMjghwqPyNPI9i5Mdzq9YezpORCkzcKPAPA3kj51CFjlPSFw7Z6XZ1hB4xpmAVqbTgd8wBHkBAthHQfK0ah45NBJadACsZaC9JSwpEMlgzUQTC6K-AVE18e7Iav66xTzGB/s1920/devops-is-not-only-a-culture-shot.png" style="display: block; padding: 1em 0px; text-align: center;">
<img alt="" border="0" data-original-height="1080" data-original-width="1920" height="360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhHtUAn5GaGvKWLN8PhyphenhypheniKyptnvV5rY7qakKcoX8z4iWzCA_6aRwFvfGE8xhhNMjghwqPyNPI9i5Mdzq9YezpORCkzcKPAPA3kj51CFjlPSFw7Z6XZ1hB4xpmAVqbTgd8wBHkBAthHQfK0ah45NBJadACsZaC9JSwpEMlgzUQTC6K-AVE18e7Iav66xTzGB/w640-h360/devops-is-not-only-a-culture-shot.png" width="640" />
</a>
<div style="text-align: center;">
A shot from the recoding
</div>
</div>
<p>
At the end, I'd like to share a couple of posts I wrote before in
that regard (DevOps methodologies in action):
</p>
<ul>
<li><a
href="https://tech.aabouzaid.com/2020/05/devops-is-not-just-a-culture-and-it-is-also-a-role.html">DevOps
is not just a culture and it is also a role!</a>: The elements of DevOps
methodologies and how they could be applied differently.</li>
<li><a
href="https://tech.aabouzaid.com/2019/08/your-company-doesnt-need-hero-but-team-work-culture.html">Your
company doesn't need a hero but a team!</a>: What are heroism signs in
companies and how to fix that?</li>
<li><a
href="https://tech.aabouzaid.com/2019/05/devops-and-change-management-agile.html">J-Curve
and change management</a>: How applying DevOps needs change and project
management skills.</li>
<li><a
href="https://tech.aabouzaid.com/2019/11/let-your-team-evolve-career.html">Let
your team evolve!</a>: Why so many people don't evolve appropriately in
their career?</li>
<li><a
href="https://tech.aabouzaid.com/2020/01/the-road-to-cut-production-incidents-by-67-percent-and-reduce-downtime-to-zero.html">The
Road to Cut Production Incidents by 67% and Reduce Downtime to Zero</a>:
I used that as a particular example in the discussion.</li>
</ul>
<p>
Enjoy :-)
</p>
Ahmed AbouZaidhttp://www.blogger.com/profile/06013456198042893076noreply@blogger.com0tag:blogger.com,1999:blog-409120552279009989.post-4546186846990131182023-11-11T21:00:00.005+02:002023-12-05T18:20:05.124+02:00Platform Engineering: Manage your infrastructure using Kubernetes and Crossplane - Presentation<img alt="" style="display: none;" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjcXcQaoztxiduzBsURbtbXDemViVzNMf8QaGnoOp68C-YfnJSHvG0Gsy9cbb13cuI9CqOtb47qC4yk01rGGK8GDIeargQ4uYsdpD3yuDLbcRryAjIAoV7u4xC8oc1wd5nwStrGdhix_hy7vkhe0hiVOcjwx3XysNt3YpCcSgz0q4tlh8lVPggs5DATWN3_/s640/Platform-Engineering-Manage-your-infrastructure-using-Kubernetes-and-Crossplane-Ahmed-AbouZaid.png"/>
<p>
Just fresh out of the kitchen, part of <a
href="https://jobstack.talentsarena.net/">Jobstack 2023</a>, today I
conducted a session about a great tool ... <a
href="https://www.crossplane.io/">Crossplane</a>, the open-source
control plane!
</p>
<p>
I've been using Crossplane for over a year and a half, and it helped
me a lot to manage my infrastructure without the need to use Terraform
(I still love Terraform, but it wasn't the best for my use case).
</p>
<p>
In this session I shed the light on how Crossplane could unify
infrastructure management within Kubernetes.
</p>
<p>
I gave a brief how Crossplane extends the functionality of Kubernetes
and allows you to create external infrastructure. You can create Cloud
resources the same way you create Kubernetes resources! I really love
its declarative, cloud-native, GitOps-friendly approach to code-driven
infrastructure management.
</p>
<h3 id="agenda">Agenda:</h3>
<ol type="1">
<li>Scenario</li>
<li>What is Crossplane?</li>
<li>How it look like?</li>
<li>Crossplane Concepts</li>
<li>How Crossplane Works</li>
<li>Pros and Cons</li>
<li>Conclusion</li>
<li>Resources</li>
<li>Questions</li>
</ol>
<p>
<strong>Note:</strong> For more resources, checkout this <a
href="https://github.com/DevOpsHiveHQ/awesome-crossplane">Awesome
Crossplane list</a>.
</p>
<hr />
<div style="text-align: center;">
<iframe src="https://www.slideshare.net/slideshow/embed_code/key/jdJSugEFHFygjB" width="640" height="474" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width: 100%;" allowfullscreen>
</iframe>
<div style="margin-bottom:5px">
The slides of
<a target="_blank" href="https://www.slideshare.net/ahmed-abouzaid/platform-engineering-manage-your-infrastructure-using-kubernetes-and-crossplane" title="Platform Engineering: Manage your infrastructure using Kubernetes and Crossplane" target="_blank">Platform Engineering: Manage your infrastructure using Kubernetes and Crossplane</a> by <a target="_blank" href="https://www.slideshare.net/ahmed-abouzaid" target="_blank">Ahmed AbouZaid</a>
</div>
</div>
<br />
<div class="separator" style="clear: both;">
<a target="_blank" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbZ7uN7WdsZQJk0rp_1uAA6pa525PwHj2haoH97iHx3Ft7yAKhWtC8dRL7kCDrl8uZHKEDk86XzcMkAyVUzwuLaWuWNkrfdt_-wCfvl_iHgjhXk6h4VPzMAe_ugSBhvUuxajfkAUbpMs2QdfYr6EBGvi6ZhaIqGf7bMBOanrq_WWCsThdIYMb1TiDSg_YB/s1920/Platform-Engineering-Manage-your-infrastructure-using-Kubernetes-and-Crossplane-Slide07-Ahmed-AbouZaid.png" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="1080" data-original-width="1920" height="361" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbZ7uN7WdsZQJk0rp_1uAA6pa525PwHj2haoH97iHx3Ft7yAKhWtC8dRL7kCDrl8uZHKEDk86XzcMkAyVUzwuLaWuWNkrfdt_-wCfvl_iHgjhXk6h4VPzMAe_ugSBhvUuxajfkAUbpMs2QdfYr6EBGvi6ZhaIqGf7bMBOanrq_WWCsThdIYMb1TiDSg_YB/w640-h361/Platform-Engineering-Manage-your-infrastructure-using-Kubernetes-and-Crossplane-Slide07-Ahmed-AbouZaid.png" width="640" /></a>
<div style="text-align: center;">
A Shot from the recoding
</div>
</div>
<h3 id="conclusion">Conclusion:</h3>
<p>
Crossplane is a great framework for managing infrastructure using the
Kubernetes style and benefits from the that ecosystem (ArgoCD, Helm,
Kustomize, etc.).
</p>
<p>
There are many use cases where it can perfectly fit in already. And
at the time of writing these words (November 2023), the <a
href="https://marketplace.upbound.io/">Marketplace</a> has numerous
enterprise and community providers configurations. Also, <a
href="https://docs.crossplane.io/latest/concepts/composition-functions/">Composition
Functions</a> graduated to beta.
</p>
<p>
However, it's a relatively new ecosystem and still evolving, so it
might not be the optimal solution for every workload. But it's probably
a matter of time to grow more. So, if it's not your fit now, consider
revisiting in the future.
</p>
<hr />
<p>
That's it, enjoy :-)
</p>
Ahmed AbouZaidhttp://www.blogger.com/profile/06013456198042893076noreply@blogger.com0tag:blogger.com,1999:blog-409120552279009989.post-87864062510204659522023-09-09T01:30:00.011+03:002023-09-16T02:59:33.281+03:00πMergerπ, a schemaless strategic merge plugin - Kustomize<div class="separator" style="clear: both;">
<a target="_blank" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5dMeUEQL2Z51s94oLF5f_EgyhFCegbunx8RjEL6VNuytfPonrdt7Nsiapjf2BCbJRK3LgDYwKsyigFN0Qn9FOJsy0qYR_FxZEwDzkuO2H9f437tLVQXgrXrutAnlkuszMfG-aSGtHuhmgUwpdjOzoFHT_sNXwSdSTdqIi2g1eHoFj8MSoh9oEYGii4w/s600/kustomize.png" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" width="600" data-original-height="165" data-original-width="600" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5dMeUEQL2Z51s94oLF5f_EgyhFCegbunx8RjEL6VNuytfPonrdt7Nsiapjf2BCbJRK3LgDYwKsyigFN0Qn9FOJsy0qYR_FxZEwDzkuO2H9f437tLVQXgrXrutAnlkuszMfG-aSGtHuhmgUwpdjOzoFHT_sNXwSdSTdqIi2g1eHoFj8MSoh9oEYGii4w/s600/kustomize.png"/>
</a>
</div>
<p>
Kustomize is a great tool. I've been using Kustomize for
almost 4 years and am happy with it. However, it's known for its strict
merging methods, where <a
href="https://tech.aabouzaid.com/2022/11/set-openapi-patch-strategy-for-kubernetes-custom-resources-kustomize.html">it
needs to have an OpenAPI schema to merge files properly</a>.
</p>
<p>
There were many use cases where I needed a more flexible way to merge
resources (away from Kustomize's strict merging). So, I've developed a
new Kustomize generator plugin (Containerized KRM and Exec KRM) that
extends Kustomize's merge strategies (schemaless StrategicMerge).
</p>
<p>
<strong>I wanted to:</strong>
</p>
<ul>
<li>Generate multiple resources from a single resource without the need
to multi-import (you can patch multiple resources with a single patch
but not the other way around)</li>
<li>An easy way to merge CustomResources without the need to provide the
OpenAPI schema for it (that's actually <a
href="https://tech.aabouzaid.com/2022/11/set-openapi-patch-strategy-for-kubernetes-custom-resources-kustomize.html">a
lot of work</a>)</li>
<li>An easy way to merge non-k8s resources and put them in a
ConfigMap.</li>
<li>A way to split long files into smaller ones.</li>
</ul>
<p align=center>...
</p>
<p>
Say Hi to <a
href="https://github.com/aabouzaid/kustomize-plugin-merger">πMergerπ</a>
</p>
<p>
Merger is a generator provides schemaless merges with different
strategies (StrategicMerge) like replace, append, and combine.
</p>
<p>
Here is an example:
</p>
<pre class="plain-pre">
apiVersion: generators.kustomize.aabouzaid.com/v1alpha1
kind: Merger
metadata:
name: merge
annotations:
config.kubernetes.io/function: |
container:
image: ghcr.io/aabouzaid/kustomize-generator-merger
mounts:
- type: bind
src: ./
dst: /mnt
spec:
resources:
- name: example
input:
# Available options: overlay,patch.
# - Overlay: Produce multiple outputs by merging each source with the destination.
# - Patch: Produce a single output by merging all sources together then with the destination.
method: overlay
files:
# The same as in the KRM container above, omit it if Exec KRM is used.
root: /mnt
sources:
- src01.yaml
- src02.yaml
destination: dst.yaml
merge:
# Available options: replace,append,combine.
# - Replace: All keys in source will merge and replace what's in the destination.
# - Append: Maps from source merged with destination, but the lists will be appended from source to destination.
# - Combine: Maps from source merged with destination, but the lists will be combined together.
strategy: combine
output:
# Available options: raw.
# In the next releases also ConfigMap and Secret will be supported.
format: raw
</pre>
<p>
For more details, check the <a
href="https://github.com/aabouzaid/kustomize-plugin-merger#common-use-cases">common
use cases</a> section.
</p>
<p align=center>...
</p>
<p>
<strong>Some takeaways I learned while developing this
project:</strong>
</p>
<ul>
<li><a target="_blank" href="https://book.kubebuilder.io/reference/markers">KubeBuilder
markers</a> could be used with the client side to auto-generate the
OpenAPI YAML schema from the code.</li>
<li>Golang compression methods (like <a
href="https://words.filippo.io/shrink-your-go-binaries-with-this-one-weird-trick/">UPX
and LZMA</a>) can reduce the binary size up to 80% compared to the
standard build method.</li>
<li><a
href="https://itnext.io/using-cosign-keyless-signature-with-github-actions-de675a1e18a2">Cosign
keyless artifacts sign</a> is pretty easy to add to the CI pipeline (no
need to manage any extra keys).</li>
<li><a target="_blank" href="https://securityscorecards.dev/">OpenSSF Scorecard</a>
offers a great integration assessing the security health metrics of
open-source projects.</li>
</ul>
<p>
Enjoy :-)
</p>
Ahmed AbouZaidhttp://www.blogger.com/profile/06013456198042893076noreply@blogger.com0tag:blogger.com,1999:blog-409120552279009989.post-73370033989064792172023-08-08T01:30:00.018+03:002023-12-05T18:18:59.311+02:00Helm chart keyless signing with Sigstore/Cosign - DevSecOps<div class="separator" style="clear: both; text-align: center;">
<a target="_blank" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhGSVfieFTUOTLbfxF7P06tz9LMdhf0rxoubZxcPUl8qkyTqbrx3lqq3JzjusQQ8hd_aicqy_A33OwkGCv-RgImMUjUjFqUnnit2NgufepKQFuQPOJfg2Z1D7Ta-NP1bgNCgeSK4utJ4CkxlpKE9f1g6Hj5xD8waQPxthh0sh-PlRXH-jmmcdWq030rHlqP/s1200/helm-chart-keyless-signing-with-sigstore-cosign.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="627" data-original-width="1200" height="334" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhGSVfieFTUOTLbfxF7P06tz9LMdhf0rxoubZxcPUl8qkyTqbrx3lqq3JzjusQQ8hd_aicqy_A33OwkGCv-RgImMUjUjFqUnnit2NgufepKQFuQPOJfg2Z1D7Ta-NP1bgNCgeSK4utJ4CkxlpKE9f1g6Hj5xD8waQPxthh0sh-PlRXH-jmmcdWq030rHlqP/s1200/helm-chart-keyless-signing-with-sigstore-cosign.png" width="640" /></a>
</div>
<p>
Software supply chain security has been one of the hot topics because
of the continues attacks and exploits shown in the latest few years. So
having a proper security practices like signing artifacts in today's
CI/CD pipeline is not a luxury! Which became a standard DevSecOps
practice.
</p>
<p>
However, for a long time it needed a lot of work to implement those
practices, hence, projects like <a target="_blank" href="https://openssf.org/">Open
Source Security Foundation</a> (OpenSSF) which created security
framework like <a target="_blank" href="https://slsa.dev/">Supply-chain Levels for
Software Artifacts</a> (SLSA) and <a
href="https://www.sigstore.dev/">Sigstore</a> which created tools like
<a target="_blank" href="https://docs.sigstore.dev/signing/quickstart/">Cosign</a> to
standardize and reduce that fatigue.
</p>
<p>
Today I want to shed some lights on Helm chart signing with Cosign
(part of Sigstore project). This post will focus on GitHub Actions and 2
ways for Helm chart signing based on the type of the Helm registry
(simple or OCI-based).
</p>
<ul>
<li><a href="#1-intro">1. Intro</a></li>
<li><a href="#2-identity-setup">2. Identity setup</a></li>
<li><a href="#31-keyless-signing-for-simple-helm-repository">3.1.
Keyless signing for simple Helm repository</a></li>
<li><a href="#32-keyless-signing-for-oci-based-helm-repository">3.2.
Keyless signing for OCI-based Helm repository</a></li>
<li><a href="#4-recap">4. Recap</a></li>
</ul>
<h2 id="1-intro">1. Intro</h2>
<p>
Before starting, what is actually <a
href="https://github.com/sigstore/cosign">Cosign</a>? I will start with
quoting from the project website:
</p>
<blockquote>
<p>
Cosign is a command line utility that can sign and verify software
artifact, such as container images and blobs.
</p>
</blockquote>
<p>
Cosign aims to make signatures <strong>invisible
infrastructure</strong>, and one of its main features what's known as
"Keyless signing". Keyless signing means rather than using keys like
GPG/PGP, it uses associates identities via OpenID Connect (i.e. it auth
against providers like Microsoft, Google, and GitHub to issues
short-lived certificates binding an ephemeral key).
</p>
<p>
Cosign could be used as a CLI also integrated part of other
build/release tools like <a
href="https://goreleaser.com/">GoReleaser</a>.
</p>
<h2 id="2-identity-setup">2. Identity setup</h2>
<p>
As mentioned, no need for signing keys here, Cosign will use the
identity context like GitHub. Let's take GitHub Actions as a base here,
but it is worth mentioning that Cosign works with many identity
providers.
</p>
<p>
From the GitHub Actions workflow point of view, you just need to have
job permission <tt>id-token: write</tt>, which allows the job to use
OICD and generate JWT (which works as a key).
</p>
<pre class="plain-pre">
name: Sign Helm chart artifact
[...]
jobs:
sign:
name: Sign
permissions:
id-token: write
[...]
</pre>
<p>
You can find a full example in the repo: <a
href="https://github.com/DevOpsHiveHQ/cosign-helm-chart-keyless-signing-example">https://github.com/DevOpsHiveHQ/cosign-helm-chart-keyless-signing-example</a>
</p>
<h2 id="31-keyless-signing-for-simple-helm-repository">
3.1. Keyless
signing for simple Helm repository</h2>
<p>
A simple Helm repository is just a file called
<tt>index.yaml</tt> which reference to the actual chart files URLs.
A popular way for that is using <a
href="https://github.com/helm/chart-releaser">Chart Releaser</a> to host
the Helm charts using GitHub Pages and Releases.
</p>
<p>
The simplest way to sign Helm chart using Cosign is to sign the
artifact then upload the signature to GitHub release page.
</p>
<p>
First, sign the chart file:
</p>
<pre class="plain-pre">
# For explicitly, it's also possible to add '--oidc-provider=github-actions',
# but no need for that, cosign will discover the context if the GH job permission is correct.
cosign sign-blob my-app-1.0.0.tgz --bundle my-app-1.0.0.tgz.cosign.bundle
</pre>
<p>
That will create a bundle file
<tt>my-app-1.0.0.tgz.cosign.bundle</tt> which contains signing
metadata like the signature and certificate (also it's possible to have
separate <tt>sig</tt> and <tt>pem</tt> files) which should be
uploaded to GitHub release page.
</p>
<p>
Now anyone can download the Helm chart file and Cosign bundle file to
verify its integrity:
</p>
<pre class="plain-pre">
cosign verify-blob my-app-1.0.0.tgz \
--bundle my-app-1.0.0.tgz.cosign.bundle \
--certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
--certificate-identity "https://github.com/DevOpsHiveHQ/cosign-helm-chart-keyless-signing-example/.github/workflows/sign.yaml@refs/heads/main"
</pre>
<p>
If the file is valid the command will show <tt>Verified OK</tt>,
otherwise it will show something like <tt>Error: none of the expected
identities matched what was in the certificate ...</tt>.
</p>
<p>
On the other hand, that's still some work! Maybe if <a
href="https://github.com/helm/chart-releaser">Chart Releaser</a> has a
native support for Cosign. Also, the Helm plugin <a
href="https://github.com/sigstore/helm-sigstore">helm-sigstore</a> could
help a bit but you need to download the Helm plugin first.
</p>
<h2 id="32-keyless-signing-for-oci-based-helm-repository">
3.2. Keyless
signing for OCI-based Helm repository</h2>
<p>
Using container image to store config like Helm chart or Terraform
module is one of the brilliant ideas.
</p>
<p>
One of the biggest changes in Helm 3 was ability to use container
registries with OCI support to store and share chart packages. It
started as an experimental feature, but by Helm v3.8.0, OCI support is
enabled by default.
</p>
<p>
So instead that simple <tt>index.yaml</tt> an OCI-based registry
could be used as a Helm repository and chart storage too. Any hosted
registries that support OCI will work for that like Docker Hub, Amazon
ECR, Azure Container Registry, Google Artifact Registry, etc.
</p>
<p>
In that setup, Cosign works a bit differently where it signs the
chart as an OCI image (the same way it signs the Docker images) and
store the signature in the OCI repository. But in that case it only
makes sense to sign the digest not the tags since the digest is
immutable.
</p>
<p>
There are two steps to make that, first push the chart to an
OCI-based Helm repository which generates the chart digest, then push
the signed digest.
</p>
<pre class="plain-pre">
# After login to the registry using "helm registry login ...".
helm push my-app-1.0.0.tgz oci://ttl.sh/charts &> push-metadata.txt
CHART_DIGEST=$(awk '/Digest: /{print $2}' push-metadata.txt)
cosign sign -y "ttl.sh/charts/my-app@${CHART_DIGEST}"
</pre>
<p>
Finally, to verify that:
</p>
<pre class="plain-pre">
cosign verify "ttl.sh/charts/my-app@${CHART_DIGEST}" \
--certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
--certificate-identity "https://github.com/DevOpsHiveHQ/cosign-helm-chart-keyless-signing-example/.github/workflows/sign.yaml@refs/heads/main"
</pre>
<h2 id="4-recap">4. Recap</h2>
<p>
Nowadays, it's critical to have standard DevSecOps practices in the
whole SDLC, and validating integrity is one of the most essential
practices in securing the software supply chain.
</p>
<p>
Any publicly distributed artifact should be signed to ensure the
origin of it, and Helm charts are no different. Cosign keyless signing
made it much easier to apply such practices without the hassles of
managing singing keys.
</p>
Ahmed AbouZaidhttp://www.blogger.com/profile/06013456198042893076noreply@blogger.com0tag:blogger.com,1999:blog-409120552279009989.post-67906019436046590892023-07-07T01:30:00.025+03:002023-07-11T14:57:01.346+03:00My Master's Dissertation: Modern Data Platform with DataOps, Kubernetes, and Cloud-Native Ecosystem<div class="separator" style="clear: both; display: none;">
<a target="_blank" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj4sD8O8Fn2-1Y2P3bKDZDAjMeilWNLE572uV5uV0WhMN871difxtHGPLz2iYJZTnOoT8hHp2j3VYMSyn_E3ArbE7i61MMArVxF4Z6DPFePvqBoDUdBLAXtew34pluFzSedQVgNxLrX0ZERnuvabtGDALIoIZZqw9CAgPhVTo5up4ELQS04NmVwiEklRpxt/s826/ahmed-abouzaid-modern-data-platform-msc-data-engineering-dissertation-cover-short.jpg" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" width="400" data-original-height="795" data-original-width="826" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj4sD8O8Fn2-1Y2P3bKDZDAjMeilWNLE572uV5uV0WhMN871difxtHGPLz2iYJZTnOoT8hHp2j3VYMSyn_E3ArbE7i61MMArVxF4Z6DPFePvqBoDUdBLAXtew34pluFzSedQVgNxLrX0ZERnuvabtGDALIoIZZqw9CAgPhVTo5up4ELQS04NmVwiEklRpxt/s826/ahmed-abouzaid-modern-data-platform-msc-data-engineering-dissertation-cover-short.jpg"/>
</a>
</div>
<p>
Almost 3 years ago (September 2020), I enrolled in a part-time master's
program in data engineering at <a target="_blank" href="https://napier.ac.uk/">Edinburgh
Napier University</a>. Finally, two weeks ago, I got an email informing
me that I successfully completed my master's and the program board of
examiners awarded me <tt>Master of Science with Distinction in Data
Engineering</tt> πππ
</p>
<p>
My graduation ceremony would have been today, but I postponed it to October
2023 for some personal matters. So it's just a matter of time till the
official graduation.
</p>
<p>
So today, I'd like to share my master's dissertation: <a
href="https://dx.doi.org/10.13140/RG.2.2.15360.71689">Modern Data
Platform with DataOps, Kubernetes, and Cloud-Native Ecosystem</a>.
</p>
<p>
The dissertation builds a proof of concept for the core of Modern
Data Platform using DataOps, Kubernetes, and Cloud-Native ecosystem to
build a resilient Big Data platform based on <strong>Data Lakehouse
architecture</strong>, which is the basis for Machine Learning (MLOps)
and Artificial Intelligence (AIOps).
</p>
<p>
It was a super challenging topic, given the fact that Data Lakehouse
architecture emerged just in 2020! But it was great to dive into it.
I've been into data for years, and it's exciting to step up my skills
from <tt>T-Shaped</tt> to <tt>Pi-Shaped</tt>
(<strong>DevOps</strong> and <strong>DataOps</strong>) :-)
</p>
<div class="separator" style="clear: both;">
<a target="_blank" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgKksaBcCAaA8gS5Rk1EB7H6PMQ7uO_NPetMMlEUgQ5SWwLI30PUjeNqXzv9N5UvtMIAOY4XjzIMKr4hxlgT29g4xU2E9uDFBV7HPBOcVDdhT9ZOL5CIOYtfe8abV_peTqXd-4sTHWxWGgKJ29vezVmPsBd2lnVIcBtDgXCyEQkcvGnXYFCaJl4gALOaKXA/s1169/ahmed-abouzaid-modern-data-platform-msc-data-engineering-dissertation-cover.jpg" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" height="600" data-original-height="1169" data-original-width="826" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgKksaBcCAaA8gS5Rk1EB7H6PMQ7uO_NPetMMlEUgQ5SWwLI30PUjeNqXzv9N5UvtMIAOY4XjzIMKr4hxlgT29g4xU2E9uDFBV7HPBOcVDdhT9ZOL5CIOYtfe8abV_peTqXd-4sTHWxWGgKJ29vezVmPsBd2lnVIcBtDgXCyEQkcvGnXYFCaJl4gALOaKXA/s1169/ahmed-abouzaid-modern-data-platform-msc-data-engineering-dissertation-cover.jpg"/>
</a>
</div>
<div class="separator" style="clear: both;">
<a target="_blank" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJSS9M7lZG4KAD8WOTihTBXpilNGYxG-3dGjlYV-aBKCj7uZfUy0299gSlNxGvpCZ6FvNC4dFO-eobSP-oT13yGV9mh8utFjGwfGiSUpoeInaZ8mHXZa5oUDvqDr8CqA4u_-6v0XqBkZTzE8MdrgO8Z_kcZ0UxBsIRZlq-Y5p8FvMdhG-QvPmdW_mMCUGb/s1169/ahmed-abouzaid-modern-data-platform-msc-data-engineering-dissertation-quote.jpg" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" height="600" data-original-height="1169" data-original-width="826" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJSS9M7lZG4KAD8WOTihTBXpilNGYxG-3dGjlYV-aBKCj7uZfUy0299gSlNxGvpCZ6FvNC4dFO-eobSP-oT13yGV9mh8utFjGwfGiSUpoeInaZ8mHXZa5oUDvqDr8CqA4u_-6v0XqBkZTzE8MdrgO8Z_kcZ0UxBsIRZlq-Y5p8FvMdhG-QvPmdW_mMCUGb/s1169/ahmed-abouzaid-modern-data-platform-msc-data-engineering-dissertation-quote.jpg"/>
</a>
</div>
<div class="separator" style="clear: both;">
<a target="_blank" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhouWXfW4fOpUFUHNCiP9a3hD2LUAJJArJ7I_ixWnzYNHYajTx27qAhWMjdVondFvE1sJ-A6GVCFsgPYS5T5FC7kuNajapnJgN7-jBaR6yfbe67EuWxiZUhmWWv0DCGQE5wS1o8BoNVCaFgT3sOVJfDytUvMdKjJ3x7rZd2nT3weDtLVg_fMppU5QoEEao4/s1169/ahmed-abouzaid-modern-data-platform-msc-data-engineering-dissertation-preface.jpg" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" height="600" data-original-height="1169" data-original-width="826" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhouWXfW4fOpUFUHNCiP9a3hD2LUAJJArJ7I_ixWnzYNHYajTx27qAhWMjdVondFvE1sJ-A6GVCFsgPYS5T5FC7kuNajapnJgN7-jBaR6yfbe67EuWxiZUhmWWv0DCGQE5wS1o8BoNVCaFgT3sOVJfDytUvMdKjJ3x7rZd2nT3weDtLVg_fMppU5QoEEao4/s1169/ahmed-abouzaid-modern-data-platform-msc-data-engineering-dissertation-preface.jpg"/>
</a>
</div>
<div class="separator" style="clear: both;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj4RejTvRNIg2aUIe8a9k0aSdHUPphVkbeYpPG4S9mTafvDRxSjImycsAt6Aec_VTW8twK9e36bYYa21SsE5gJ7MOuvM_nGcEnjqHuvuFTuLh9F54kldc_SkMBoHyz5RuV9zf8Dht-muYt8Am9bv2dxAvC8Nc3DOPuWkolIaikLkKSNGaqBH8VkNZ0-GypS/s1169/ahmed-abouzaid-modern-data-platform-msc-data-engineering-dissertation-ack.jpg" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" height="600" data-original-height="1169" data-original-width="827" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj4RejTvRNIg2aUIe8a9k0aSdHUPphVkbeYpPG4S9mTafvDRxSjImycsAt6Aec_VTW8twK9e36bYYa21SsE5gJ7MOuvM_nGcEnjqHuvuFTuLh9F54kldc_SkMBoHyz5RuV9zf8Dht-muYt8Am9bv2dxAvC8Nc3DOPuWkolIaikLkKSNGaqBH8VkNZ0-GypS/s1169/ahmed-abouzaid-modern-data-platform-msc-data-engineering-dissertation-ack.jpg"/>
</a>
</div>
<div class="separator" style="clear: both; text-align: center;">
<p>
Full-text: <a target="_blank" href="https://dx.doi.org/10.13140/RG.2.2.15360.71689">Modern Data Platform with DataOps, Kubernetes, and Cloud-Native Ecosystem</a>
</p>
</div>Ahmed AbouZaidhttp://www.blogger.com/profile/06013456198042893076noreply@blogger.com0tag:blogger.com,1999:blog-409120552279009989.post-6151034605698567042023-06-06T15:41:00.024+03:002024-01-11T08:54:00.599+02:00Your DevOps learning roadmap is broken! - Career
<div class="separator" style="clear: both; display: none;">
<a target="_blank" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjpCyvN1jCJ0pB1YfoKXv1_PL8n0ZHjUGp1_P5_q_Z6TcdW-wLv4zfjUkL8_8huGcp6hMN2e_bQYSTc3WclTQyQ0Swq096qjltjJgtzg6nf_4dwRuid_vPnrfAd59D6LnkPViN-u_P33LCBJD1k5NZYNgA139QdPrKs--kt_8fBx_KAbFW7Bu5XC-g_Hwi3/s2492/dynamic-mvp-roadmap-triple.png" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" width="600" data-original-height="1073" data-original-width="2492" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjpCyvN1jCJ0pB1YfoKXv1_PL8n0ZHjUGp1_P5_q_Z6TcdW-wLv4zfjUkL8_8huGcp6hMN2e_bQYSTc3WclTQyQ0Swq096qjltjJgtzg6nf_4dwRuid_vPnrfAd59D6LnkPViN-u_P33LCBJD1k5NZYNgA139QdPrKs--kt_8fBx_KAbFW7Bu5XC-g_Hwi3/s600/dynamic-mvp-roadmap-triple.png"/>
</a>
</div>
<p>
...
<br>
<b>β οΈ Update β οΈ</b><br>
If you are looking for the solution for this dilemma, then check this out: <a target="_blank" href="https://github.com/DevOpsHiveHQ/dynamic-devops-roadmap" target="_blank">How to become a DevOps Engineer in 2024 with the Dynamic DevOps Roadmap</a>.
<br>
...
</p>
<div class="separator" style="clear: right; float: right; margin-bottom: 1em; margin-left: 2em; text-align: center; width: 35%">
<a style="display: block; padding: 1em 0; text-align: center;" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjrmYT_WtuQJNA05EuIUtZmHv-EFelqDj3pShwW6ONPVzVIHagwEsxXO1G8bPWth2fuHuxaov9Yu3efHWSwlC4WFnlrSoprMG30B8y7sUmStyjHrFlgfZ3UT-Th42x_k5ofrZQleO2874Jb470KNU1HsQ_YfbD31tsXNw7csNTBYozHqh_2WqpaBTxVEJS3/s600/broken-roadmap.png">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjrmYT_WtuQJNA05EuIUtZmHv-EFelqDj3pShwW6ONPVzVIHagwEsxXO1G8bPWth2fuHuxaov9Yu3efHWSwlC4WFnlrSoprMG30B8y7sUmStyjHrFlgfZ3UT-Th42x_k5ofrZQleO2874Jb470KNU1HsQ_YfbD31tsXNw7csNTBYozHqh_2WqpaBTxVEJS3/s400/broken-roadmap.png" />
</a>
Can you read it? Probably not, it's already broken!
</div>
<p>
As of 2023, the <tt>DevOps Engineer</tt> role remains <a
href="https://www.linkedin.com/business/talent/blog/talent-strategy/most-in-demand-jobs">one
of the top 10 most in-demand jobs</a> across all industries (not just
the tech field!) And that was the case for the last 5 years at least and
is expected to continue for the foreseeable future.
</p>
<p>
While DevOps is a hot topic all the time, it's in particular hard to
start DevOps engineer as your first job. Many engineers believe that
it's not possible at all to begin as a DevOps professional without first
working as a developer or in operations (I totally disagree with
that!).
</p>
<p>
Almost every day, I see people struggling on their way to start as
fresh/junior DevOps engineers. They usually follow some roadmap
(typically <a target="_blank" href="https://roadmap.sh/devops">roadmap.sh/devops</a>).
But still, they cannot land their first job, and sadly, many of them
eventually give up!
</p>
<p>
This blog post explains why most of roadmaps don't work for DevOps
roles and won't help you start your first job as a DevOps engineer.
Also, the post discusses the best way to start in a DevOps role without
prior work experience. While it might not work for everyone, I can say
that it has been successful with all the people I have mentored in the
last couple of years.
</p>
<hr />
<p>
<strong>ToC</strong>
</p>
<ul>
<li><a href="#devops-topologies">DevOps Topologies</a></li>
<li><a href="#t-shaped-skills">T-Shaped Skills</a></li>
<li><a href="#mvp-learning-roadmap">MVP Learning Roadmap</a></li>
<li><a href="#the-solution">The Solution!</a></li>
</ul>
<p>
<strong>TL;DR</strong>
</p>
<p>
In 2023 starting a DevOps engineer role is challenging because the
DevOps model has various implementations and patterns. It's even more
complicated (but still possible) if that's your first job without
previous software industry experience. Yet, many learning roadmaps like
<a target="_blank" href="https://roadmap.sh/devops">roadmap.sh/devops</a> still follow a
linear path (i.e., learn some topic to the end, then move to another,
and so on) which doesn't work well with the DevOps role because a
skilled DevOps engineer is a T-Shaped skilled. Adopting a dynamic MVP
learning roadmap increases your chances of entering the market and
starting your first job as a DevOps engineer without previous software
hands-on experience.
</p>
<hr />
<h2 id="devops-topologies">DevOps Topologies</h2>
<p>
First, let's start with the <tt>DevOps</tt> model itself. Being a
high-level methodology that can be implemented in various ways makes it
super challenging. Hence, the DevOps engineer role has no unified
definition or standard requirements.
</p>
<p>
I will not delve into the clichΓ© <tt>DevOps is not a role, it is a
culture</tt> (because in reality, it doesn't work like that! <a
href="https://tech.aabouzaid.com/2020/05/devops-is-not-just-a-culture-and-it-is-also-a-role.html">DevOps
is not just a culture, and it is also a role</a>), but here I want to
emphasize that not all DevOps engineers are the same!
</p>
<p>
Here I want to mention "<a
href="https://web.devopstopologies.com/">DevOps Topologies</a>" which
covers different team structures that implement DevOps. It shows many
bad and good DevOps patterns.
</p>
<p>
Given that no one-size-fits-all team topology works for every
organization, it's wrong to say it's not possible to start your career
as a DevOps engineer. But there are situations where it's extremely
challenging to do so, such as being the sole DevOps engineer in a team
or even in a company.
</p>
<p>
So, what can you do to increase your chances of landing your first
job as a DevOps engineer? You should have T-shaped skills and leave no
stone unturned in your learning journey!
</p>
<h2 id="t-shaped-skills">T-Shaped Skills</h2>
<div class="separator" style="clear: right; float: right; margin-bottom: 1em; margin-left: 2em; text-align: center; width: 35%">
<a style="display: block; padding: 1em 0; text-align: center;" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh205lc5EklBl5TeLyGSqBQHoKvBWM7RSqnAyZezoPisv1CrewmTIkzcrgRn1yK-dbVSkjAD5q94T6pzBh4xvhAEhTAaSKWb_ESa9kAlZmhCXmah9yZs6LV1Gm4ULZrHoKq6DgPXEF5u8QEMOfkuAonxOhe_B9CYFDybH2n1RHwFyr6czhUs_yrKYUOeOF2/s1128/t-shaped-skills.png">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh205lc5EklBl5TeLyGSqBQHoKvBWM7RSqnAyZezoPisv1CrewmTIkzcrgRn1yK-dbVSkjAD5q94T6pzBh4xvhAEhTAaSKWb_ESa9kAlZmhCXmah9yZs6LV1Gm4ULZrHoKq6DgPXEF5u8QEMOfkuAonxOhe_B9CYFDybH2n1RHwFyr6czhUs_yrKYUOeOF2/s400/t-shaped-skills.png" />
</a>
The "T-Shaped skills" helps DevOps engineers to efficivtly handle various challanges.
</div>
<p>
The <tt>T-shaped skills</tt> refer to combining broad and deep
skills in a specific field. The horizontal bar of the "T" represents a
broad range of general knowledge and skills across different disciplines
or areas, and the vertical stem of the "T" represents deep expertise in
a specific area. It's simply a mix between being a
<tt>specialist</tt> and <tt>generalist</tt> at the same
time!
</p>
<p>
The T-shaped skills will help you to work in companies with different
DevOps patterns. You can easily transition between different areas in
the DevOps spectrum. Not only that but also it will help you to handle
new challenges effectively. In fact, the best DevOps engineers I have
come across possessed T-shaped skills.
</p>
<p>
Does it mean there's no <tt>I-shaped</tt> DevOps engineer who
specializes in certain areas and no little knowledge in other areas? I
would say it's possible, but it may limit the available opportunities
and companies you can work with.
</p>
<p>
Actually, as you progress in your career, it's better to develop more
specialization (i.e., more vertical stems), and after a couple of years
in the industry, your next step should be <tt>Pi-Shaped</tt> skills
(search also for <tt>M-Shaped</tt> and <tt>Comb-Shaped</tt>
skills, but it's a topic for another post).
</p>
<p>
To summarize this section, you should aim to gain exposure to various
areas of DevOps practices and technologies without delving too deep into
each one, yet, you need to dive in-depth into some of them (according to
the market or organizations your target). And due to that, your roadmap
shouldn't be linear but follow the MVP-style approach!
</p>
<h2 id="mvp-learning-roadmap">MVP Learning Roadmap</h2>
<p>
You probably heard before about the "Minimum Viable Product" or MVP,
a basic version of a product with enough features to satisfy early users
and gather feedback for further development, which's commonly associated
with Agile methodologies. Interestingly, this approach can also be
applied to learning roadmaps too!
</p>
<p>
Over the years, I saw people want to start their career as DevOps but
completely lost! That's probably because they try to progress in a
linear fashion. Typically they will follow some roadmap like <a
href="https://roadmap.sh/devops">roadmap.sh/devops</a> and start
learning the topics one by one top down. I.e., they spend a couple of
weeks with Linux, then a month learning programming language, then some
time with the containers, and more time with Kubernetes. And several
months passed, and they found themselves stuck in the middle of the
roadmap but still unable to get any job because many topics were
untouched!
</p>
<p>
The truth is that the linear vertical learning path <strong>DOES NOT
WORK</strong> in the DevOps field! You need to follow an MVP-style
learning roadmap where you learn in iterations and touch multiple parts
at the same time (not necessarily equally). Each iteration should focus
on a primary topic while exploring related side topics. So after a
month, you already know about each topic in the roadmap, and after the
2nd month, you have the basics, and after the 3rd month, you have a good
base, and so on.
</p>
<p>
By adopting an <strong>MVP-style learning roadmap</strong>, you can
ensure that you cover various aspects of DevOps while continuously
building upon your knowledge. This approach allows for a more
well-rounded understanding and a better chance of landing your first job
as a DevOps engineer.
</p>
<div class="separator" style="clear: both; text-align: center;">
<a target="_blank" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjpCyvN1jCJ0pB1YfoKXv1_PL8n0ZHjUGp1_P5_q_Z6TcdW-wLv4zfjUkL8_8huGcp6hMN2e_bQYSTc3WclTQyQ0Swq096qjltjJgtzg6nf_4dwRuid_vPnrfAd59D6LnkPViN-u_P33LCBJD1k5NZYNgA139QdPrKs--kt_8fBx_KAbFW7Bu5XC-g_Hwi3/s2492/dynamic-mvp-roadmap-triple.png" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" width="600" data-original-height="1073" data-original-width="2492" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjpCyvN1jCJ0pB1YfoKXv1_PL8n0ZHjUGp1_P5_q_Z6TcdW-wLv4zfjUkL8_8huGcp6hMN2e_bQYSTc3WclTQyQ0Swq096qjltjJgtzg6nf_4dwRuid_vPnrfAd59D6LnkPViN-u_P33LCBJD1k5NZYNgA139QdPrKs--kt_8fBx_KAbFW7Bu5XC-g_Hwi3/s600/dynamic-mvp-roadmap-triple.png"/>
</a>
3 different high-level roadmaps models with different approaches to learning, the first two from the left follow the MVP-style, and the last one is linear.
<br/>
<br/>
</div>
<ul>
<li>
<p>
<strong>The model on the left iterates horizontally in equal
chunks over each area (good).</strong> It's simple and straightforward,
each area (e.g., OS and code, containers and cloud, etc) has a fixed
weight based on importance in the daily work. You don't need to think a
lot about the next step, from the left to the right, you learn about
each area and reach basic knowledge in all of them.
</p>
</li>
<li>
<p>
<strong>The model in the center iterates horizontally in dynamic
chunks over each area (better).</strong> It's the same as the previous
one but a bit more advanced, where it needs hands-on knowledge to decide
the right weight for each area (based on many factors like targeted
market or learner skills or background). This approach is more
efficient; however, it requires guidance from an experienced DevOps
engineer (e.g., a mentor or career coach) to define the weights
correctly. It's even more critical when you have some sort of
constraints like time or so (they are usually there in career
shifts).
</p>
</li>
<li>
<p>
<strong>The model on the right iterates vertically over each area
(bad).</strong> Don't do that! It has several drawbacks. For example, it
delays your market fit, where in most cases, you cannot work as a DevOps
engineer until you have completed all areas. Additionally, there is no
space to review your learning approaches or holistic feedback in
general. Finally, it's missing the connection between different areas,
at the end of the day at work, you don't use a single skill at a time. I
actually saw many disappointed people in the middle of the roadmap
because they still didn't get the full picture.
</p>
</li>
</ul>
<p>
So to ensure a more effective learning journey, it's recommended to
adopt an MVP-style learning roadmap that allows for iterative learning
across multiple areas while also considering the relevance and
importance of each area in real-world DevOps work.
</p>
<h2 id="the-solution">The Solution!</h2>
<p align="center">
<a href="https://github.com/DevOpsHiveHQ/dynamic-devops-roadmap" imageanchor="1">
β Check out the Dynamic Roadmap content β
</a>
</p>
<div class="separator" style="clear: right; float: right; margin-bottom: 1em; margin-left: 2em; text-align: center; width: 40%">
<a style="display: block; padding: 1em 0; text-align: center;" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiCvgAX4wWnBJFZkMM7ODfrWHeNncycuibuK-gcqBq_OlbNSqDxZWXYC_-fnHNwsn1xxD4OVvm2RJuDg9eMdnlHIMRpUHHhqEqo_9KPRxWrRiuFNTixTP_j2n_gyUj1F9kU3kyne8wQC6veN83s4IKrWnbcbTPCU6GjM1n1sGdGqoM5fEb3aF2d-J6tUHrT/s1600/dynamic-mvp-roadmap-single.png">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiCvgAX4wWnBJFZkMM7ODfrWHeNncycuibuK-gcqBq_OlbNSqDxZWXYC_-fnHNwsn1xxD4OVvm2RJuDg9eMdnlHIMRpUHHhqEqo_9KPRxWrRiuFNTixTP_j2n_gyUj1F9kU3kyne8wQC6veN83s4IKrWnbcbTPCU6GjM1n1sGdGqoM5fEb3aF2d-J6tUHrT/s400/dynamic-mvp-roadmap-single.png" />
</a>
A dynamic MVP-style learning roadmap is one of the best ways to start as a DevOps engineer.
</div>
<p>
Let's put everything together. The Based on my experience mentoring
people in different stages (starting their first job, career shift,
moving to another work style or company), I have found that the approach
of using a dynamic MVP-style roadmap with hands-on projects designed by
an experienced DevOps engineer has been highly successful. That means
each project will cover all DevOps areas used in the job. It's also
essential to understand the targeted market and organizations because,
with different DevOps topologies, the DevOps engineer role requirements
vary.
</p>
<p>
In conclusion, to start working DevOps engineer, you don't need to
know "everything" about the software development life cycle (SDLC), nor
start as Dev or Ops and then switch to DevOps. In many DevOps
topologies, you can secure your first job as a DevOps engineer if you
invest enough time in learning (not only the technical aspects) and
follow an MVP-style roadmap. Also, undoubtedly having a senior DevOps on
both ends (during the learning and in the company where you apply) will
make your start much easier.
</p>
Ahmed AbouZaidhttp://www.blogger.com/profile/06013456198042893076noreply@blogger.com0tag:blogger.com,1999:blog-409120552279009989.post-21539353781647901892023-05-05T01:30:00.007+03:002023-05-14T12:56:10.099+03:00KubeconformValidator, my first Kustomize validator plugin - Golang<div class="separator" style="clear: both;">
<a target="_blank" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5dMeUEQL2Z51s94oLF5f_EgyhFCegbunx8RjEL6VNuytfPonrdt7Nsiapjf2BCbJRK3LgDYwKsyigFN0Qn9FOJsy0qYR_FxZEwDzkuO2H9f437tLVQXgrXrutAnlkuszMfG-aSGtHuhmgUwpdjOzoFHT_sNXwSdSTdqIi2g1eHoFj8MSoh9oEYGii4w/s600/kustomize.png" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" width="600" data-original-height="165" data-original-width="600" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5dMeUEQL2Z51s94oLF5f_EgyhFCegbunx8RjEL6VNuytfPonrdt7Nsiapjf2BCbJRK3LgDYwKsyigFN0Qn9FOJsy0qYR_FxZEwDzkuO2H9f437tLVQXgrXrutAnlkuszMfG-aSGtHuhmgUwpdjOzoFHT_sNXwSdSTdqIi2g1eHoFj8MSoh9oEYGii4w/s600/kustomize.png"/>
</a>
</div>
<h2 id="tldr">TL;DR</h2>
<p>
In the past, Kustomize suggested using <tt>transformers</tt> to
validate resources, but later, it introduced <tt>validators</tt>,
which are like transformers but read-only.
</p>
<p>
Say Hi to <a
href="https://github.com/aabouzaid/kustomize-kubeconformvalidator/blob/main/main.go">KubeconformValidator</a>,
a plugin built around <tt>Kubeconform</tt> to validate manifests
schema within Kustonize π
</p>
<h2 id="details">Details</h2>
<ul>
<li><a target="_blank" href="https://github.com/instrumenta/kubeval">Kubeval</a> is not
maintained anymore, and their repo suggests using <a
href="https://github.com/yannh/kubeconform">Kubeconform</a> as a
replacement.</li>
<li>The <a
href="https://github.com/kubernetes-sigs/kustomize/blob/master/cmd/config/docs/api-conventions/functions-spec.md">KRM</a>
model is used in the plugin, so no more Kustomize legacy plugins.</li>
<li>During the weekend, I did it in 5 short iterations (in hours) to
discover different options and structures.</li>
<li>In the past, I used <a
href="https://pkg.go.dev/github.com/GoogleContainerTools/kpt-functions-sdk/go/fn">kpt-functions-sdk/fn</a>
to work with KRM, but I decided to try Kustomize's <a
href="https://pkg.go.dev/sigs.k8s.io/kustomize/kyaml/fn/framework">kyaml/fn/framework</a>,
and it's great <span class="emoji"
data-emoji="heart_eyes">π</span></li>
<li>The <tt>kyaml/fn/framework</tt> saved a lot of work with KRM and
let met to just focus on the plugin logic. For example, I don't need to
deal with <a
href="https://github.com/aabouzaid/kustomize-kubeconformvalidator/blob/main/plugin-schema.yaml">OpenAPI
Schema</a> validation, it does it perfectly.</li>
</ul>
<h2 id="example">Example</h2>
<pre class="plain-pre">
apiVersion: validators.kustomize.aabouzaid.com/v1alpha1
kind: KubeconformValidator
metadata:
name: validate
annotations:
config.kubernetes.io/function: |
# Exec KRM functions.
exec:
path: ../dist/kubeconformvalidator
# # Containerized KRM functions.
# container:
# image: aabouzaid/kubeconformvalidator
# network: true
spec:
# Configure Kubeconform.
config:
output: json
skip:
- AlertmanagerConfig
# Also, direct Kubeconform args could be used
# but "spec.args" has lower priority over "spec.config".
# https://github.com/yannh/kubeconform#Usage
# args:
# - -output
# - json
# - -skip
# - AlertmanagerConfig
</pre>
<p>
That's it! Enjoy, and don't forget to take a look at <a
href="https://github.com/aabouzaid/awesome-kustomize/">awesome Kustomize
list</a>! :-)
</p>
Ahmed AbouZaidhttp://www.blogger.com/profile/06013456198042893076noreply@blogger.com0tag:blogger.com,1999:blog-409120552279009989.post-35945825618139860912023-01-01T01:30:00.001+02:002023-01-01T01:30:00.149+02:00Awesome Kustomize list - Kubernetes<div class="separator" style="clear: both;">
<a target="_blank" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj8BQFdSFJ522q0jegE1-yZrGwH2GmCaAS-hagx9kXnSXA6y9JEHVjt_rWst-_YdnxNTBV9o6pUmzjuCrFvUaPXEjnFoFvwqoYXx9PXxEoj7z0gpYU2opF61H1NYzssozAc3WlPX1WnrsbrLG9BYDHmj00YloHdhU-j5OGNzmFFxcq0o2SdZI48RJv1lA/s1080/awesome-kustomize.png" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" width="320" data-original-height="660" data-original-width="1080" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj8BQFdSFJ522q0jegE1-yZrGwH2GmCaAS-hagx9kXnSXA6y9JEHVjt_rWst-_YdnxNTBV9o6pUmzjuCrFvUaPXEjnFoFvwqoYXx9PXxEoj7z0gpYU2opF61H1NYzssozAc3WlPX1WnrsbrLG9BYDHmj00YloHdhU-j5OGNzmFFxcq0o2SdZI48RJv1lA/s320/awesome-kustomize.png"/>
</a>
</div>
<p>
After 3 years of using Kustomize intensively to manage Kubernetes
manifests, I just created a list of "<a
href="https://github.com/aabouzaid/awesome-kustomize/"><strong>Awesome
Kustomize</strong></a>" resources π€©οΈ
</p>
<p>
If you are using Kubernetes, this repo will be a pretty good starting
point for Kustomize plugins, guides, tips and tricks, and more.
</p>
<p>
Kustomize is not a new tool, it started in 2018, and now it's built
into "kubectl" (since v1.14), which means it's now considered the
official method to deal with advanced Kubernetes scenarios (e.g.,
GitOps).
</p>
<p>
If you are SysOps, DevOps, SRE, or anyone working intensively with
Kubernetes, you definitely should read more about Kustomize!
</p>
<p>
Also, if you just use "kubectl" or Helm, then consider using
Kustomize. It could help you in many different ways (it's not a
replacement for Helm, <a
href="https://tech.aabouzaid.com/2020/09/3-ways-to-customize-off-the-shelf-helm-charts-with-kustomize-kubernetes.html">you
can use both</a>).
</p>
<p>
Follow the repo to get the updates ποΈ
</p>
<p>
<a
href="https://github.com/aabouzaid/awesome-kustomize">https://github.com/aabouzaid/awesome-kustomize</a>
</p>
Ahmed AbouZaidhttp://www.blogger.com/profile/06013456198042893076noreply@blogger.com0tag:blogger.com,1999:blog-409120552279009989.post-60552700088957608282022-12-31T01:30:00.007+02:002023-01-10T15:47:17.643+02:002022 Highlights<div class="separator">
<div style="clear: right; float: right; margin-bottom: 1em; margin-left: 2em; text-align: center; width: 30%">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj32rCkENVH2Ix_t7uIuarU4z1N04tGVjAyUaPWQyInmRrfso4prREW75a-g7jA2i2kcpMunEadjR8NcRkdTZ-uoad2pS8IM1hB6irirb9vpjR3tMPzF1o5zEkXg4Es2kZ2jEIQKGm1E5Rv8_rEFZ-TH7bs1qKj7cufwBVBba--Rx9h8kbjb5Bs2nldQA/w200-h200/LFGLREqd.jpeg" />
<br/>
Just a random image generated with AI!
</div>
</div>
<p>
Finally, 2022 is over! What a crazy year! In many countries, the
Covid-19 pandemic is about to come to an end, but a global economic
recession is almost at the door!
</p>
<p>
On a personal level, it wasn't an easy year for sure, but it was good
in many different ways.
</p>
<h2 id="top-5-highlights-in-2022">Top 5 highlights in 2022</h2>
<ol type="1">
<li>
<p>
<strong>Career:</strong> Started the <tt>Distribution
team</tt> at <a target="_blank" href="https://camunda.com/">Camunda</a> π€©οΈ which is
responsible for building and deploying the <a
href="https://camunda.com/platform/">Camunda Platform 8</a> Self-Managed
(now using an umbrella [Helm <a
href="https://github.com/camunda/camunda-platform-helm">chart</a>).
Later on, there will be a Kubernetes Operator. That's a great career
boost; I just started with many new and exciting challenges. And BTW, my
team will begin hiring in 2023!
</p>
</li>
<li>
<p>
<strong>Coding for Kubernetes:</strong> Big refactoring for <a
href="https://tech.aabouzaid.com/2022/03/refactoring-bank-vaults-operator-for-full-vault-management-support.html">Bank-Vaults
operator</a> which is the biggest open-source contribution to a project
I don't own/manage. It polished my Golang skills, and I learned many new
things (and had fun where I redesigned the operator logo ποΈ).
</p>
</li>
<li>
<p>
<strong>Security Knowledge-sharing:</strong> In 2021, I got my <a
href="https://tech.aabouzaid.com/2021/11/now-i-am-a-certified-kubernetes-security-specialist-plus-exam-tips.html">CKS
certificate</a>. Then, at the beginning of 2022, I started a security
initiative at Camunda to enhance security practices. Then, later on, I
conducted a session about <a
href="https://tech.aabouzaid.com/2022/07/kubernetes-security-best-practices-with-tips-for-the-cks-exam.html">Kubernetes
Security Best Practices</a> (with some tips for the CKS exam) which was
a great case that includes theory, applied practice, and
knowledge-sharing!
</p>
</li>
<li>
<p>
<strong>Advanced CI/CD Knowledge-sharing:</strong> I wrote a
detailed post about my experience with custom step <a
href="https://tech.aabouzaid.com/2022/02/extending-jenkins-to-run-resilient-pipelines-with-long-running-jobs.html">conditionalRetry</a>,
which handles failures on spot/preemptible infrastructure so you could
save up to 90% of the costs and have stable builds as well! It's been
released as <strong>open source</strong>, and you can <a
href="https://github.com/camunda-community-hub/camunda-jenkins-shared-library/blob/main/docs/conditionalRetry.md">use
that in your pipeline</a>!
</p>
</li>
<li>
<p>
<strong>Activities:</strong> Helped more people in their careers,
once by moderating the DevOps circle at [JobStack <a
href="https://tech.aabouzaid.com/2022/06/moderating-devops-circle-at-jobstack-2022.html">2022</a>,
and also in the voluntary mentorship that I do from time to
time.
</p>
</li>
</ol>
<p>
Besides these highlights, I had some nice stuff during the year. For
example:
</p>
<ul>
<li>Added more features to <a
href="https://github.com/aabouzaid/kubech">kubech</a> (which is a tool
to set kubectl context/namespace per shell/terminal )</li>
<li>Virtually attended KubeCon Europe 2022, and the content was
great!</li>
<li>Reached my writing goal this year and wrote <a
href="https://tech.aabouzaid.com/2022/">12 blog posts in 2022</a>!</li>
</ul>
<p>
And since we are on this topic, here are the top 5 visited blog posts
in 2022!
</p>
<h2 id="top-5-posts-in-2022">Top 5 posts in 2022</h2>
<ol type="1">
<li>
<p>
<a
href="https://tech.aabouzaid.com/2021/05/delete-a-manifest-from-kustomize-base.html">Delete
a manifest from Kustomize base - Kubernetes</a>
</p>
</li>
<li>
<p>
<a
href="https://tech.aabouzaid.com/2020/09/3-ways-to-customize-off-the-shelf-helm-charts-with-kustomize-kubernetes.html">3
ways to customize off-the-shelf Helm charts with Kustomize -
Kubernetes</a>
</p>
</li>
<li>
<p>
<a
href="https://tech.aabouzaid.com/2020/04/validate-format-lint-and-test-terraform-iac-ci.html">Validate,
format, lint, secure, and test Terraform IaC - CI/CD</a>
</p>
</li>
<li>
<p>
<a
href="https://tech.aabouzaid.com/2020/06/now-i-am-a-certified-kubernetes-application-developer-plus-10-exam-tips.html">Now
I'm a Certified Kubernetes Application Developer + 10 exam
tips</a>
</p>
</li>
<li>
<p>
<a
href="https://tech.aabouzaid.com/2016/01/continuous-delivery-and-maturity-model.html">Continuous
Delivery and Maturity Model - DevOps</a>
</p>
</li>
</ol>
<p>
No wonder that Kustomize post is the hights post; that's because
there is not much content about it even though it's built-in kubectl
now! (since v1.14), I probably need to give it more attention since
there is an increase in the demand for it.
</p>
<p>
For that reason, I just started <a
href="https://github.com/aabouzaid/awesome-kustomize"><strong>Awesome
Kustomize</strong></a>, which is a curated and collaborative list of
awesome Kustomize resources ποΈ
</p>
<hr />
<p>
Enjoy ποΈ
</p>
Ahmed AbouZaidhttp://www.blogger.com/profile/06013456198042893076noreply@blogger.com0tag:blogger.com,1999:blog-409120552279009989.post-86621533187018133452022-12-12T01:30:00.002+02:002022-12-17T15:02:30.143+02:00Pass extra data to the Containerized KRM function - Kustomize<div class="separator" style="clear: both;"><a target="_blank" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5dMeUEQL2Z51s94oLF5f_EgyhFCegbunx8RjEL6VNuytfPonrdt7Nsiapjf2BCbJRK3LgDYwKsyigFN0Qn9FOJsy0qYR_FxZEwDzkuO2H9f437tLVQXgrXrutAnlkuszMfG-aSGtHuhmgUwpdjOzoFHT_sNXwSdSTdqIi2g1eHoFj8MSoh9oEYGii4w/s600/kustomize.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="600" data-original-height="165" data-original-width="600" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5dMeUEQL2Z51s94oLF5f_EgyhFCegbunx8RjEL6VNuytfPonrdt7Nsiapjf2BCbJRK3LgDYwKsyigFN0Qn9FOJsy0qYR_FxZEwDzkuO2H9f437tLVQXgrXrutAnlkuszMfG-aSGtHuhmgUwpdjOzoFHT_sNXwSdSTdqIi2g1eHoFj8MSoh9oEYGii4w/s600/kustomize.png"/></a></div>
<p>
A couple of months ago, I wrote some <a
href="https://tech.aabouzaid.com/2022/07/notes-about-krm-functions-kustomize.html">notes
about KRM Functions in Kustomize</a>. Today in this blog post, I show
why I <strong>love</strong> and <strong>hate</strong> the current state
of KRM functions ποΈ
</p>
<a name='more'></a>
<h2 id="toc">ToC</h2>
<ul>
<li><a href="#overview">Overview</a></li>
<li><a href="#prerequisites">Prerequisites</a></li>
<li><a href="#examples">Examples</a>
<ul>
<li><a href="#level-1-pass-no-data">Level 1: Pass no data</a></li>
<li><a href="#level-2-pass-simple-data-types">Level 2: Pass simple data
types</a></li>
<li><a href="#level-3-pass-complex-data-types">Level 3: Pass complex
data types</a></li>
</ul></li>
<li><a href="#conclusion">Conclusion</a></li>
</ul>
<h2 id="overview">Overview</h2>
<p>
I admired the idea of <a
href="https://kubectl.docs.kubernetes.io/guides/extending_kustomize/containerized_krm_functions/">Containerized
KRM function</a> that's because you don't need to deal with managing and
downloading the Kustomize plugins. It was super annoying to manage
plugins across multiple operating systems (e.g., for different team
members).
</p>
<p>
As mentioned before, the KRM manifest is passed to the container
directly as STDIN, but if you want to pass extra files, you need to
mount the files into the KRM function container, which is fine. However,
there are some limitations, like you can only mount files relative to
the base instead of overlay!
</p>
<p>
Let's take a look at different cases working with containerized KRM
functions.
</p>
<h2 id="prerequisites">Prerequisites</h2>
<p>
This post covers the above intermediate topic in Kustomize, so it
assumes that you understand how Kustomize works and the structure of <a
href="https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/">Kustomization
File</a>.
</p>
<h2 id="examples">Examples</h2>
<h3 id="level-1-pass-no-data">Level 1: Pass no data</h3>
<p>
In this case, you don't actually need to pass any "extra" data to the
containerized KRM function. All data required by the function are in the
manifest files.
</p>
<p>
Here is a dummy example since it's not the main topic here (I just
copied it from the <a
href="https://kubectl.docs.kubernetes.io/guides/extending_kustomize/containerized_krm_functions/">Kustomize
docs</a>):
</p>
<pre class="plain-pre">
apiVersion: someteam.example.com/v1
kind: ChartInflator
metadata:
name: notImportantHere
annotations:
config.kubernetes.io/function: |
container:
image: example.docker.com/my-functions/chart-inflator:0.1.6
spec:
chartName: minecraft
</pre>
<p>
<strong>Note:</strong> If you want to use Helm charts within
Kustomize files, Kustomize support that out of the box now via <a
href="https://tech.aabouzaid.com/2020/09/3-ways-to-customize-off-the-shelf-helm-charts-with-kustomize-kubernetes.html#method-1-kustomize-plugins">HelmChartInflationGenerator,
which is a built-in plugin</a>.
</p>
<h3 id="level-2-pass-simple-data-types">
Level 2: Pass simple data
types</h3>
<p>
Now let's look at a real example where you need to pass extra files
to the containerized KRM function.
</p>
<p>
Here is an example of the <a
href="https://github.com/kubernetes-sigs/kustomize/tree/91a2c2b/functions/examples/validator-kubeval">Kubeval
plugin</a>:
</p>
<pre class="plain-pre">
apiVersion: examples.config.kubernetes.io/v1beta1
kind: Kubeval
metadata:
name: my-validator
annotations:
config.kubernetes.io/function: |
container:
image: validator-kubeval:latest
mounts:
- type: bind
src: ./schemas
dst: /my_schemas
spec:
strict: true
schemaLocation: "file:///my_schemas"
</pre>
<p>
In this case, it's straightforward, it will mount the local dir
<tt>./schemas</tt> on container dir <tt>/my_schemas</tt>, and
the plugin inside the container will use it.
</p>
<p>
However, there is a limitation here, the mount paths must be under
the current kustomization directory, and it's not permitted to use a
<tt>src</tt> from the parent directory, which is usually used when
you have some common values shared across multiple Kustomizations.
</p>
<p>
So if you try to use something like this <tt>src:
../schemas</tt>, you will see an error like this:
</p>
<blockquote>
Error: plugin Kubeval.v1beta1.examples.config.kubernetes.io/validate.[noNs] with mount path '../my_schemas' is not permitted; mount paths must be under the current kustomization directory
</blockquote>
<h3 id="level-3-pass-complex-data-types">
Level 3: Pass complex data
types</h3>
<p>
If you want to pass more extra data it will be a bit complex. To be
more accurate, the problem here is not more or complex data, but the
plugin itself should work a bit differently if more data types are
needed for the plugin to work correctly.
</p>
<p>
Let's take <a
href="https://github.com/goabout/kustomize-sopssecretgenerator">SopsSecretGenerator
plugin</a> as an example, which generates Kubernetes
<tt>Secrets</tt> from sops-encrypted files. I like this plugin
because you don't need to download <a
href="https://github.com/mozilla/sops">SOPS</a> binary; the plugin
includes SOPS libs as part of it.
</p>
<p>
The <tt>SopsSecretGenerator</tt> plugin needs different files to
work, but mainly it needs a GPG key and sops-encrypted files. The plugin
itself supports KRM (I've added it in the <a
href="https://github.com/goabout/kustomize-sopssecretgenerator/pull/32">PR
#32</a>). However, its container doesn't support running containerized
KRM functions because there is no mechanism to import the GPG key into
the container keyring.
</p>
<p>
Of course, that should be part of the upstream, but given that KRM
support is still super young and not many plugins support KRM in the
first place. But here, an example shows how much work is needed to make
a plugin works as a containerized KRM function.
</p>
<p>
As I mentioned, the current container image of
<tt>SopsSecretGenerator</tt> plugin doesn't support importing the
GPG key, so we need to build a new image that's able to import the GPG
key.
</p>
<pre class="plain-pre">
# syntax=docker/dockerfile:1
ARG SSG_UPSTREAM_VERSION=v1.6
FROM goabout/kustomize-sopssecretgenerator:$SSG_UPSTREAM_VERSION as ssg
FROM alpine
RUN apk add --no-cache gnupg
COPY --from=ssg /SopsSecretGenerator /SopsSecretGenerator
ENV GPG_KEYS_DIR=/mnt/gpg/keys
COPY <<-EOT /docker-entrypoint.sh
# Create a writable new home dir because the KRM container works as "nobody" user with home "/".
mkdir -p /tmp/user
export HOME=/tmp/user
gpg --quiet --import $GPG_KEYS_DIR/*
/SopsSecretGenerator $@
EOT
ENTRYPOINT ["sh", "/docker-entrypoint.sh"]
</pre>
<p>
Now let's build it using Docker BuildKit:
</p>
<pre class="plain-pre">
DOCKER_BUILDKIT=1 docker build -t sopssecretgenerator-custom .
</pre>
<p>
As mentioned before, you cannot use bind type to mount files outside
the kustomization directory. So the only option here is to copy them to
a named volume and reference the volume name. To do so, and since Docker
doesn't provide a way to copy data to volume directly, I spin a
temporary container and use it to copy the GPG key to it.
</p>
<pre class="plain-pre">
docker run --rm \
-v <PATH_TO_LOCAL_GPG_KYES_DIR>:/mnt \
-v my-gpg-key-volume:/data busybox \
cp -r /mnt /data
</pre>
<p>
And here is the final manifest with the local volume for the GPG file
and bind of current sops-encrypted secrets:
</p>
<pre class="plain-pre">
apiVersion: goabout.com/v1beta1
kind: SopsSecretGenerator
metadata:
name: my-secret
annotations:
config.kubernetes.io/function: |
container:
image: sopssecretgenerator-custom
mounts:
- type: volume
src: my-gpg-key-volume
dst: /mnt/gpg
- type: bind
src: ./sops-encrypted-secrets
dst: /mnt/sops-encrypted-secrets
envs:
- /mnt/sops-encrypted-secrets/my-secret.env
</pre>
<h2 id="conclusion">Conclusion</h2>
<p>
As you see, containerized KRM functions provide a super flexible way
to extend Kustomize. However, particular limitations still make it hard
to use in some cases. So, in addition to containerized KRM functions, it
is inevitable also to use <a
href="https://kubectl.docs.kubernetes.io/guides/extending_kustomize/exec_krm_functions/">Exec
KRM functions</a>
</p>
<p>
Given that KRM functions in Kustomize are still <tt>alpha</tt>, I
believe many things will improve over time, and I can't wait to see it
stable!
</p>
<p>
---
</p>
<p>
That's it! Enjoy :-)
</p>
Ahmed AbouZaidhttp://www.blogger.com/profile/06013456198042893076noreply@blogger.com0tag:blogger.com,1999:blog-409120552279009989.post-32884710300604457062022-11-22T01:30:00.010+02:002022-11-29T23:26:18.488+02:00Set OpenAPI patch strategy for Kubernetes Custom Resources - Kustomize<p>
Kustomize supports 2 main client-side patching methods for Kubernetes
manifests, <a
href="https://github.com/kubernetes-sigs/kustomize/blob/master/examples/jsonpatch.md">JSON
Patching</a> and <a
href="https://github.com/kubernetes/community/blob/master/contributors/devel/sig-api-machinery/strategic-merge-patch.md">Strategic
Merge Patch</a>. In the <tt>JSON Patching</tt> method, you have a
"meta" syntax that specifies operation/target/value. In the
<tt>Strategic Merge Patch</tt> method, you can override values by
providing a patch file with the same structure but with new values, and
it will override the original values (it simply merges the 2 files with
the same structure).
</p>
<p>
Each method has pros and cons but generally speaking, I would
arguably say that <tt>Strategic Merge Patch</tt> is better for big
changes/patches, and <tt>JSON Patching</tt> is better for smaller
fine-grained patches. And for my use case, I will use <tt>Strategic
Merge Patch</tt>, but I just faced a problem with patching Kubernetes
Custom Resources!
</p>
<div class="separator" style="clear: both;">
<a target="_blank" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgc-O-7LIRjYVRFW5_sIPSXaWTbqtMkzowqStXPQMcVH6tu3I_em9zhBH4QIGnrJtSq2vEWPNzXjEP5a1YnswifbWu7xZLRTybzf4neNtgV-2iom85mfs3_WgGxy2xfuk29mideUifm7zI7zjCVKmxtJ5UTFP0xPHvjmpu9G0wWcqQS7haCERHdDJoNrA/s958/kustomize-patch-strategy-for-k8s-custom-resources.png" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" width="600" data-original-height="562" data-original-width="958" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgc-O-7LIRjYVRFW5_sIPSXaWTbqtMkzowqStXPQMcVH6tu3I_em9zhBH4QIGnrJtSq2vEWPNzXjEP5a1YnswifbWu7xZLRTybzf4neNtgV-2iom85mfs3_WgGxy2xfuk29mideUifm7zI7zjCVKmxtJ5UTFP0xPHvjmpu9G0wWcqQS7haCERHdDJoNrA/s600/kustomize-patch-strategy-for-k8s-custom-resources.png"/>
</a>
</div>
<h2 id="toc">ToC</h2>
<ul>
<li><a href="#tldr">TL;DR</a></li>
<li><a href="#1-task">1. Task</a></li>
<li><a href="#2-issue-reproduction">2. Issue reproduction</a></li>
<li><a href="#3-background">3. Background</a></li>
<li><a href="#4-solution">4. Solution</a>
<ul>
<li><a href="#41-get-the-custom-resource-openapi-schema">4.1 Get the
custom resource OpenAPI schema</a></li>
<li><a href="#42-find-the-desired-key-path">4.2 Find the desired key
path</a></li>
<li><a href="#43-create-the-custom-openapi-schema-file">4.3 Create the
custom OpenAPI schema file</a></li>
<li><a
href="#44-update-kustomizationyaml-with-the-openapi-schema-file">4.4
Update kustomization.yaml with the OpenAPI schema file</a></li>
</ul></li>
<li><a href="#conclusion">Conclusion</a></li>
</ul>
<h2 id="tldr">TL;DR</h2>
<p>
Kustomize's default patch strategy for the <tt>lists</tt>
(arrays) is <tt>replace</tt>. That means the patch list will
override the original list, which is not always the desired behavior.
That behavior could be changed only if an OpenAPI schema for a
Kubernetes resource is available to define the patch strategy.
</p>
<p>
The OpenAPI schema for Kustomize core resources (like Namespace,
Deployment, Pod, etc.) is already part of Kustomize, so changing the
patch strategy works out of the box for these resources. However, if you
have a Kubernetes Custom Resource, you need to provide to Kustomize the
OpenAPI schema of that custom resource. And that's only useful if the
custom resource includes the OpenAPI extensions related to merging
strategy.
</p>
<p>
This post shows how to add those extensions to have control over the
patch strategy. You can jump directly to the <a
href="#4-solution">solution section</a> if you already know all these
details.
</p>
<h2 id="1-task">1. Task</h2>
<p>
I want to use Kustomize to patch <a
href="https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/">Kubernetes
Custom Resources</a> like Prometheus <a
href="https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/user-guides/alerting.md">AlertmanagerConfig</a>),
and I want to use <tt>merge</tt> as a patch strategy for
<tt>lists</tt>. That means the original lists in the same path
should be merged, not overridden by the patch list. That works out of
the box for Kustomize core resources but not for custom resources.
First, let's see that in action, then dive into the explanation
afterward.
</p>
<h2 id="2-issue-reproduction">2. Issue reproduction</h2>
<p>
Let's have a look at this example using the core resource
<tt>Pod</tt>, given this Kustomization file:
</p>
<pre class="plain-pre">
# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- pod.yaml
patches:
- pod-patch01.yaml
- pod-patch02.yaml
</pre>
<p>
And these resources and patches files:
</p>
<pre class="plain-pre">
# pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: nginx
spec:
containers:
- name: nginx
env:
- name: MY_ENV_VAR_01
value: source
# pod-patch01.yaml
apiVersion: v1
kind: Pod
metadata:
name: nginx
spec:
containers:
- name: nginx
env:
- name: MY_ENV_VAR_01
value: patch 01
# pod-patch02.yaml
apiVersion: v1
kind: Pod
metadata:
name: nginx
spec:
containers:
- name: nginx
env:
- name: MY_ENV_VAR_02
value: patch 02
</pre>
<p>
The <tt>kustomize build .</tt> will return:
</p>
<pre class="plain-pre">
apiVersion: v1
kind: Pod
metadata:
name: nginx
spec:
containers:
- name: nginx
env:
- name: MY_ENV_VAR_02
value: patch 02
- name: MY_ENV_VAR_01
value: patch 01
</pre>
<p>
As you see, the env key <tt>MY_ENV_VAR_01</tt> overrode by the
value from <tt>pod-patch01.yaml</tt>, and the env key
<tt>MY_ENV_VAR_02</tt> has just been added from
<tt>pod-patch02.yaml</tt>. That's great; the lists are merged based
on the <tt>name</tt> key.
</p>
<p>
...
</p>
<p>
However, if you tried to do that with a <tt>CustomResource</tt>
like <tt>AlertmanagerConfig</tt>, it would not work! Let's give it a
try! Given this Kustomization file:
</p>
<pre class="plain-pre">
# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- alertmanagerconfig.yaml
patches:
- alertmanagerconfig-patch01.yaml
- alertmanagerconfig-patch02.yaml
</pre>
<p>
And these resources and patches files:
</p>
<pre class="plain-pre">
# alertmanagerconfig.yaml
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
name: example
spec:
receivers:
- name: 'webhook01'
webhookConfigs:
- url: 'http://example.com/'
# alertmanagerconfig-patch01.yaml
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
name: example
spec:
receivers:
- name: 'webhook01'
webhookConfigs:
- url: 'http://example01.com/'
# alertmanagerconfig-patch02.yaml
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
name: example
spec:
receivers:
- name: 'webhook02'
webhookConfigs:
- url: 'http://example02.com/'
</pre>
<p>
The <tt>kustomize build .</tt> will return:
</p>
<pre class="plain-pre">
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
name: example
spec:
receivers:
- name: webhook02
webhookConfigs:
- url: http://example02.com/
</pre>
<p>
As you see, the last patch from the file
<tt>alertmanagerconfig-patch02.yaml</tt> replaced everything in the
<tt>spec.receivers</tt> list, and that's the default behavior in
Kustomize. The patch list will replace everything in the original list.
<strong>Why?</strong> Because that's the safest choice since Kustomize
doesn't know anything about <tt>AlertmanagerConfig</tt> schema!
Before diving into the fix, let's learn more about the
<tt>why</tt>.
</p>
<h2 id="3-background">3. Background</h2>
<p>
The <tt>Strategic Merge Patch</tt> is a client-side merge method
that merges 2 or more Kubernetes manifests together based on the
manifest <tt>apiVersion</tt>, <tt>kind</tt>, and
<tt>metadata.name</tt>. To merge 2 YAML files, you need to decide
the "<strong>merge strategy</strong>" for different data types, i.e.,
what should happen for the "string", "int", "list", "map", and so on?
Should they merge together? Or do the patch values override the original
values?
</p>
<p>
Also, each data type could be patched differently; for example, how
to patch a <tt>list</tt>? Kustomize provides <a
href="https://github.com/kubernetes/community/blob/master/contributors/devel/sig-api-machinery/strategic-merge-patch.md#basic-patch-format">different
patch formats</a> like <tt>merge</tt>, <tt>replace</tt>, and
<tt>delete</tt>. In fact, in a previous post (<a
href="https://tech.aabouzaid.com/2021/05/delete-a-manifest-from-kustomize-base.html">Delete
a manifest from Kustomize base</a>), I mentioned the <tt>delete</tt>
patch strategy, which works out of the box with core Kubernetes
primitive (namespace, deployment, pod, etc.), but not the
<tt>CustomResources</tt>.
</p>
<p>
<strong>Why does it work with core resources only?</strong> Because
of 2 things.
</p>
<ol type="1">
<li>The Kubernetes project includes specific keys (extensions) in the
core resources OpenAPI schema to deal with that. Namely the OpenAPI
extensions <tt>x-kubernetes-patch-strategy</tt> and
<tt>x-kubernetes-patch-merge-key</tt> (see them in Kubernetes <a
href="https://raw.githubusercontent.com/kubernetes/kubernetes/master/api/openapi-spec/swagger.json">swagger.json</a>).</li>
<li>The OpenAPI schema for Kubernetes' core resources is embedded in
Kustomize.</li>
</ol>
<p>
If those keys are not included in the OpenAPI schema, and Kustomize
doesn't have access to the OpenAPI schema, the default behavior will be
applied, which in Kustomize, the patch list will <tt>fully
replace</tt> the original list.
</p>
<h2 id="4-solution">4. Solution</h2>
<p>
Now there are 2 cases, First, if the custom resource definition
already has the <tt>x-kubernetes-patch-</tt>*`, and second, if the
custom resource definition doesn't have them at all. The Kustomize
supports "<a
href="https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/openapi/">openapi</a>"
field, which specifies where Kustomize gets its OpenAPI schema.
</p>
<p>
For the first case, it's easy; you just need to point Kustomize to
the OpenAPI schema and that's it!
</p>
<pre class="plain-pre">
# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
openapi:
# It could be also a URL.
path: monitoring.coreos.com_v1alpha1_alertmanagerconfig.json
[...]
</pre>
<p>
However, for the second case, in a Platonic world, you should contact
the upstream to add the OpenAPI extension keys
<tt>x-kubernetes-patch-strategy</tt> and
<tt>x-kubernetes-patch-merge-key</tt>. But as you know, in reality,
that will take ages, and in the best-case scenario, it will not happen
overnight! So the pragmatic solution is to tell Kustomize how to deal
with that custom resource via OpenAPI schema.
</p>
<p>
We will simply get the Custom Resource's OpenAPI schema and add the
<tt>x-kubernetes-patch-*</tt> keys to it with the merge strategy,
which can also be customized using <a
href="https://github.com/kubernetes/community/blob/master/contributors/devel/sig-api-machinery/strategic-merge-patch.md#basic-patch-format">different
patch</a> formats like <tt>merge</tt>, <tt>replace</tt>, and
<tt>delete</tt>.
</p>
<p>
<strong>The following are the step to get the OpenAPI schema of a
custom resource, clean it, add the merge strategy keys, and finally use
it in kustomization.yaml file.</strong>
</p>
<h3 id="41-get-the-custom-resource-openapi-schema">
4.1 Get the custom
resource OpenAPI schema</h3>
<p>
You can get the OpenAPI schema for the resource from the upstream
project, or if you have already installed its
<tt>CustomResourceDefinition</tt>, then you can get it directly by
calling Kubernetes API. And since K8s API will return every definition
it has (probably thousands of lines), we will use <tt>jq</tt> to get
the exact custom resource OpenAPI schema.
</p>
<p>
Here is a snippet that will help to get the OpenAPI definition for a
particular resource:
</p>
<pre class="plain-pre">
get_openapi_definition () {
jq \
--arg group "${1}" \
--arg version "${2}" \
--arg kind "${3}" \
'.definitions | with_entries(select(.value."x-kubernetes-group-version-kind"[0] |
.group==$group and
.version==$version and
.kind==$kind
))'
}
</pre>
<p>
And we can get the schema for the exact resource from Kubernetes API
by running the following (remember, you should have installed the CRD
for that resource into your Kubernetes cluster to be able to do
that):
</p>
<pre class="plain-pre">
kustomize openapi fetch | get_openapi_definition "monitoring.coreos.com" "v1alpha1" "AlertmanagerConfig" > alertmanagerconfig_openapi_schema_map.json
</pre>
<h3 id="42-find-the-desired-key-path">4.2 Find the desired key path</h3>
<p>
Here is most of the manual work, but the good news is that you need
to do it once. We need 2 things, the data of the path
<tt>spec.receivers</tt> and the key
<tt>x-kubernetes-group-version-kind</tt>. Open the schema file and
remove everything not under the hierarchy of the path we want to
customize.
</p>
<p>
Here is what it looks like after removing everything unrelated:
</p>
<pre class="plain-pre">
# alertmanagerconfig_openapi_schema_map.json
{
"com.coreos.monitoring.v1alpha1.AlertmanagerConfig": {
"properties": {
"spec": {
"properties": {
"receivers": {
"type": "array"
}
},
"type": "object"
}
},
"type": "object",
"x-kubernetes-group-version-kind": [
{
"group": "monitoring.coreos.com",
"kind": "AlertmanagerConfig",
"version": "v1alpha1"
}
]
}
}
</pre>
<h3 id="43-create-the-custom-openapi-schema-file">
4.3 Create the custom
OpenAPI schema file</h3>
<p>
Now we just need to put everything together adding
<tt>x-kubernetes-patch-*</tt> keys and the <tt>definitions</tt>
parent. The final result will look like the following:
</p>
<pre class="plain-pre">
# monitoring.coreos.com_v1alpha1_alertmanagerconfig.json
{
"definitions": {
"com.coreos.monitoring.v1alpha1.AlertmanagerConfig": {
"properties": {
"spec": {
"properties": {
"receivers": {
"x-kubernetes-patch-merge-key": "name",
"x-kubernetes-patch-strategy": "merge",
"type": "array"
}
},
"type": "object"
}
},
"type": "object",
"x-kubernetes-group-version-kind": [
{
"group": "monitoring.coreos.com",
"kind": "AlertmanagerConfig",
"version": "v1alpha1"
}
]
}
}
}
</pre>
<h3 id="44-update-kustomizationyaml-with-the-openapi-schema-file">
4.4
Update kustomization.yaml with the OpenAPI schema file</h3>
<p>
The final step, we need to tell Kustomize about our custom OpenAPI
schema file as follows (it also accepts YAML files in case you like to
convert it):
</p>
<pre class="plain-pre">
# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
openapi:
path: monitoring.coreos.com_v1alpha1_alertmanagerconfig.json
resources:
- alertmanagerconfig.yaml
patches:
- alertmanagerconfig-patch01.yaml
- alertmanagerconfig-patch02.yaml
</pre>
<p>
Now the <tt>kustomize build .</tt> will return:
</p>
<pre class="plain-pre">
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
name: example
spec:
receivers:
- name: webhook02
webhookConfigs:
- url: http://example02.com/
- name: webhook01
webhookConfigs:
- url: http://example01.com/
</pre>
<p>
Great, it works as expected! ποΈ And the custom resource list is
merged based on the <tt>name</tt> key (you can choose any merge key
based on your use case).
</p>
<h2 id="conclusion">Conclusion</h2>
<p>
<a
href="https://kubectl.docs.kubernetes.io/references/kustomize/">Kustomize</a>
is super powerful and has many capabilities to manage your entire
Kubernetes infrastructure as code! And the most fantastic thing? It's
now part of <tt>kubectl</tt>, so it's almost the standard way to
deal with advanced Kubernetes manifest structure.
</p>
<p>
Enjoy :-)
</p>
Ahmed AbouZaidhttp://www.blogger.com/profile/06013456198042893076noreply@blogger.com0tag:blogger.com,1999:blog-409120552279009989.post-82573058195273749632022-11-11T01:30:00.018+02:002022-11-18T18:26:25.569+02:00Handling nullable nested values - Helm<div class="separator" style="clear: both; text-align: center;">
<img src="https://www.vectorlogo.zone/logos/helmsh/helmsh-ar21.svg" title="Helm" alt="Helm" width="240" height="120"/>
</div>
<p>
I'd say this is one of the best tricks I've learned about Helm
recently! For me, one of the most annoying things in Helm was that it's
hard to handle nullable nested values!
</p>
<p>
For example, if you want to get the value of a nested key like
<a name='more'></a><tt>.Values.foo.bar.baz</tt>, and if
<tt>.Values.foo.bar</tt> is optional, you need to make sure that
<tt>bar</tt> is not <tt>nil</tt> otherwise, Helm will throw an
error.
</p>
<p>
Recently I just learned that you can use this syntax to call nested
values even if their parent is <tt>nil</tt>!
</p>
<pre class="plain-pre">
{{ $bazValue := (((.Values.foo).bar).baz) }}</pre>
<p>
This way it doesn't matter if <tt>.Values.foo.bar</tt> is not
defined or not, and you don't need to check the whole chain just to get
the nested value.
</p>
<hr />
<p>
All credit goes to <a
href="https://stackoverflow.com/a/68807258/4547221">Torrey on Stack
Overflow</a>. Check the link for more details about it how that
works.
</p>
<p>
βHappy Helming!β
</p>
Ahmed AbouZaidhttp://www.blogger.com/profile/06013456198042893076noreply@blogger.com0tag:blogger.com,1999:blog-409120552279009989.post-13221622789587558932022-09-09T00:30:00.006+02:002022-09-21T01:46:01.375+02:00How to create Makefile targets with dynamic parameters and autocompletion - Make<p>
Make is one of the oldest build automation tools ever (the original
<tt>Make</tt> was created in 1976!). And since then, it got many
implementations as <tt>BSD make</tt>, <tt>GNU make</tt>, and
<tt>Microsoft nmake</tt>. It uses a declarative syntax, and
sometimes that's the best and worst thing about it!
</p>
<p>
You probably used Make at least once. It's widely adopted in the tech
industry, or as someone said, "Make is the second best tool to automate
anything!". ποΈ
</p>
<p>
After <a name='more'></a>this quick intro, in this post I show a trick that
I used for a long time to create <strong>Makefile targets with dynamic
parameters and autocompletion</strong>.
</p>
<p>
But what does that actually mean? Let's have a look!
</p>
<h2 id="makefile-target-with-a-simple-parameter">
Makefile target with a
simple parameter</h2>
<p>
The standard way in Make is to use variables as parameters. So you
need to pass the variable externally in the command line:
</p>
<pre class="plain-pre">
hello:
@echo "Hello $(NAME)!"</pre>
<p>
Then run it, and it will print the hello with the variable.
</p>
<pre class="plain-pre">
$ make hello NAME=Ahmed
Hello Ahmed!</pre>
<p>
It's working as expected, and it could print any <tt>name</tt>,
but what if you have a limit list of variables like this:
</p>
<pre class="plain-pre">
$ make build TARGET=prod/backend</pre>
<p>
In this case, it will be a bit annoying because this doesn't support
autocompletion which means:
</p>
<ol type="1">
<li>You need to know and write the whole parameter and the argument
every time.</li>
<li>You don't know what values are available to that parameter and could
easily misspell the target.</li>
</ol>
<p>
Is there a better way to do it? Yes, dynamic parameters!
</p>
<h2 id="makefile-target-with-a-dynamic-parameter">
Makefile target with a
dynamic parameter</h2>
<p>
Let's have this example, I have a small Kubernetes project with
multi-components/environments and I don't really want to define the
exact list of envs or components every time I add one. So I need a way
to create targets with dynamic parameters that should support
autocompletion.
</p>
<p>
Here how I want it:
</p>
<pre class="plain-pre">
# Build all prod components.
$ make build prod
# Or just building a single component from each env.
$ make build prod/backend
# It also works with autocompletion and it will list the available options.
$ make build prod/[tab tab]
backend database frontend</pre>
<p>
Here is my way to do it! This working example is mainly for
Kubernetes, but <strong>the same idea could be applied for anything
else</strong>.
</p>
<pre class="plain-pre">
# Makefile
# Silence any make output to avoid interfere with manifests apply.
MAKEFLAGS += -s
SHELL := bash
#
# Kustomize - Build.
KUSTOMIZE_PLUGIN_HOME := $$HOME/.config/kustomize/plugin
# Find directories with Kustomization file.
define kustomize_get_dirs
$(shell dirname $(shell grep -rl '^kind: Kustomization' $(1)))
endef
# Run "kustomize build" with certain options.
define kustomize_build
export KUSTOMIZE_PLUGIN_HOME=${KUSTOMIZE_PLUGIN_HOME}; \
kustomize build $(1) \
--enable-alpha-plugins \
--load-restrictor LoadRestrictionsNone
endef
# Find Kustomization files for each env.
COMMON := $(call kustomize_get_dirs, common)
PROD := $(call kustomize_get_dirs, prod)
STAGE := $(call kustomize_get_dirs, stage)
ALL_ENVS := \
$(COMMON) \
$(PROD) \
$(STAGE)
# This is just to load all build folders as dynamic make targets.
$(ALL_ENVS):
# Use Make targets as input to "build" target, but exclude "build" itself.
.PHONY build
build:
$(call kustomize_build, $(filter-out $@, $(MAKECMDGOALS)))</pre>
<p>
<strong>So what is the trick here?</strong>
</p>
<p>
The trick to creating dynamic parameters with autocompletion for
Makefile targets is to load the parameters as targets! Let's take a look
at the example mentioned above:
</p>
<pre class="plain-pre">
| This is a predefined Make target.
|
+---+
make build prod/backend
+----------+
|
| This is a dynamic (auto-generated) Make target
| and it's used as an input for the previous target "build".
| This target is defined/generated by "$(ALL_ENVS):" line.</pre>
<p>
<strong>The steps to apply the same notion to anything:</strong>
</p>
<ol type="1">
<li>Create a Make function (A) that creates any patterns you want to
apply.</li>
<li>Load the patterns from the first function dynamically using
<tt>$():</tt> as a target.</li>
<li>Create a Make function (B) that will use the input to do a specific
task.</li>
<li>Create a Make target that will use function (B) to execute any other
targets as input.</li>
</ol>
<p>
---
</p>
<p>
That was one of many <tt>Make</tt> capabilities that make your
life easier, and that's why I love <tt>Make</tt>, where it always
fills the gap between automation tools!
</p>
<p>
That's it! Happy automation :-)
</p>Ahmed AbouZaidhttp://www.blogger.com/profile/06013456198042893076noreply@blogger.com0tag:blogger.com,1999:blog-409120552279009989.post-3008708846467533172022-08-08T00:30:00.024+02:002023-11-01T23:52:25.360+02:002 ways to route Ingress traffic across namespaces - Kubernetes<div class="separator" style="clear: both;">
<a target="_blank" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhKhoZzL5hn4x3fY1dY68YnRAupqU_Bs3CoHEFrHyBxhIyxC47ehVlRmmCDVfGXVw1xN2y8FGpgyh76eWnaAEY1vW9BuuBBNm15GlKs5L24L-Rtaoyctr8PPJ9hJjDkFgw0fMvkDOv_-LDqrAXdami6N354Wlftd-s03cTNWVjsk4A4Dvp0sVJEdxQQXA/s768/Kubernetes-Ingress.png" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" width="600" data-original-height="427" data-original-width="768" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhKhoZzL5hn4x3fY1dY68YnRAupqU_Bs3CoHEFrHyBxhIyxC47ehVlRmmCDVfGXVw1xN2y8FGpgyh76eWnaAEY1vW9BuuBBNm15GlKs5L24L-Rtaoyctr8PPJ9hJjDkFgw0fMvkDOv_-LDqrAXdami6N354Wlftd-s03cTNWVjsk4A4Dvp0sVJEdxQQXA/s768/Kubernetes-Ingress.png"/>
</a>
</div>
<p>
The tech industry is full of workarounds, and you probably rely on
one or more. There is no problem with that per se, but it's important to
review your workarounds from time to time because there could be a new
standard/intuitive way to make it.
</p>
<h2 id="the-problem">The Problem</h2>
<p>
A couple of years ago I had a use case where a single domain has 2
sub-paths each of them using its own service in different namespaces.
Let's see this example:
</p>
<pre class="plain-pre">
example.com/app => service "backend" in namespace "app"
example.com/blog => service "wordpress" in namespace "blog"
</pre>
<p>
The problem was that the <tt>Ingress</tt> object is
<tt>namespaced</tt> which means that it interacts with services
within the same namespace. Also, only one ingress object per host/domain
is allowed.
</p>
<p>
So at that time I found a generic solution which looks like a
workaround. Actually, by thinking about it now, it was not a bad
workaround. It depends on how you manage your infrastructure, and you
can think about it as a centralized vs decentralized approach.
</p>
<h2 id="the-solution">The Solution</h2>
<p>
So here are the 2 ways to route Ingress traffic across namespaces in
Kubernetes. The 1st is a generic way <strong>that will work with any
Ingress controller</strong>. The 2nd relies on the Ingress controller
capabilities <a
href="https://github.com/nginxinc/kubernetes-ingress">NGINX Ingress
Controller by NGINX, Inc.</a> (<strong>NOT</strong> <a
href="https://github.com/kubernetes/ingress-nginx">Ingress-NGINX
Controller</a> by Kubernetes).
</p>
<h3 id="option-one-generic-method---externalname-service">
Option One:
Generic method - ExternalName Service</h3>
<p>
This method relies on native Kubernetes <a
href="https://kubernetes.io/docs/concepts/services-networking/service/#externalname">ExternalName
Service</a> which is simply a DNS <tt>CNAME</tt>! This method is
centralized where it uses the normal Ingress object in addition to
ExternalName Service within the same namespace as a bridge to the
services in any other namespace.
</p>
<p>
The following is an example of that setup with a single Ingress
resource and 2 ExternalName services (3 endpoints which are
<tt>/</tt>, <tt>/coffee</tt>, and <tt>/tea</tt>).
</p>
<div class="separator" style="clear: both;">
<a target="_blank" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrpuGFLT_Rqm3xO4GZclhjWFet0XHhdwpbzMyFSSMQTLbzGywk4q7pMmJB_LVf1uWXhhUCm-LroxFJiBbAwS3pO_sYxfblc82oPMo5260_aoPC_zaqN4SfRoBTdOb6Qgi_rY3O_ErepNwV59Lmwe2oScg4m3fpmm15So_Msu-KADPo8Cl5L8sZvU60Ng/s1046/cross-namespace-ingress-with-externalname-service.png" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" width="600" data-original-height="602" data-original-width="1046" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrpuGFLT_Rqm3xO4GZclhjWFet0XHhdwpbzMyFSSMQTLbzGywk4q7pMmJB_LVf1uWXhhUCm-LroxFJiBbAwS3pO_sYxfblc82oPMo5260_aoPC_zaqN4SfRoBTdOb6Qgi_rY3O_ErepNwV59Lmwe2oScg4m3fpmm15So_Msu-KADPo8Cl5L8sZvU60Ng/s1046/cross-namespace-ingress-with-externalname-service.png"/>
</a>
</div>
<p>
Config for <tt>shop.example.com</tt> including the 2 sub-paths
<tt>/coffee</tt> and <tt>/tea</tt> in addition to the root
<tt>/</tt>.
</p>
<pre class="plain-pre">
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: shop-ingress
namespace: shop
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
tls:
- hosts:
- shop.example.com
secretName: shop-secret
rules:
- host: shop.example.com
http:
paths:
- path: /coffee
pathType: Prefix
backend:
service:
name: coffee-svc-bridge
port:
number: 80
- path: /tea
pathType: Prefix
backend:
service:
name: tea-svc-bridge
port:
number: 80
</pre>
<p>
The <tt>coffee-svc-bridge</tt> service in the <tt>shop</tt>
namespace is a <tt>CNAME</tt> for the <tt>coffee-svc</tt>
service in <tt>coffee</tt> namespace:
</p>
<pre class="plain-pre">
apiVersion: v1
kind: Service
metadata:
name: coffee-svc-bridge
namespace: shop
spec:
type: ExternalName
externalName: coffee-svc.coffee
</pre>
<p>
The <tt>tea-svc-bridge</tt> service in the <tt>shop</tt>
namespace is a <tt>CNAME</tt> for the <tt>tea-svc</tt> service
in <tt>tea</tt> namespace:
</p>
<pre class="plain-pre">
apiVersion: v1
kind: Service
metadata:
name: tea-svc-bridge
namespace: shop
spec:
type: ExternalName
externalName: tea-svc.tea
</pre>
<p>
As you see, the Ingress config comes in 1 part and is normal. And use
the ExternalName services as a bridge to access the services in the
other namespaces.
</p>
<h3
id="option-two-controller-specific-method---mergeable-ingress-resources">Option
Two: Controller-specific method - Mergeable Ingress Resources</h3>
<p>
The other option is using controller-specific capabilities to achieve
that goal. There are dozens of Ingress controllers for Kubernetes like
the Ingress-NGINX (by Kubernetes project), NGINX Ingress Controller (by
NGINX, Inc.), Traefik, HAProxy, Istio, and many more.
</p>
<p>
Here I will cover only <a
href="https://github.com/nginxinc/kubernetes-ingress">NGINX Ingress
Controller by NGINX, Inc.</a>, but the idea is the same, using the
controller-specific features.
</p>
<p>
If you took a look at the official Nginx docs you will find the <a
href="https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/cross-namespace-configuration">Cross-namespace
Configuration</a> page suggests using <a
href="https://github.com/nginxinc/kubernetes-ingress/tree/v2.3.0/examples/mergeable-ingress-types">Mergeable
Ingress Resources</a>.
</p>
<p>
That approach relies on a simple idea, there is a single Ingress
resource that has all configurations related to the host/domain and that
resource is called "master", and any number of the Ingress resources
handles the paths under that host/domain and each of these resources is
called "minion".
</p>
<p>
Each one of the master or minion can or can not contain some Ingress
annotations based on their role. Here I will use here the examples from
the official documentation.
</p>
<div class="separator" style="clear: both;">
<a target="_blank" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjo8zRFuoXLTM9buG8nn9eKiE6___XYt6zfYl1_qKl7IMgHiIsbSrZcu1TvuQkmtBX1wbK0Hb2qSwLBLEu5XqTH9qngUYZ5gI5G1y8aiADFaqz5s1znK2tymbNjGO5-joQ6fci5A6XP2D1Dp_jM5HbPybZgUWm7DoAjEWXAieJoIu6W5XqAAdgRDmXAAA/s1046/cross-namespace-ingress-with-mergeable-ingress-resources.png" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" width="600" data-original-height="467" data-original-width="1046" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjo8zRFuoXLTM9buG8nn9eKiE6___XYt6zfYl1_qKl7IMgHiIsbSrZcu1TvuQkmtBX1wbK0Hb2qSwLBLEu5XqTH9qngUYZ5gI5G1y8aiADFaqz5s1znK2tymbNjGO5-joQ6fci5A6XP2D1Dp_jM5HbPybZgUWm7DoAjEWXAieJoIu6W5XqAAdgRDmXAAA/s1046/cross-namespace-ingress-with-mergeable-ingress-resources.png" />
</a>
</div>
<p>
Config for <tt>shop.example.com</tt> like TLS and host-level
annotations.
</p>
<pre class="plain-pre">
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: shop-ingress-master
namespace: shop
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.org/mergeable-ingress-type: "master"
spec:
tls:
- hosts:
- shop.example.com
secretName: shop-secret
rules:
- host: shop.example.com
</pre>
<p>
Config for <tt>shop.example.com/coffee</tt> which is in the
<tt>coffee</tt> namespace and routes the traffic of the
<tt>coffee-svc</tt> service.
</p>
<pre class="plain-pre">
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: shop-ingress-coffee-minion
namespace: coffee
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.org/mergeable-ingress-type: "minion"
spec:
rules:
- host: shop.example.com
http:
paths:
- path: /coffee
pathType: Prefix
backend:
service:
name: coffee-svc
port:
number: 80
</pre>
<p>
Config for <tt>shop.example.com/tea</tt> which is in the
<tt>tea</tt> namespace and routes the traffic of the
<tt>tea-svc</tt> service.
</p>
<pre class="plain-pre">
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: shop-ingress-tea-minion
namespace: tea
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.org/mergeable-ingress-type: "minion"
spec:
rules:
- host: shop.example.com
http:
paths:
- path: /tea
pathType: Prefix
backend:
service:
name: tea-svc
port:
number: 80
</pre>
<p>
As you see, the Ingress config is split into 2 parts, the host/domain
config, and the paths config. Each one of them could be in a different
namespace and handles the services in that namespace.
</p>
<h2 id="conclusion">Conclusion</h2>
<p>
Maybe the first approach looks like a workaround, but for many
workloads could be better and easier to follow and digest. But in
general, it's good to have different ways to use what's fit better.
</p>
<p>
Enjoy :-)
</p>
Ahmed AbouZaidhttp://www.blogger.com/profile/06013456198042893076noreply@blogger.com0tag:blogger.com,1999:blog-409120552279009989.post-1923601397667763982022-07-27T00:30:00.006+02:002022-07-30T08:58:32.754+02:00Notes about KRM Functions - Kustomize
<p>
Recently I dived into the new plugin system in Kustomize, <a
href="https://github.com/kubernetes-sigs/kustomize/blob/master/cmd/config/docs/api-conventions/functions-impl.md">KRM
Functions</a>, so I wanted to know more about it. <a
href="https://github.com/kubernetes/design-proposals-archive/blob/main/architecture/resource-management.md">Kubernetes
Resource Model</a> or <tt>KRM</tt> for short is simply a unified way
to work with resources in Kubernetes ecosystem. For example, all plugins
will have the same input and output format.
</p>
<p>
Here is a summary I found useful to share:
</p>
<ul>
<li>Kustomize decided to adapt KRM (Kubernetes Resource Model) functions
from <a target="_blank" href="https://kpt.dev/">kpt</a> ... and that's actually not new,
it's been there for some time (around 2020).</li>
<li>The goal is to deprecate the old plugins style model. Kustomize
already deprecated both <a
href="https://kubectl.docs.kubernetes.io/guides/extending_kustomize/go_plugins/">Go
Plugins</a> and <a
href="https://kubectl.docs.kubernetes.io/guides/extending_kustomize/exec_plugins/">Exec
plugins</a> in favour of KRM style.</li>
<li>KRM Functions style has 2 ways for the plugins: <a
href="https://kubectl.docs.kubernetes.io/guides/extending_kustomize/containerized_krm_functions/">Containerized
KRM Functions</a> and <a
href="https://kubectl.docs.kubernetes.io/guides/extending_kustomize/exec_krm_functions/">Exec
KRM Functions</a>.</li>
<li>The containerized KRM function is really useful one because you
don't need to manage and download the Kustomize plugins (it was super
annoying to manage plugins especially across multiple OS).</li>
</ul>
<p>
However, KRM functions are still alpha but look super promising,
however, they are still buggy or incomplete for some use cases!
</p>
<ul>
<li><strong>KRM exec</strong> has <a
href="https://github.com/kubernetes-sigs/kustomize/issues/4347">a
bug</a> which makes it almost unusable. In the <a
href="https://github.com/kubernetes-sigs/kustomize/pull/4654">PR no.
#4654</a> I've a proposal to fix that issue.</li>
<li><strong>KRM container</strong> has also <a
href="https://github.com/GoogleContainerTools/kpt/issues/3118">some
issues</a>! It only works with KRM resources but not any external files
(for example, if a plugin reads files from the disk, like creating
ConfigMap from a text file, that will not work at the moment).</li>
</ul>
<p>
In May 2022, I decided to go a bit further and try to implement the
KRM style to one of the existing plugins. So I've selected
<tt>SopsSecretGenerator</tt> Kustomize plugin and introduced KRM
support in the <a
href="https://github.com/goabout/kustomize-sopssecretgenerator/pull/32">PR
no. #32</a> So if you want to have an idea how KRM style looks like in
action, then take a look at the change I made in that PR (It has been
merged already).
</p>
<p>
In conclusion, KRM Functions look super promising but they are not
that mature yet in Kustomize and they don't fit all the use cases.
</p>
Ahmed AbouZaidhttp://www.blogger.com/profile/06013456198042893076noreply@blogger.com0tag:blogger.com,1999:blog-409120552279009989.post-17145003843351153692022-07-02T00:30:00.015+02:002022-07-08T17:05:23.204+02:00Kubernetes Security Best Practices with tips for the CKS exam - Presentation<img alt="" style="display: none;" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1dx17_hFpGGZ44IhPUSVbuXcOur8wXesy_Cj00iYY0d-4SpRQYlBACCPaRoOglNbkoLlgB8LMzohoHJc21QX9F2v1KF69H0WFwk6Sl2WWtUHlf0IXIkgbZHQSchRfk2GqLvAy2Tp8xx7-Mgchb-gghzgItQb2xS43SFtd863Ia5l29aijGvRHZ8kTbQ/s1707/Kubernetes-Security-Best-Practices-With-tips-for-the-CKS-exam-Ahmed-AbouZaid.png"/>
<p>
At the end of last year (2021), and after a couple of years of
Kubernetes production hands-on, I found that it was time to dive more
into Kubernetes security, and after a lot of reading and practising, I
got my CKS certificate. Also, for the last 3 quarters, security was one
of the focus areas in my team, and I was taking care of it.
</p>
<p>
For that reason, I decided to consolidate that into a session which
is a combination of Kubernetes Security Best Practices and tips for the
Certified Kubernetes Security Specialist (CKS) exam to share the
knowledge in my team as well across teams.
</p>
<p>
The session is just 15 Min in total. The first 6 Min are for everyone
and the rest for Kubernetes specialists or anyone who wants to dive more
into Kubernetes security topics. If you are just interested in the
tools, then jump to section #5 <tt>Kubernetes Security Starter
Kit</tt>. If you are just interested in the CKS exam tips, then jump
to section #6 <tt>CKS Exam Overview and Tips</tt>.
</p>
<h3 id="agenda">Agenda:</h3>
<ol type="1">
<li>Introduction</li>
<li>Shift-left and DevSecOps</li>
<li>General Security Concepts</li>
<li>The 4C's of Cloud Native Security</li>
<li>Kubernetes Security Starter Kit</li>
<li>CKS Exam Overview and Tips</li>
</ol>
<p>
<strong>Note:</strong> If you want more details about CKS, checkout
my previous post for more info <a
href="https://tech.aabouzaid.com/2021/11/now-i-am-a-certified-kubernetes-security-specialist-plus-exam-tips.html">Now
I'm a Certified Kubernetes Security Specialist + exam tips</a>.
</p>
<hr />
<div style="text-align: center;">
<iframe src="//www.slideshare.net/slideshow/embed_code/key/CXQvouF2PBaKbV" width="640" height="475" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width: 100%;" allowfullscreen>
</iframe>
<div style="margin-bottom:5px">
<a target="_blank" href="//www.slideshare.net/ahmed-abouzaid/kubernetes-security-best-practices-with-tips-for-the-cks-exam" title="Kubernetes Security Best Practices - With tips for the CKS exam" target="_blank">The slides of Kubernetes Security Best Practices session</a> by <a target="_blank" href="https://www.slideshare.net/ahmed-abouzaid" target="_blank">Ahmed AbouZaid</a>
</div>
</div>
<br />
<div style="text-align: center;">
<iframe width="640" height="360" src="https://www.youtube.com/embed/86ASVhwR8xU" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen>
</iframe>
<div style="margin-bottom:5px">
The recording of Kubernetes Security Best Practices session
</div>
</div>
<br />
<h3 id="overview">Overview:</h3>
<p>
A dive into Kubernetes Security Best Practices and tips for the
Certified Kubernetes Security Specialist (CKS) exam.
</p>
<p>
The 1-3 sections are for everyone and will cover the container era's
security. So it doesn't matter your title or background; they are a good
start for anyone.
</p>
<p>
The 4-6 sections will dive more into Kubernetes security, so DevOps
engineers and SREs will probably find that more interesting. But in
general, anyone interested in Kubernetes security is more than
welcome.
</p>
<hr />
<p>
That's it, enjoy :-)
</p>
Ahmed AbouZaidhttp://www.blogger.com/profile/06013456198042893076noreply@blogger.com0tag:blogger.com,1999:blog-409120552279009989.post-67725149495251045062022-06-22T00:30:00.024+02:002022-07-03T17:56:48.102+02:00Moderating DevOps circle at JobStack 2022<p>
Last Satruday (18.06.2022), I had a great chance to moderate and participate the DevOps circle in <a href="https://jobstack.talentsarena.net" target="_blank">JobStack 2022</a> by <a href="https://talentsarena.net/" target="_blank">Talents Arena</a>. JobStack is the biggest virtual tech job fair in the region (MENA) and this was the 4rd edition.
</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgEDUdRF3MsdkHD8huE6KNWjnpahlPfdlVht-XqfYkh8Mm7YeqH--liVlKXtAkzBYxU3WyiowFBtQg_koXT9UVIJG_UNJrVs-1NC3YpupTmV4yqKMp3XeB-Iy2hfggwcdhXDJY5AT3i3oe-uNxTg_6K2k6BzBtBq2J5ZTDB1kJrfoPplp0uMt4-TswuRQ/s1024/devops-circle-jobstack-2022.jpeg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="600" data-original-height="768" data-original-width="1024" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgEDUdRF3MsdkHD8huE6KNWjnpahlPfdlVht-XqfYkh8Mm7YeqH--liVlKXtAkzBYxU3WyiowFBtQg_koXT9UVIJG_UNJrVs-1NC3YpupTmV4yqKMp3XeB-Iy2hfggwcdhXDJY5AT3i3oe-uNxTg_6K2k6BzBtBq2J5ZTDB1kJrfoPplp0uMt4-TswuRQ/s600/devops-circle-jobstack-2022.jpeg"/></a></div>
<p>
With <a href="https://www.linkedin.com/in/hussein-abdelwahed/" target="_blank">Hussein El-Sayed</a> (Software engineer III at AWS) and <a href="https://www.linkedin.com/in/maradwan/" target="_blank">Mohamed Radwan</a> (Sr. Cloud Architect at T-Systems) ... we answered many different questions about DevOps.
The circle or the AMA session was heavily based on a previous collaborative session between us in 2022 (<a href="https://www.youtube.com/watch?v=sApv-E87mXI" target="_blank">DevOps! What, Why, and How?</a> - Arabic)
</p>
<p>
The whole event was great and had a lot of fruitful sessions and discussions.
</p>Ahmed AbouZaidhttp://www.blogger.com/profile/06013456198042893076noreply@blogger.com0tag:blogger.com,1999:blog-409120552279009989.post-68839479927026935232022-04-04T00:30:00.004+02:002022-07-05T15:41:07.237+02:00Apply Kustomize builtin transformers on a single resource - Kubernetes<div class="separator" style="clear: both;">
<a target="_blank" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjpB04ybrYCTT3s4thZW0KqSQ8XTlNhLog_iCpF8ljvL7DcyJSihxK7wqxL9bE0BVdLlngcAO0JaCDe9GCpGxwbMlxHH_s1ZfZMy_ABb0Z4Fn6XMlI4yHQW6wO2yDgwB8kgfetX0GPfA6A9OM8XVJvTbycn03-ilYuGavVR_VEPljndRvl9RBhVE5az6w/s600/kustomize.png" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" width="600" data-original-height="165" data-original-width="600" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjpB04ybrYCTT3s4thZW0KqSQ8XTlNhLog_iCpF8ljvL7DcyJSihxK7wqxL9bE0BVdLlngcAO0JaCDe9GCpGxwbMlxHH_s1ZfZMy_ABb0Z4Fn6XMlI4yHQW6wO2yDgwB8kgfetX0GPfA6A9OM8XVJvTbycn03-ilYuGavVR_VEPljndRvl9RBhVE5az6w/s600/kustomize.png"/>
</a>
</div>
<p>
<a
href="https://tech.aabouzaid.com/search/label/Kustomize">Kustomize</a>
is a template-free declarative management tool for Kubernetes resources.
Kustomize has 2 main concepts: Generators and Transformers. In short,
the first able to create K8s manifests, and the second is able to
manipulate K8s manifests. In this post, I'm interested in the Kustomize
Transformers.
</p>
<a name='more'></a>
<p>
In Kustomize, "<a
href="https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/#everything-is-a-transformer">everything
is a transformer</a>"! In fact, the <tt>transformers have a general
generative power</tt> too! But always remember that transformers run
after generators. Anyway, this is not the topic of this post. I just
wanted to highlight how powerful Kustomize transformers are.
</p>
<p>
---
</p>
<p>
Kustomize has a bunch of default/builtin transformers like
<tt>namePrefix</tt>, <tt>nameSuffix</tt>,
<tt>commonAnnotations</tt>, and <tt>commonLabels</tt> (you
probably used one of them if you just used Kustomize before). Those
builtin transformers are global by default which means they are applied
to <strong>all resources</strong> managed by Kustomize.
</p>
<p>
<strong>The question is, what if you want to apply a change like
<tt>namePrefix</tt> to a specific resource?</strong>
</p>
<p>
You can apply any of <a
href="https://github.com/kubernetes-sigs/kustomize/tree/master/api/internal/builtins">Kustomize
builtin transformers</a> to certain resources. Here is an example:
</p>
<pre class="plain-pre">
# kustomization.yaml
transformers:
- |-
apiVersion: builtin
kind: PrefixSuffixTransformer
metadata:
name: my-service
prefix: dev-
fieldSpecs:
- path: metadata/name</pre>
<p>
It works exactly like <tt>namePrefix</tt> but on a single
resource called <tt>my-service</tt> which will be transformed to
<tt>dev-my-service</tt>.
</p>
<p>
That's it! :-)
</p>
Ahmed AbouZaidhttp://www.blogger.com/profile/06013456198042893076noreply@blogger.com0tag:blogger.com,1999:blog-409120552279009989.post-86356087246676151342022-03-03T01:30:00.014+02:002022-04-01T14:16:13.653+02:00Refactoring Bank-Vaults operator for full Vault management support<div class="separator" style="clear: both;">
<a target="_blank" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEidwJIXM3swuR1uuySKQqchsxJ8LMhfPS_VO6_ZPsXEugqGQ8zYoWJaZJ-gINkPvoQrH7yakdRUkmnv0ygEzXUj_9boDUF3b5l9NY8ikBSer5SmFx4vr46UL43t2XCsGfcmGOlNQcETE4NHTGY_6HJC3T1-BfGsfKm8YURrPNMrIKtXFXi0ci1v0d0YgQ/s1024/bank-vaults-logo.png" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" width="600" data-original-height="561" data-original-width="1024" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEidwJIXM3swuR1uuySKQqchsxJ8LMhfPS_VO6_ZPsXEugqGQ8zYoWJaZJ-gINkPvoQrH7yakdRUkmnv0ygEzXUj_9boDUF3b5l9NY8ikBSer5SmFx4vr46UL43t2XCsGfcmGOlNQcETE4NHTGY_6HJC3T1-BfGsfKm8YURrPNMrIKtXFXi0ci1v0d0YgQ/s1024/bank-vaults-logo.png"/>
</a>
</div>
<p>
In Q1 2022, my friend <a
href="https://www.linkedin.com/in/wazery/">Islam Wazery</a> and I were
working on an interesting enhancement for the open-source Vault
Kubernetes Operator, <a
href="https://github.com/banzaicloud/bank-vaults">Bank-Vaults</a>.
</p>
<p>
It's one of my biggest open-source contributions recently. In this
meta post, I like to share some details about the problem we were trying
to solve, goal, available solutions, implementation details, and
challenges during working on the new feature.
</p>
<hr />
<h2 id="toc">ToC</h2>
<ul>
<li><a href="#1-intro">1. Intro</a></li>
<li><a href="#2-problem">2. Problem</a></li>
<li><a href="#3-goal">3. Goal</a></li>
<li><a href="#4-available-solutions">4. Available solutions</a>
<ul>
<li><a href="#41-purge-anything-not-in-the-config">4.1 Purge anything
not in the config</a></li>
<li><a href="#42-compare-differences-between-the-old-and-new-config">4.2
Compare differences between the old and new config</a></li>
<li><a href="#43-manage-changes-statefully">4.3 Manage changes
statefully</a></li>
<li><a href="#44-handel-config-individually">4.4 Handel config
individually</a></li>
</ul></li>
<li><a href="#5-implementation">5. Implementation</a></li>
<li><a href="#6-challenges">6. Challenges</a>
<ul>
<li><a href="#61-project-complexity">6.1 Project complexity</a></li>
<li><a href="#62-refactoring-the-write-path">6.2 Refactoring the write
path</a></li>
<li><a href="#63-only-generic-acceptance-tests">6.3 Only generic
acceptance tests</a></li>
<li><a href="#64-coordination">6.4 Coordination</a></li>
</ul></li>
<li><a href="#7-result">7. Result</a></li>
<li><a href="#8-links">8. Links</a></li>
</ul>
<hr />
<h2 id="1-intro">1. Intro</h2>
<p>
Anyone who used Kubernetes knows that the <tt>Secret</tt>
resources are encoded, not encrypted, so you probably need another
solution to manage your secrets and sensitive data. HashiCorp Vault is
one of the best tools for that purpose.
</p>
<p>
In case you didn't use <a
href="https://www.vaultproject.io/">Vault</a> before, here is a short
intro from <a target="_blank" href="https://www.vaultproject.io/docs/what-is-vault">its
docs</a>:
</p>
<blockquote>
<p>
Vault is an identity-based secrets and encryption management system.
A secret is anything that you want to tightly control access to, such as
API encryption keys, passwords, or certificates. Vault provides
encryption services that are gated by authentication and authorization
methods. Using Vault's UI, CLI, or HTTP API, access to secrets and other
sensitive data can be securely stored and managed, tightly controlled
(restricted), and auditable.
</p>
</blockquote>
<p>
HashiCorp already provides <a
href="https://www.vaultproject.io/docs/platform/k8s">resources</a> to
install Vault on Kubernetes as well as a <a
href="https://github.com/hashicorp/vault-helm">Helm chart for Vault</a>.
However, there is no official solution from HashiCorp to manage Vault
itself on Kubernetes. And here comes <strong>Bank-Vaults, the Vault
Swiss Army Knife!</strong>
</p>
<p>
<a target="_blank" href="https://github.com/banzaicloud/bank-vaults">Bank-Vaults</a>
by <a target="_blank" href="https://github.com/banzaicloud">Banzai Cloud</a> is an
open-source umbrella project which provides various tools (Operator,
Configurer, Vault Env injector, and more) for ease of use and operation
of Hashicorp Vault.
</p>
<p>
The most exciting part here is the Vault <a
href="https://tech.aabouzaid.com/search?q=Kubernetes+Operator">Operator</a>
by Bank-Vaults. Which allows you to manage Vault on Kubernetes. And
based on my research, it's the only operator in the market for Vault, so
I started a PoC to use it in production.
</p>
<h2 id="2-problem">2. Problem</h2>
<p>
After the initial setup, It seemed that the Bank-Vaults operator was
mature and production-ready. It had many features like bootstrap,
sealing, unsealing, cloud backend, and all the features we need, but
it's missing an important feature, it didn't support full Vault
management! It handled the creation of Vault's config (like policies,
secrets engines, auth methods, etc.), but it didn't handle the removal
of the config! And that was confirmed by the <a
href="https://github.com/banzaicloud/bank-vaults/issues/605">issue no.
#605</a> <strong>which had been unresolved for more than 2
years!</strong> (Aug 2019)
</p>
<p>
Next, I've checked the operator code, and it turned out that the
operator works only with create/update, but it doesn't have any
mechanism to work the config removal. No one fixed that because it's a
full feature that needs much work (well, that's why it needed more than
2 years to fix).
</p>
<p>
That means the operator doesn't <strong>fully</strong> manage Vault!
Unfortunately, this is a deal-breaker to use the operator in production.
And to fix that, there are several ways to remove the unmanaged config.
In the following sections, I dive more into the available mechanisms to
handle the config removal, but first, let's set the goal.
</p>
<h2 id="3-goal">3. Goal</h2>
<p>
I already experienced managing Vault using <a
href="https://tech.aabouzaid.com/search/label/Terraform">Terraform</a>
but doing that on Kubernetes would be a snowflake where it needs an
extra stateful tool! I like to use Terraform for the infrastructure like
Kubernetes clusters but not for the apps. Hence, I don't want to go that
way.
</p>
<p>
So the ultimate goal is that the operator should be able to manage
Vault completely. It should add and remove Vault config like policies,
secrets engines, auth methods, etc. And that should be done using the
Kubernetes ecosystem in a cloud-native approach.
</p>
<h2 id="4-available-solutions">4. Available solutions</h2>
<p>
Based on my previous experience with code, infrastructure as code,
and configuration management tools, there are several ways to achieve
config removal, and each one of them has pros and cons.
</p>
<h3 id="41-purge-anything-not-in-the-config">
4.1 Purge anything not in
the config</h3>
<p>
The first mechanism is simply the <tt>purge</tt> approach, where
the operator removes anything not in the configuration. This mechanism
compares Bank-Vaults config and Vault config and removes the
differences.
</p>
<p>
So this approach is somewhat radical. It doesn't allow any manual
changes, and any change outside the configuration will be removed. But
the good side is that, well, it doesn't allow any manual changes! So the
configuration is the source of truth. However, there is mitigation to
allow some manual changes by excluding some of the configs. I will
discuss it in the implementation section.
</p>
<h3 id="42-compare-differences-between-the-old-and-new-config">
4.2
Compare differences between the old and new config</h3>
<p>
The second mechanism is the <tt>last-diff</tt> approach, where
the operator compares the old and the new config and removes anything
not in the new config. This way is considered "semi-stateful" where you
need to have the old config and the new config to compare them. This
approach allows manual changes outside the operator, but the operator is
only aware of the last changes.
</p>
<h3 id="43-manage-changes-statefully">4.3 Manage changes statefully</h3>
<p>
The third mechanism is the <tt>diff</tt> approach, where the
operator maintains a state of all its operations, and with any new
change, it compares the changes with the state (this is the
<tt>Terraform</tt> way). This way is fully-stateful, which allows
for tracking the changes done by the operator and allows manual changes
outside the operator.
</p>
<h3 id="44-handel-config-individually">
4.4 Handel config
individually</h3>
<p>
Finally, the fourth mechanism is the <tt>flag</tt> approach,
where the operator manages the config according to a config flag. For
example, each policy in the operator config could have a field called
"state", and its value could be "present" or "absent" (this is the style
of config management tools like <tt>Ansible</tt>). In this solution,
it's possible to have managed, and unmanaged config but the biggest
downside is that you need to deal with the config on the individual
level.
</p>
<h2 id="5-implementation">5. Implementation</h2>
<p>
In the cloud-native era, the first style looks the most suitable
approach where full management is assumed. So anything that is not in
the config would be removed. And to mitigate that behavior, it's
'possible to exclude some sections like policies, auth methods, etc., so
they could have manual changes if needed.
</p>
<p>
Vault has main 7 configuration sections:
</p>
<ul>
<li>Audit</li>
<li>Auth</li>
<li>Groups</li>
<li>GroupAliases</li>
<li>Plugins</li>
<li>Policies</li>
<li>Secrets</li>
</ul>
<p>
Each section already has the "add" mechanism, and it's able to create
the config in Vault, and the goal is to add the "remove" mechanism to
have full <tt>CRUD</tt> (Create, read, update and delete). However,
the "adding" code wasn't follow Golang style and it needed to be
refactored. So for each section, the code is refactored first, then the
"removing" code is added.
</p>
<p>
Let's take <tt>policies</tt> as an example (which will be the
same way for all 7 configs mentioned above); the "removing" part works
as the following:
</p>
<ol type="1">
<li>Bank-Vaults operator reads its config file with managed
policies.</li>
<li>Then, it calls Vault to get all already configured policies.</li>
<li>Then, it compares what's in the config (the desired state) with
Vault (the actual state).</li>
<li>Finally, if there are differences, then the Bank-Vaults operator
calls Vault to delete the unmanaged policies.</li>
</ol>
<p>
The final step is creating E2E tests to run in the CI (Github
Actions). The tests simply check different cases like removing a config
while the purge option is disabled/enabled fully/partially. Now let's
take a look at the challenges I had while working on this feature.
</p>
<h2 id="6-challenges">6. Challenges</h2>
<p>
In the following sections, I'd like to share the top challenges while
introducing full Vault management in the Bank-Vaults operator.
</p>
<h3 id="61-project-complexity">6.1 Project complexity</h3>
<p>
Bank-Vaults is not just the operator; it's an umbrella project to
work with Vault. It's a mono repo with many shared parts. For example,
the operator relies on a CLI tool with the same name.
</p>
<p>
Hence, the first challenge was to understand the project structure
and where exactly to change, and how the changes could affect the rest
of the project.
</p>
<h3 id="62-refactoring-the-write-path">
6.2 Refactoring the write
path</h3>
<p>
After a thoughtful dive into the project, it was clear what and where
I should change to fully manage Vault by Bank-Vaults (so it can add and
remove config in Vault). However, the write path code in the operator
(that's responsible for creating and updating managed config) doesn't
follow the Golang style. It was more like Python written in Golang. It
reminded me of when I wrote Golang for the first time, coming from a
Python background.
</p>
<p>
Leaving the write path code as it is would make the code oddly bad
and redundant. So the first step was refactoring the "write path" code,
then adding the "remove path" code (which is responsible for removing
any unmanaged config). And this was the second challenge to solve before
the actual implementation.
</p>
<h3 id="63-only-generic-acceptance-tests">
6.3 Only generic acceptance
tests</h3>
<p>
Another challenge was that the part I wanted to change didn't have
any unit tests, but only generic acceptance tests were available. Which
makes things harder to change. I needed to pay extra attention to ensure
I didn't break anything while refactoring the existing code and
introducing the new feature. That also means I should write some E2E
tests to avoid this situation in the future.
</p>
<h3 id="64-coordination">6.4 Coordination</h3>
<p>
As I mentioned before, this feature is a bit big, and it would be
implemented by 2 people (my friend Wazery and me). At the same time,
it's a new project we didn't work on before, and we didn't work together
before. So we needed to make sure that everything was clear and both of
us were aligned to deliver this feature in high quality.
</p>
<h2 id="7-result">7. Result</h2>
<p>
With the PR no. <a
href="https://github.com/banzaicloud/bank-vaults/pull/1538">#1538</a>,
and <a
href="https://github.com/banzaicloud/bank-vaults/releases/tag/1.15.1">Bank-Vaults
v1.15.1</a> was able to fully or partially purging unmanaged
configuration in Vault.
</p>
<p>
The user has the option to <a
href="https://github.com/banzaicloud/bank-vaults/issues/1533">fully or
partially purge unmanaged config</a> as shown here:
</p>
<pre class="plain-pre">
purgeUnmanagedConfig:
# This will purge any unmanaged config in Vault.
enabled: true
# This will prevent purging unmanaged config for secret engines in Vault.
exclude:
secrets: true</pre>
<p>
To avoid behavior change, and since this feature is
<tt>destructive</tt>, it was safe to make it disabled by default.
The user needs to enable it explicitly in Bank-Vaults config. And as
usual, it's recommended to test it in a non-production environment
first.
</p>
<h2 id="8-links">8. Links</h2>
<p>
<strong>Wazery PRs:</strong>
</p>
<ul>
<li><a target="_blank" href="https://github.com/banzaicloud/bank-vaults/pull/1513">Auth
methods</a> (this one is really big)</li>
<li><a target="_blank" href="https://github.com/banzaicloud/bank-vaults/pull/1532">Audit
devices</a></li>
<li><a
href="https://github.com/banzaicloud/bank-vaults/pull/1535">Startup
secrets</a></li>
</ul>
<p>
<strong>My PRs:</strong>
</p>
<ul>
<li><a
href="https://github.com/banzaicloud/bank-vaults/pull/1503">Policies</a></li>
<li><a
href="https://github.com/banzaicloud/bank-vaults/pull/1514">Secrets
engines</a></li>
<li><a
href="https://github.com/banzaicloud/bank-vaults/pull/1519">Plugins</a></li>
<li><a
href="https://github.com/banzaicloud/bank-vaults/pull/1520">Identity
groups</a></li>
<li><a
href="https://github.com/banzaicloud/bank-vaults-docs/pull/89">Purge
unmanaged config docs</a></li>
<li><a
href="https://github.com/banzaicloud/bank-vaults/pull/1538">Finalize
config purge feature + E2E tests</a></li>
</ul>
<hr />
<p>
That's it! It was a pretty exciting journey for me π€©οΈ
</p>
Ahmed AbouZaidhttp://www.blogger.com/profile/06013456198042893076noreply@blogger.com0tag:blogger.com,1999:blog-409120552279009989.post-60841104255735360502022-02-02T01:30:00.004+02:002022-02-27T06:56:10.578+02:00Extending Jenkins to run resilient pipelines with long-running jobs - CI/CD<div class="separator" style="clear: both;">
<a target="_blank" href="https://blogger.googleusercontent.com/img/a/AVvXsEh34jDIBiamKrKQwC_DAg3ePQMOOeNnEVZB1R7TjXDLGLtB1q_A-tVSKlddY8xvPHYUScJKg3bHGWNZKIxrrU2iuSIf8gEuueqoxHp13tnnMXZFzb8LoKChkUUHkvF8DZmMvR5f7d3e4Xr-6cIcWOuGra6NeST3E1_Bhyyvz1Ri4n3ONm4j2RIh0jvGqQ=s910" style="display: block; padding: 1em 0px; text-align: center;">
<img alt="" border="0" data-original-height="481" data-original-width="910" height="338" src="https://blogger.googleusercontent.com/img/a/AVvXsEh34jDIBiamKrKQwC_DAg3ePQMOOeNnEVZB1R7TjXDLGLtB1q_A-tVSKlddY8xvPHYUScJKg3bHGWNZKIxrrU2iuSIf8gEuueqoxHp13tnnMXZFzb8LoKChkUUHkvF8DZmMvR5f7d3e4Xr-6cIcWOuGra6NeST3E1_Bhyyvz1Ri4n3ONm4j2RIh0jvGqQ=w640-h338" width="640" />
</a>
</div>
<p>
In 2021, I wrote a high-level blog post about <a
href="https://tech.aabouzaid.com/2021/09/how-a-small-task-force-revamped-and-modernized-a-gigantic-ci-pipeline.html">how
a small task force revamped and modernized a gigantic CI pipeline</a>
Which was an overview of the challenges we had while building advanced
declarative pipelines.
</p>
<p>
In today's post, I dive more into the technical details and
implementation and how we overcame long-running job limitations in
Jenkins Declarative Pipelines.
</p>
<a name='more'></a>
<p>
The post sheds the light on the new custom step <a
href="https://github.com/camunda-community-hub/camunda-jenkins-shared-library/blob/main/docs/conditionalRetry.md">conditionalRetry</a>
which's used to handel failures on spot/preemptible infrastructure so
you could save up to 90% of the costs and have stable builds as
well!
</p>
<p>
Sounds interesting? Read more details about it at: <a
href="https://camunda.com/blog/2022/02/how-we-overcame-long-running-job-limitations-in-jenkins-declarative-pipelines/">How
We Overcame Long-Running Job Limitations in Jenkins Declarative
Pipelines</a>
</p>
<p>
BTW, You can use that in your pipeline ... it's open-source ;-)
</p>
<p>
Enjoy :-)
</p>
Ahmed AbouZaidhttp://www.blogger.com/profile/06013456198042893076noreply@blogger.com0tag:blogger.com,1999:blog-409120552279009989.post-25564335362136253712022-01-11T01:30:00.007+02:002022-04-15T03:30:17.372+02:00Asdf-vm, a universal version manager - Tools<p>
Almost 2 years ago, I started to use <a
href="https://asdf-vm.com/">asdf</a> which is a great tool with a weird
name (at first sight, I felt like I just pressed 4 continuous letters
with a QWERTY layout!). For me, it's a <strong>version manager
Kraken</strong>!
</p>
<div class="separator" style="clear: both;">
<a target="_blank" href="https://blogger.googleusercontent.com/img/a/AVvXsEgiqxRtnpJznNyWl9t3YQx92lAbct9QdHMuYfllHaqVWGnSZ_y-qkQ66Z9C29wEYIRX7jgijmNrGFsp-YpNwpu48RZniYTSUGFMfQIi_oLytk7W57j2ran1pa5UZxMYxjUR7gYdVz8bBoSTVeN7BzAma-eUqvJ1fY5YOkiWSNFrF8IiecnOFH7mff_WCw=s512" style="display: block; padding: 1em 0px; text-align: center;">
<img alt="Kraken" border="0" data-original-height="473" data-original-width="512" height="369" src="https://blogger.googleusercontent.com/img/a/AVvXsEgiqxRtnpJznNyWl9t3YQx92lAbct9QdHMuYfllHaqVWGnSZ_y-qkQ66Z9C29wEYIRX7jgijmNrGFsp-YpNwpu48RZniYTSUGFMfQIi_oLytk7W57j2ran1pa5UZxMYxjUR7gYdVz8bBoSTVeN7BzAma-eUqvJ1fY5YOkiWSNFrF8IiecnOFH7mff_WCw=w400-h369" width="400" />
</a>
</div>
<a name='more'></a>
<center>
<p>
Image source: <a target="_blank" href="https://openclipart.org/detail/193917/attack-the-tower-kraken">Openclipart</a>
</p>
</center>
<p>
So what's actually <tt>asdf</tt>? Simply asdf manages your local
tools/languages version, supports plugins and works on Linux and MacOS.
Let me quote the intro from the tool page:
</p>
<blockquote>
<p>
asdf is a CLI tool that can manage multiple language runtime versions
on a per-project basis. It is like gvm, nvm, rbenv & pyenv (and
more) all in one! Simply install your language's plugin!
</p>
</blockquote>
<p>
For example, I'm working with different Kubernetes clusters, and not
all of them with the same version. And according to the Kubernetes <a
href="https://kubernetes.io/releases/version-skew-policy/">version skew
policy</a>, <tt>kubectl</tt> is supported within one minor version
(older or newer) of <tt>kube-apiserver</tt>. That means I need
different versions of kubectl to work with Kubernetes clusters if the
difference between them is more or less 1 version.
</p>
<p>
Here it comes <tt>asdf</tt> where it allows you to have different
kubectl versions per folder/project! And it supports tons of tools and
languages (for example, you can have different Python versions per
folder/project).
</p>
<p>
And <tt>asdf</tt> is not only a great tool for individual usage
but also on the team level. Where you are sure all team members are
using the same versions for the same projects. That really reduces
change errors and conflicts.
</p>
<p>
I will not write any further about <tt>asdf</tt>. This post is
just this quick intro about a tool I really like. To start using it,
take a look at <tt>asdf</tt> <a
href="https://asdf-vm.com/guide/introduction.html">Introduction</a> as
well as <a target="_blank" href="https://asdf-vm.com/guide/getting-started.html">Getting
Started</a> page; they have all that you need to start with it.
</p>
<p>
Enjoy :-)
</p>
Ahmed AbouZaidhttp://www.blogger.com/profile/06013456198042893076noreply@blogger.com0tag:blogger.com,1999:blog-409120552279009989.post-30894277391995599462021-12-12T01:30:00.003+02:002022-12-22T01:11:06.631+02:00Why Kubernetes? - DevOps
<p>
This post was part of <a
href="https://tech.aabouzaid.com/search/label/Developer%20Enablement">developer
enablement</a> activities at <a target="_blank" href="https://camunda.com/">Camunda</a>.
My team had a question about Kubernetes benefits, and the content of
this post was the answer I wrote to that question.
</p>
<div class="separator" style="clear: both;">
<a target="_blank" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgI-KSAMTqgK-6TWUBIGkvYkvhUMQAqQr4xTDhNOGJLW3J-_8htjVS1jrIPwLVduMLMJBYraxRANPHWb3nCGudkwQo70yYspEjCTdSA196VznGK4H8mF3wAL4xf9u23X3EBcsO3ggil8GgkXVJZzWuHBEmQziAkK2BVFNlaGaVCpCq930Zxo5velXsP0g/s800/kubernetes-logo.png" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" width="400" data-original-height="535" data-original-width="800" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgI-KSAMTqgK-6TWUBIGkvYkvhUMQAqQr4xTDhNOGJLW3J-_8htjVS1jrIPwLVduMLMJBYraxRANPHWb3nCGudkwQo70yYspEjCTdSA196VznGK4H8mF3wAL4xf9u23X3EBcsO3ggil8GgkXVJZzWuHBEmQziAkK2BVFNlaGaVCpCq930Zxo5velXsP0g/s400/kubernetes-logo.png"/>
</a>
</div>
<blockquote>
What are the benefits of using Kubernetes?
</blockquote>
<a name='more'></a>
<ul>
<li><a href="#1-why-was-kubernetes-created">1. Why was Kubernetes
created?</a></li>
<li><a href="#2-what-are-kubernetes-features">2. What are Kubernetes'
features?</a>
<ul>
<li><a href="#21-open-source">2.1 Open-source!</a></li>
<li><a href="#22-no-vendor-lock-in">2.2 No vendor lock-in</a></li>
<li><a href="#23-maturity">2.3 Maturity</a></li>
<li><a href="#24-big-community-and-ecosystem">2.4 Big community and
ecosystem</a></li>
<li><a href="#25-extensible-architecture">2.5 Extensible
architecture</a></li>
</ul></li>
<li><a href="#3-how-do-kubernetes-help-us">3. How do Kubernetes help
us?</a></li>
</ul>
<p>
---
</p>
<h2 id="1-why-was-kubernetes-created">
1. Why was Kubernetes
created?</h2>
<p>
First, we need to know the problem that Kubernetes is trying to
solve.
</p>
<p>
When Docker containers were introduced in 2013, and many companies
adopted the new technology, everything changed. Literally, everything,
from how the code works to deployment, monitoring, and operations.
</p>
<p>
Hence, the need for a new system emerged to deal with this unique
ecosystem. And that was exactly what happened. Many systems were born to
solve this problem ... "<strong>How to manage the containers in
production?</strong>", which implies "**How to create and manage <a
href="https://github.com/cncf/toc/blob/master/DEFINITION.md">Cloud-native
software</a>?**".
</p>
<p>
Many systems entered the race, but Kubernetes was the winning horse
of that race! Since then, Kubernetes (aka <strong>K8s</strong>) has been
the leading platform for managing containers.
</p>
<h2 id="2-what-are-kubernetes-features">
2. What are Kubernetes'
features?</h2>
<h3 id="21-open-source">2.1 Open-source!</h3>
<p>
Because K8s is an open-source project, it's been grown so fast and
supported by many big companies like Google, Red Hat, IBM, Microsoft,
VMware, and many others. The fact that K8s is an open-source project
affects and touches many other features.
</p>
<h3 id="22-no-vendor-lock-in">2.2 No vendor lock-in</h3>
<p>
Mainly because it's open-source, anyone can use it anywhere,
on-premise, private Cloud, public Cloud, or even on your laptop! So K8s
can work on multi-cloud and be installed and migrated from one provider
to another easily (compared to many other systems).
</p>
<h3 id="23-maturity">2.3 Maturity</h3>
<p>
As mentioned, K8s was the winning horse of the container
orchestration race, and many companies have adopted it in production
workloads. Also, as an open-source project, K8s has a mature steering
committee. So it's less likely to have crazy changes affect the project
(or changes that serve some companies' goals).
</p>
<h3 id="24-big-community-and-ecosystem">
2.4 Big community and
ecosystem</h3>
<p>
For those historical reasons, now K8s is considered the leading
open-source platform to manage containers, and it's used everywhere.
Thus, no need to reinvent the wheel for common use cases but focus on
business logic. And when help is needed, it's easy to find it.
</p>
<h3 id="25-extensible-architecture">2.5 Extensible architecture</h3>
<p>
One of the most outstanding features of K8s is extensibility! There
is a clear and straightforward way to extend K8s. No need for crazy
hacks nor to have external systems to support custom use cases (many
other projects have failed just because of that!) And for the exact same
reason, we don't need to do everything ourselves, and we can use what
the community and other companies provide to extend K8s.
</p>
<h2 id="3-how-do-kubernetes-help-us">3. How do Kubernetes help us?</h2>
<p>
All of that sounds like "features" more than "benefits" right?
</p>
<p>
Using Kubernetes, those features are translated to the following
benefits:
</p>
<ul>
<li>Support a lot of common ops use cases out of the box.</li>
<li>Easy to automate manual tasks and processes.</li>
<li>Increased productivity and speed up the development cycle.</li>
<li>Help to manage massive infrastructure with less effort.</li>
<li>Efficiency to manage infrastructure and reduce costs.</li>
<li>Scalability and modularity are supported easily.</li>
</ul>
<p>
---
</p>
<p>
Finally, check out this interesting report about <a
href="https://www.datadoghq.com/container-report/">world container
usage</a>.
</p>
Ahmed AbouZaidhttp://www.blogger.com/profile/06013456198042893076noreply@blogger.com0tag:blogger.com,1999:blog-409120552279009989.post-53427750051082330922021-11-01T01:30:00.029+02:002022-07-03T21:21:54.464+02:00Now I'm a Certified Kubernetes Security Specialist + exam tips<p>
On Saturday 30.10.2021 and in less than 24 hours of the exam, I got an email that I passed the CKS exam on the first try and I'm now a Certified Kubernetes Security Specialist. So now I have the 3 Kubernetes certificates (CKA, CKAD, and CKS). ππ
</p>
<div class="separator" style="clear: both;">
<a target="_blank" href="https://blogger.googleusercontent.com/img/a/AVvXsEhmj7piO8wRR7tuSrL7GTsOFbMRquelThvrKZgrsuJupmg5sKKXN9y4NTpxnF_5X5KjNs2CJlaabufVBwmK3c8isYCydYc9DUL291NCgByNO0m4uVcMAGWHkEHIXS-U5SOdtrwRbDaJDBFFuJf8lp5AZ3EsZtMhwfoeNVpuSDwk9Rts21KWcyXVSmdhAg=s1720" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" width="600" data-original-height="1329" data-original-width="1720" src="https://blogger.googleusercontent.com/img/a/AVvXsEhmj7piO8wRR7tuSrL7GTsOFbMRquelThvrKZgrsuJupmg5sKKXN9y4NTpxnF_5X5KjNs2CJlaabufVBwmK3c8isYCydYc9DUL291NCgByNO0m4uVcMAGWHkEHIXS-U5SOdtrwRbDaJDBFFuJf8lp5AZ3EsZtMhwfoeNVpuSDwk9Rts21KWcyXVSmdhAg=s600"/>
</a>
</div>
<p>
So is it now <tt>DevSecOps</tt>? ποΈ Well ... let's take a look on some details about the "why" of getting a certificate.
</p>
<a name='more'></a>
<h2 id="goals-and-motivation">Goals and motivation</h2>
<p>
First of all, I'm not a big fan of just collecting certificates (I just got 7 of them in 10 years ... only 2 of them are MCQ and the reset are hands-on performance based), however, I know how working on a certificate can helps a lot! It can fill in the gaps, boost, and polish your technical skills. For me, each certificate has its own goals and motivation.
</p>
<p>
For example <strong>CKA</strong> (<a target="_blank" href="https://tech.aabouzaid.com/2019/09/now-i-am-a-certified-kubernetes-administrator-plus-10-exam-tips.html">I got it in 2019</a>), even I had a proper Kubernetes production experience, I studied for certificate to make sure I cover different workloads and to made sure that I have wider knowledge about Kubernetes ecosystem.
</p>
<p>
And for <strong>CKAD</strong> (<a target="_blank" href="https://tech.aabouzaid.com/2020/06/now-i-am-a-certified-kubernetes-application-developer-plus-10-exam-tips.html">I got it in 2020</a>), my main goal was to find how to make a smooth Kubernetes transformation in DevOps manner, especially for engineers without Ops background e.g. software engineers (so I actually reviewed CKAD curriculum from different sources to find the best fit for developers from different backgrounds!).
</p>
<p>
Also, <strong>CKS</strong> was the same; it has its own goals and motivation. Most of my security knowledge mainly came from my early career before Kubernetes, so I wanted to extend my knowledge in that domain to apply security best practices in the Kubernetes ecosystem. As a Linux system engineer in an enterprising hosting company, you deal with many clients with different stacks and web applications; security has always been a critical pillar for reliable service. Even though the security knowledge and skills that I learned from that time still help me till now; but after moving to the Kubernetes ecosystem a couple of years ago, I needed to refresh that knowledge in a structured way.
</p>
<p>
TBH, I enjoyed the CKS topics a lot! CKS covers the whole stack, starting with understanding the Kubernetes attack surface, operating system hardening, cluster hardening, supply chain security, service vulnerabilities, monitoring and observability, and many more. The content covers a wide range of practices or, in other words, the necessary topics that should be part of the day-to-day security ecosystem in Kubernetes! Essential practices that everyone should do as well as advanced practices that are needed in some use cases (e.g. critical industries, for example, health and finance).
</p>
<h2 id="resources">Resources</h2>
<p>
This time I just focused on one a main source for study and exam preparation. That was the KodeKloud course <a target="_blank" href="https://kodekloud.com/courses/certified-kubernetes-security-specialist-cks/">Certified Kubernetes Security Specialist (CKS)</a> by Mumshad Mannambeth. Which I recommend it 100% for all Kubernetes certificates. (CKA, CKAD, and CKS).
</p>
<p>
But as usual, don't put all your eggs in one basket! So the second source for review was <a target="_blank" href="https://killer.sh/">Killer.sh</a> (aka KillerShell) which is simply a Kubernetes exam simulator. In fact, it's now officially integrated by Linux Foundation, and you get 2 free simulator sessions when you purchase your exam (for CKA, CKAD, and CKS)!
</p>
<p>
IMO, if you don't want to spend money on the certificate itself, you can just go with the KillerShell exam simulator and you will get almost the same experience!
</p>
<h2 id="tips">Tips</h2>
<p>
So, most of the exam tips from my CKA exam post (<a target="_blank" href="https://tech.aabouzaid.com/2019/09/now-i-am-a-certified-kubernetes-administrator-plus-10-exam-tips.html">10 tips for Certified Kubernetes Administrator exam</a>) are still valid.
</p>
<p>
However, I'd like to mention some points here:
</p>
<ul>
<li>Read the <a target="_blank" href="https://killer.sh/faq">KillerShell FAQ page</a>; it has a lot of helpful info.</li>
<li>KillerShell exam simulator is excellent, but it's harder than the actual exam (that's mentioned in the FAQ). Which TBH I find annoying! I got confused when I was thinking about my exam strategy! (more details about exam strategy in the CKA exam tips post)</li>
<li>Both KodeKloud and KillerShell web terminals are not 100% identical to that's used in the exam Especially the "auto-copy"! It's not only that you probably will be slower in the exam than the pre-exam practices, but also this no auto-copy is confusing!</li>
<li>Don't panic if you didn't do all subtasks within the task! According to KillerShell FAQ: "The Linux Foundation will calculate a score based on successful subtasks. Also considering if some subtasks are harder than others".</li>
<li>Since CKS covers many tools not part of Kubernetes itself (like <a target="_blank" href="https://falco.org/docs/">Falco</a> and <a target="_blank" href="https://github.com/aquasecurity/trivy">Tivy</a>), Bookmark the docs URL, so it's easy to access if needed in the exam (you can open 1 extra tab for official docs).</li>
<li>Get a bigger screen! The laptop or small screen will limit you a lot! (I have 24-Inch and it worked pretty good)</li>
<li>If you are using wireless mouse and keyboard, make sure they are charged.</li>
<li>One of the annoying thing in the exam environment is that no time "counter"! It's just a time "bar" with no accurate indication how much time do you have! And you need to ask the remote exam supervisor via the chat! If it's allowed (I didn't tried that in the exam) probably I'd like to have a system indicator like <a target="_blank" href="https://extensions.gnome.org/extension/3955/kitchen-timer/">Kitchen Timer</a> (since you cannot have physical timer on you desk).</li>
<li><strong>Disable your Ctrl+w shortcut!</strong> This is really important! This shortcut is used almost everywhere! And it closes the tab in Chrome-like browsers (e.g. Chrome, Chromium, Brave, etc.). You will not believe how many times you will hit it unconsciously just by muscle memory! (more details about this in the CKA exam tips post)</li>
</ul>
<p>
Finally, one tip that's not needed anymore is enabling the kubectl <tt>k</tt> shortcut with autocomplete. It's already there now.
</p>
<p>
For me, it was a great and exciting experience and totally worth it on many levels :-)
</p>
Ahmed AbouZaidhttp://www.blogger.com/profile/06013456198042893076noreply@blogger.com0