tag:blogger.com,1999:blog-25039254922900592024-02-07T07:14:03.175+01:0042wim's blog42wimhttp://www.blogger.com/profile/03140786286473099167noreply@blogger.comBlogger22125tag:blogger.com,1999:blog-2503925492290059.post-25329509276941789872020-11-29T18:47:00.000+01:002020-11-29T18:47:24.046+01:00<h1>Using nomad to deploy/manage containers (on a mostly IPv6 network)</h1>
<h2>Overview</h2>
<p>This blog will show an overview of our container deployment using Nomad and IPv6-(only) where possible.</p>
<p>The focus for this setup was to empower developers and keeping things simple for them and us (the admins).<br />
They can configure firewalls, storage, traefik, internet accessibility and more and have an https enabled project running in minutes.</p>
<p>As we run our own physical datacenters we have the luxury to use IPv6 where we want/can and do not need the mess of overlay networks and NAT.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg8YdjWeC-fBmckr1Ab1SqHt3iCtnXoK53FOFlWRSZebZkrpPstM0IQizk3Hx5RbhMamm56b9FgXPV8AIsihd8RLLozyF45WScV76t8yibwbVaTesuxhR0eYHBcjJAVsbF0h7U4jzWqVA/s865/nomad+%25282%2529.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="703" data-original-width="865" height="520" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg8YdjWeC-fBmckr1Ab1SqHt3iCtnXoK53FOFlWRSZebZkrpPstM0IQizk3Hx5RbhMamm56b9FgXPV8AIsihd8RLLozyF45WScV76t8yibwbVaTesuxhR0eYHBcjJAVsbF0h7U4jzWqVA/w640-h520/nomad+%25282%2529.png" width="640" /></a></div><br /><p><br /></p>
<p>Our developers create a nomadjob using a nomadgen.toml file which simplifies the nomad hcl plans (and extends it with features nomad itself cannot do).<br />
You can find an example below. Most of the lines are self explanatory, i've added some comments</p><p><br /></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjk14BbODatMCNUMBzyAI4udxD7NGyA6UBVGOKR7LIP4OwU72SHjLCTHyYBtS7za34t4JCfvl_lq8nIUCGTNoT_LXB5RcH4Ol0qtsCGMojCLqQE_6tIgpptp2j3jj9SEz2vcF8xGM90Tw/s764/toml.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="527" data-original-width="764" height="442" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjk14BbODatMCNUMBzyAI4udxD7NGyA6UBVGOKR7LIP4OwU72SHjLCTHyYBtS7za34t4JCfvl_lq8nIUCGTNoT_LXB5RcH4Ol0qtsCGMojCLqQE_6tIgpptp2j3jj9SEz2vcF8xGM90Tw/w640-h442/toml.PNG" width="640" /></a></div><br /><p>This nomadgen.toml gets checked into gitea, where Jenkins will pick it up and send it to nomadguard.</p>
<p>Nomadguard will:</p>
<ul>
<li>turn the toml back into a complete hcl with all of our infrastructure parameters filled in.</li>
<li>run our nomad validator on it which checks if the user specified correct settings, uses the correct namespace and authorizations etc.</li>
<li>runs nomad modifier on it, which allows us to modify jobs to automatically scale multi-containers over our 2 datacenters, move on specific nodes, limit memory, cpu or other things we need to do without developer interaction.</li>
</ul>
<p>This modified nomad job will then get deployed to the nomad scheduler which will send it to a nomad node that has the resources the developer asked for.</p>
<p>On that nomad node the nomad agent will contact vault to resolve any needed secrets and it will start the docker container. At startup the incoming and outgoing firewall for that container will be configured. As we're using IPv6 our containers are routable and direct accessible, so we need to have a firewall in place for that. We use a modified <a href="https://github.com/gliderlabs/registrator">registrator</a> to add specific ipset entries to allow access to or from the container.</p>
<p>At the same time nomad will also register it's IPv6 addresses in our consul DNS (so traefik can start sending traffic to it).</p>
<p>If the container is running a http/https service this will be automatically exposed on https://p-es-cerebro.cloud.internal.domain (where p is the first letter of the tier (production/quality/test))</p>
<p>All of this infrastructure around the containers is IPv6 only and the containers themselves only allow IPv6 ingress, but they do have dualstack egress as some of the services they need (on the inter- or intranet) are not dualstack yet.</p>
<p>At the edge we have a netscaler that talks dualstack (IPv4/IPv6) to the users and moves to IPv6 only when talking to traefik and containers.</p>
<p>We're using default docker from Centos 7 and every nomad node is getting a IPv6 /80 range routed which it can use to give the containers their ipv6 addresses.</p>
<h2>Extra tooling</h2>
<p>We have some more tooling available for developers as they need to debug their deployments.
This is where we have nomadctl which allows them to ask information about their job, see logs (coming from elasticsearch) and enter containers.</p>
<pre><code class="language-text">$ nomadctl ps cerebro
Exec ID |Job/Task |Node |Uptime |CPU |Mem(max) |Extra
|p-es-cerebro OK | | | | |
c4996e3c72 |p-es-cerebro |p-cloud-dc1-9 |3 days ago |26 |948 MiB(1.2 GiB) |
59702e6d47 |p-es-cerebro |p-cloud-dc2-8 |3 days ago |27 |872 MiB(1.1 GiB) |
</code></pre>
<p>Or exec into a container</p>
<pre><code class="language-bash">$ nomadctl exec c4996e3c72
Welcome wim (ssh cert verified)
welcome to p-es-cerebro on p-cloud-dc1-9
# ss -an | grep 9000
tcp LISTEN 0 100 :::9000 :::*
</code></pre>
<h2>Issues</h2>
<p>Of course there were issues, but not that many ;-)</p>
<ul>
<li>Especially in the early days of our setup we had some IPv6 issues in the hashicorp tools, but as they are opensource it's easy to fix those. (in contrast to hardware vendors where bugs are ignored or takes years to fix ..)</li>
<li>Nomad 0.8 to 0.9 was troublesome because a lot of nomad stuff internally was rewritten and caused some issues in our setup.</li>
</ul>
<p>The main takeaway after 5 years is that the nomad/consul/vault infrastructure is really solid and needs no babysitting.<br />
And yes, IPv6 is (mostly) ready for production!</p><p><br /></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEit_46wVtGMr50H7nl7_qMOt7NfwSNz8DUMj9ClH4RyEKnAsrLeSoMZcouFGl-2XuZsxKahooxY1v7ucxzjUlG8_rA4CRpM8xaf9QdbtrjKgN8Dbg6lMSyMGwAWGChLu9ksRtx4Q9wOZw/s297/ipv6ready.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="170" data-original-width="297" height="129" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEit_46wVtGMr50H7nl7_qMOt7NfwSNz8DUMj9ClH4RyEKnAsrLeSoMZcouFGl-2XuZsxKahooxY1v7ucxzjUlG8_rA4CRpM8xaf9QdbtrjKgN8Dbg6lMSyMGwAWGChLu9ksRtx4Q9wOZw/w225-h129/ipv6ready.png" width="225" /></a></div><br /><p><br /></p>
42wimhttp://www.blogger.com/profile/03140786286473099167noreply@blogger.com0tag:blogger.com,1999:blog-2503925492290059.post-48383540682970385712019-03-30T15:23:00.001+01:002019-03-30T15:30:42.693+01:00Routable IPv6 containers with podman<div class="code-line code-line" data-line="1" style="font-size: 14px; position: relative;">
<div style="text-align: center;">
<h4>
Hacking podman to have "rootless" routable ipv6 containers using a small root daemon.</h4>
<br /></div>
</div>
<div class="code-line code-line" data-line="3" style="font-size: 14px; position: relative;">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhYfddeUpoMzjWo_Aidr0Om-gaM8S3rKjLViTqIaKgzLxDIxcXujT6DyMdaVG3UKnuJxQPzqdN8_o0gz12DVc7-CjVEO_gynlKwLIXL42_xUCCpuWbYgubo5URlNgeC5Msq5IR9DVm7Hw/s1600/v6podman2.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="199" data-original-width="574" height="110" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhYfddeUpoMzjWo_Aidr0Om-gaM8S3rKjLViTqIaKgzLxDIxcXujT6DyMdaVG3UKnuJxQPzqdN8_o0gz12DVc7-CjVEO_gynlKwLIXL42_xUCCpuWbYgubo5URlNgeC5Msq5IR9DVm7Hw/s320/v6podman2.PNG" width="320" /></a></div>
</div>
<div class="code-line code-line" data-line="5" style="font-size: 14px; position: relative;">
<br />
Podman is great, but to have it replace our current docker setup it also needs ipv6 support (which it has using slirp4netns), but this isn't reachable from other containers or outside the host.</div>
<div class="code-line code-line" data-line="7" style="font-size: 14px; position: relative;">
<br />
We don't care about incoming legacy IP (ipv4).<br />
<br /></div>
<br />
<h2 class="code-line code-line" data-line="9" id="what-do-we-want" style="font-weight: normal; position: relative;">
What do we want</h2>
<div class="code-line code-line" data-line="10" style="font-size: 14px; position: relative;">
When a user starts a container, the container should have a routable IPV6 address and register it's name in consul. That way we can have multiple containers talk to eachother, no matter from which host they're started. (and this all needs to work on centos 7.6)</div>
<h2 class="code-line code-line" data-line="13" id="what-do-we-need" style="font-weight: normal; position: relative;">
</h2>
<br />
<h2 class="code-line code-line" data-line="13" id="what-do-we-need" style="font-weight: normal; position: relative;">
What do we need</h2>
<br />
<h3 class="code-line code-line" data-line="14" id="from-podman" style="font-weight: normal; position: relative;">
</h3>
<h3 class="code-line code-line" data-line="14" id="from-podman" style="font-weight: normal; position: relative;">
From podman</h3>
<ul style="font-size: 14px;">
<li class="code-line code-line" data-line="15" style="position: relative;">The id of the user that started the container</li>
<li class="code-line code-line" data-line="16" style="position: relative;">The PID of the container so we can use this to enter the same network namespace</li>
<li class="code-line code-line" data-line="17" style="position: relative;">The name of the container so we can register this in consul</li>
<li class="code-line code-line" data-line="18" style="position: relative;">A way to talk to v6pod</li>
</ul>
<br />
<h3 class="code-line code-line" data-line="20" id="external-to-podman-v6pod-will-handle-this" style="font-weight: normal; position: relative;">
</h3>
<h3 class="code-line code-line" data-line="20" id="external-to-podman-v6pod-will-handle-this" style="font-weight: normal; position: relative;">
External to podman (v6pod will handle this)</h3>
<ul style="font-size: 14px;">
<li class="code-line code-line" data-line="21" style="position: relative;">Be compatible with our current docker IPv6 ranges (/80)</li>
<li class="code-line code-line" data-line="22" style="position: relative;">Creating a bridge and add the gateway IPv6 address to it (::1)</li>
<li class="code-line code-line" data-line="23" style="position: relative;">Creating a veth pair</li>
<li class="code-line code-line" data-line="24" style="position: relative;">Generate a (dynamic) IPv6 address in the /80 range and add to veth that will come into the container</li>
<li class="code-line code-line" data-line="25" style="position: relative;">Add one of the veths to the bridge, the other in the network namespace of the user</li>
<li class="code-line code-line" data-line="26" style="position: relative;">Add a default IPv6 route to the bridge</li>
<li class="code-line code-line" data-line="27" style="position: relative;">Register the name of the container to the generated IPv6 address in consul</li>
<li class="code-line code-line" data-line="28" style="position: relative;">Deregister the name when the container stops</li>
</ul>
<h2 class="code-line code-line" data-line="30" id="modifying-libpodpodman" style="font-weight: normal; position: relative;">
</h2>
<h2 class="code-line code-line" data-line="30" id="modifying-libpodpodman" style="font-weight: normal; position: relative;">
</h2>
<h2 class="code-line code-line" data-line="30" id="modifying-libpodpodman" style="font-weight: normal; position: relative;">
Modifying libpod/podman</h2>
<div class="code-line code-line" data-line="31" style="font-size: 14px; position: relative;">
Code here: <a href="https://github.com/42wim/libpod/tree/rootlessv6" style="text-decoration-line: none;" title="https://github.com/42wim/libpod/tree/rootlessv6">https://github.com/42wim/libpod/tree/rootlessv6</a><br />
<br /></div>
<h3 class="code-line code-line" data-line="33" id="1-executing-user" style="position: relative;">
1) Executing user</h3>
<div class="code-line code-line" data-line="34" style="font-size: 14px; position: relative;">
Some investigation into what happens when running <code style="color: var(--vscode-textPreformat-foreground); font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px;">podman run</code> (rootless)<br />
Podman tries to create a user namespace, join this and become root in it and re-executes itself in that namespace.</div>
<div class="code-line code-line" data-line="37" style="font-size: 14px; position: relative;">
We need to save the id of the executing user somewhere, the environment looks a good place.<br />
So we create a <code style="color: var(--vscode-textPreformat-foreground); font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px;">v6pod_user</code> variable which contains the userid of the user running podman.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiyvAOsPLZS_b5CbtRUmtbik4eYW_PTZljTaIMCvj5LN4iCmVMYYhYl39-2qiqDzsFxp2y37T_r2MSx0OiXQnuHndxyAgOb3BvWkTTbvF-ZZaQgK74SF23oNZIXjgG_zCyTcSHobWdL6g/s1600/podman1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="772" data-original-width="726" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiyvAOsPLZS_b5CbtRUmtbik4eYW_PTZljTaIMCvj5LN4iCmVMYYhYl39-2qiqDzsFxp2y37T_r2MSx0OiXQnuHndxyAgOb3BvWkTTbvF-ZZaQgK74SF23oNZIXjgG_zCyTcSHobWdL6g/s400/podman1.png" width="375" /></a></div>
<br /></div>
<h3 class="code-line code-line" data-line="42" id="2-pid-of-the-container" style="position: relative;">
2) Pid of the container</h3>
<div class="code-line code-line" data-line="43" style="position: relative;">
<div style="font-size: 14px;">
This could be added somewhere better probably, but I kept it in the same method.</div>
<div style="font-size: 14px;">
We don't have access to the container PID yet there because it hasn't started, but we already have the container ID that will be used.</div>
<div style="font-size: 14px;">
So I save the container ID in the <code style="color: var(--vscode-textPreformat-foreground); font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px;">v6pod_id</code> environment variable.</div>
<div style="font-size: 14px;">
v6pod will then look into <code style="color: var(--vscode-textPreformat-foreground); font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px;">/run/user/" + userID + "/runc/" + containerID + "/state.json</code> file to get the PID</div>
<div style="font-size: 14px;">
<br /></div>
<span style="font-size: 14px;">Overview below of what happens when podman runs executes again but now in the user namespace.</span><br />
<div style="font-size: 14px;">
<br /></div>
<div style="font-size: 14px;">
<br /></div>
<div class="separator" style="clear: both; font-size: 14px; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjlMBdop47rr0pvHvGVlxpO2uRC-Pi0XS1tnZLU-xDdx8r36Z9Je6e_phyphenhyphen1bXurqmG6-1yDat13qFDHJvOJh2Yb-D0vbFn3_J_hGu6Uz5hKDXJwhNOHGdPkMlKLRUbUfpKKTvYP33MK4A/s1600/podman2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="893" data-original-width="1347" height="424" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjlMBdop47rr0pvHvGVlxpO2uRC-Pi0XS1tnZLU-xDdx8r36Z9Je6e_phyphenhyphen1bXurqmG6-1yDat13qFDHJvOJh2Yb-D0vbFn3_J_hGu6Uz5hKDXJwhNOHGdPkMlKLRUbUfpKKTvYP33MK4A/s640/podman2.png" width="640" /></a></div>
<div style="font-size: 14px;">
<br /></div>
</div>
<div class="code-line code-line" data-line="48" style="font-size: 14px; position: relative;">
<br /></div>
<h3 class="code-line code-line" data-line="50" id="3-name-of-the-container" style="position: relative;">
3) Name of the container</h3>
<div class="code-line code-line" data-line="51" style="font-size: 14px; position: relative;">
We could've set this using another variable, but to be more flexible (maybe we need more information about the container in the future) we choose not to.</div>
<div class="code-line code-line" data-line="53" style="font-size: 14px; position: relative;">
Podman saves it create-config in the path: <code style="color: var(--vscode-textPreformat-foreground); font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px;">"/run/user/" + userID + "/libpod/tmp/socket/" + containerID + "/artifacts/create-config"</code> which contains a lot information and also the container name.<br />
<br /></div>
<h3 class="code-line code-line" data-line="55" id="4-talk-with-v6pod" style="position: relative;">
4) Talk with v6pod</h3>
<div class="code-line code-line" data-line="56" style="position: relative;">
<div style="font-size: 14px;">
Here we just hijack the <code style="color: var(--vscode-textPreformat-foreground); font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px;">slirp4netns</code> command (which enables userspace networking) and replace it with a <code style="color: var(--vscode-textPreformat-foreground); font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px;">v6pod-slirp4netns</code> bash file which contains:</div>
<div style="font-size: 14px;">
<br /></div>
<span style="font-family: "verdana" , sans-serif; font-size: x-small;">#!/bin/bash</span><br />
<span style="font-family: "verdana" , sans-serif; font-size: x-small;">/bin/curl -XPOST -d "user=$v6pod_user&id=$v6pod_id" http://localhost:6781/api/activate</span><br />
<span style="font-family: "verdana" , sans-serif; font-size: x-small;">/bin/slirp4netns "$@"</span><br />
<span style="font-family: "verdana" , sans-serif; font-size: x-small;">/bin/curl -XPOST -d "user=$v6pod_user&id=$v6pod_id" http://localhost:6781/api/deactivate</span><br />
<div style="font-size: 14px;">
<br /></div>
<div style="font-size: 14px;">
<br /></div>
<div style="font-size: 14px;">
So we use the variables we set above to do all the networking stuff we need, then let slirp4netns do it setup so we still have outgoing IPv4 besides IPv6. When the container ends, slirp4netns exists and we do a deregistration.</div>
</div>
<h2 class="code-line code-line" data-line="66" id="modifying-slirp4netns" style="font-weight: normal; position: relative;">
</h2>
<h2 class="code-line code-line" data-line="66" id="modifying-slirp4netns" style="font-weight: normal; position: relative;">
</h2>
<h2 class="code-line code-line" data-line="66" id="modifying-slirp4netns" style="font-weight: normal; position: relative;">
<br /></h2>
<h2 class="code-line code-line" data-line="66" id="modifying-slirp4netns" style="font-weight: normal; position: relative;">
Modifying slirp4netns</h2>
<div class="code-line code-line" data-line="67" style="font-size: 14px; position: relative;">
slirp4netns sets an ipv4 and ipv6 address and gateways. We do the IPv6 part now, so this needs to be disabled in slirp4netns.<br />
<br /></div>
<div class="code-line code-line" data-line="70" style="font-size: 14px; position: relative;">
<span style="color: black;">source <a href="https://github.com/42wim/slirp4netns/commits/v6rootless" style="text-decoration-line: none;" title="https://github.com/42wim/slirp4netns/commits/v6rootless">https://github.com/42wim/slirp4netns/commits/v6rootless</a></span><br />
<br /></div>
<h2 class="code-line code-line" data-line="72" id="v6pod" style="font-weight: normal; position: relative;">
v6pod</h2>
<div class="code-line code-line" data-line="73" style="font-size: 14px; position: relative;">
v6pod is a go daemon with a rest interface that has a /activate and /deactivate entrypoint.</div>
<div class="code-line code-line" data-line="75" style="font-size: 14px; position: relative;">
It implements the requirements of above:</div>
<ul style="font-size: 14px;">
<li class="code-line code-line" data-line="76" style="position: relative;">Be compatible with our current docker IPv6 ranges (/80)</li>
<li class="code-line code-line" data-line="77" style="position: relative;">Creating a bridge and add the gateway IPv6 address to it (::1)</li>
<li class="code-line code-line" data-line="78" style="position: relative;">Creating a veth pair</li>
<li class="code-line code-line" data-line="79" style="position: relative;">Generate a (dynamic) IPv6 address in the /80 range and add to veth that will come into the container</li>
<li class="code-line code-line" data-line="80" style="position: relative;">Add one of the veths to the bridge, the other in the network namespace of the user</li>
<li class="code-line code-line" data-line="81" style="position: relative;">Add a default IPv6 route to the bridge</li>
<li class="code-line code-line" data-line="82" style="position: relative;">Register the name of the container to the generated IPv6 address in consul</li>
<li class="code-line code-line" data-line="83" style="position: relative;">Deregister the name when the container stops</li>
</ul>
<div class="code-line code-line" data-line="85" style="font-size: 14px; position: relative;">
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhkfZSKOIvEab_6QRi-OUT8aBW_xSDy-LT9ZsnRd1lyqO2rmJABp50mgZrBEqIngeN3RdGWayfPuIIQ8ncXShuVi3nVPmS3ZT-lku25MnBSJ_b1wfJn6qvx61carFrzEHdGdiVZNrObqg/s1600/v6pod2.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="939" data-original-width="695" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhkfZSKOIvEab_6QRi-OUT8aBW_xSDy-LT9ZsnRd1lyqO2rmJABp50mgZrBEqIngeN3RdGWayfPuIIQ8ncXShuVi3nVPmS3ZT-lku25MnBSJ_b1wfJn6qvx61carFrzEHdGdiVZNrObqg/s640/v6pod2.PNG" width="472" /></a></div>
<br />
<br />
Soon at <a href="https://github.com/42wim/v6pod">https://github.com/42wim/v6pod</a><br />
<br /></div>
42wimhttp://www.blogger.com/profile/03140786286473099167noreply@blogger.com0tag:blogger.com,1999:blog-2503925492290059.post-61041558976626913822018-10-20T22:22:00.002+02:002018-10-20T22:27:53.777+02:00Buildah inside a centos 7.5 docker container on a centos 7.5 host<h1 class="code-line" data-line="0" id="buildah-in-a-centos-75-docker-container-on-a-centos-75-host" style="border-bottom: 1px solid rgba(255, 255, 255, 0.18); border-left-color: rgba(255, 255, 255, 0.18); border-right-color: rgba(255, 255, 255, 0.18); border-top-color: rgba(255, 255, 255, 0.18); font-family: -apple-system, BlinkMacSystemFont, "Segoe WPC", "Segoe UI", HelveticaNeue-Light, Ubuntu, "Droid Sans", sans-serif; font-weight: normal; line-height: 1.2; padding-bottom: 0.3em; position: relative;">
</h1>
<h1 class="code-line" data-line="0" style="border-bottom: 1px solid rgba(255, 255, 255, 0.18); border-left-color: rgba(255, 255, 255, 0.18); border-right-color: rgba(255, 255, 255, 0.18); border-top-color: rgba(255, 255, 255, 0.18); font-family: -apple-system, BlinkMacSystemFont, "Segoe WPC", "Segoe UI", HelveticaNeue-Light, Ubuntu, "Droid Sans", sans-serif; font-weight: normal; line-height: 1.2; padding-bottom: 0.3em; position: relative;">
<div class="code-line" data-line="2" style="font-size: 14px; position: relative;">
Our current solution uses Jenkins to start a Nomad job which starts a (unprivileged) docker container in which a developers Dockerfile is being build (as root) using the docker on the host.</div>
<div class="code-line" data-line="4" style="font-size: 14px; position: relative;">
<br /></div>
<div class="code-line" data-line="4" style="font-size: 14px; position: relative;">
The goal is to replace the docker build in the container by buildah so that we don't need to make the docker on the host available inside the container.</div>
<div class="code-line" data-line="6" style="font-size: 14px; position: relative;">
<br /></div>
<div class="code-line" data-line="6" style="font-size: 14px; position: relative;">
The path to this wasn't as straightforward unfortunately, a lot of yaks needed shaving.</div>
<div class="code-line" data-line="6" style="font-size: 14px; position: relative;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3uCtJ2dVOU6SMXDAL4DpZU2V4K_ibDa0X4hT7UBgw6qxpvImyFvcxe9-boOf2xOBkwPAOUzb8cWYPKmzaJuCS3FWd117XRwoqvg-c7ZSexQpYnTRm_r0MMfjc8A94d0g_3pjtUfxCyA/s1600/2018-10-20-17-21-41.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="502" data-original-width="1010" height="316" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3uCtJ2dVOU6SMXDAL4DpZU2V4K_ibDa0X4hT7UBgw6qxpvImyFvcxe9-boOf2xOBkwPAOUzb8cWYPKmzaJuCS3FWd117XRwoqvg-c7ZSexQpYnTRm_r0MMfjc8A94d0g_3pjtUfxCyA/s640/2018-10-20-17-21-41.png" width="640" /></a></div>
<div class="code-line" data-line="6" style="font-size: 14px; position: relative;">
</div>
</h1>
<h1 class="code-line" data-line="10" id="start-of-the-journey" style="border-bottom: 1px solid rgba(255, 255, 255, 0.18); border-left-color: rgba(255, 255, 255, 0.18); border-right-color: rgba(255, 255, 255, 0.18); border-top-color: rgba(255, 255, 255, 0.18); font-family: -apple-system, BlinkMacSystemFont, "Segoe WPC", "Segoe UI", HelveticaNeue-Light, Ubuntu, "Droid Sans", sans-serif; font-weight: normal; line-height: 1.2; padding-bottom: 0.3em; position: relative;">
Start of the journey</h1>
<h1 class="code-line" data-line="0" style="border-bottom: 1px solid rgba(255, 255, 255, 0.18); border-left-color: rgba(255, 255, 255, 0.18); border-right-color: rgba(255, 255, 255, 0.18); border-top-color: rgba(255, 255, 255, 0.18); font-family: -apple-system, BlinkMacSystemFont, "Segoe WPC", "Segoe UI", HelveticaNeue-Light, Ubuntu, "Droid Sans", sans-serif; font-weight: normal; line-height: 1.2; padding-bottom: 0.3em; position: relative;">
<div class="code-line" data-line="12" style="font-size: 14px; position: relative;">
We're starting with a basic container where we install buildah in</div>
<pre style="background-color: rgba(10, 10, 10, 0.4); border-radius: 3px; font-size: 14px; overflow: auto; padding: 16px; white-space: pre-wrap;"><code class="code-line" data-line="14" style="font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px; position: relative;"><code style="font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px;"># docker run --rm -ti centos:7 /bin/bash
[root@7387c68139dd /]# yum -y install buildah
</code></code></pre>
<div class="code-line" data-line="19" style="font-size: 14px; position: relative;">
And a very simple Dockerfile</div>
<pre style="background-color: rgba(10, 10, 10, 0.4); border-radius: 3px; font-size: 14px; overflow: auto; padding: 16px; white-space: pre-wrap;"><code class="code-line" data-line="21" style="font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px; position: relative;"><code style="font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px;">FROM centos:7
RUN uptime
</code></code></pre>
</h1>
<h1 class="code-line" data-line="26" id="yak-1---overlay-problems" style="border-bottom: 1px solid rgba(255, 255, 255, 0.18); border-left-color: rgba(255, 255, 255, 0.18); border-right-color: rgba(255, 255, 255, 0.18); border-top-color: rgba(255, 255, 255, 0.18); font-family: -apple-system, BlinkMacSystemFont, "Segoe WPC", "Segoe UI", HelveticaNeue-Light, Ubuntu, "Droid Sans", sans-serif; font-weight: normal; line-height: 1.2; padding-bottom: 0.3em; position: relative;">
Yak 1 - overlay problems</h1>
<h1 class="code-line" data-line="0" style="border-bottom: 1px solid rgba(255, 255, 255, 0.18); border-left-color: rgba(255, 255, 255, 0.18); border-right-color: rgba(255, 255, 255, 0.18); border-top-color: rgba(255, 255, 255, 0.18); font-family: -apple-system, BlinkMacSystemFont, "Segoe WPC", "Segoe UI", HelveticaNeue-Light, Ubuntu, "Droid Sans", sans-serif; font-weight: normal; line-height: 1.2; padding-bottom: 0.3em; position: relative;">
<div class="code-line" data-line="27" style="font-size: 14px; position: relative;">
Out of the box running buildah in the container will give an overlay error.</div>
<pre style="background-color: rgba(10, 10, 10, 0.4); border-radius: 3px; font-size: 14px; overflow: auto; padding: 16px; white-space: pre-wrap;"><code class="code-line" data-line="29" style="font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px; position: relative;"><code style="font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px;"># buildah bud -t test .
ERRO[0000] 'overlay' is not supported over extfs at "/var/lib/containers/storage/overlay"
ERRO[0000] 'overlay' is not supported over extfs at "/var/lib/containers/storage/overlay"
kernel does not support overlay fs: 'overlay' is not supported over extfs at "/var/lib/containers/storage/overlay": backing file system is unsupported for this graph driver
kernel does not support overlay fs: 'overlay' is not supported over extfs at "/var/lib/containers/storage/overlay": backing file system is unsupported for this graph driver
</code></code></pre>
<div class="code-line" data-line="37" style="font-size: 14px; position: relative;">
<em>Spoiler</em>: The real reason this doesn't work is because it tries to do a <code style="font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px;">mount</code> call, which can only be done with the <code style="font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px;">SYS_ADMIN</code> capability (or in a privileged container).</div>
<div class="code-line" data-line="37" style="font-size: 14px; position: relative;">
<br /></div>
<div class="code-line" data-line="39" style="font-size: 14px; position: relative;">
Using <code style="font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px;">--storage-driver vfs</code> fixed this problem.</div>
<div class="code-line" data-line="41" style="font-size: 14px; position: relative;">
<br /></div>
<div class="code-line" data-line="41" style="font-size: 14px; position: relative;">
On to the next one.</div>
</h1>
<h1 class="code-line" data-line="43" id="yak-2---mount-namespace-error-aka-unshareclonenewns-permission-aka-the-wrong-yak" style="border-bottom: 1px solid rgba(255, 255, 255, 0.18); border-left-color: rgba(255, 255, 255, 0.18); border-right-color: rgba(255, 255, 255, 0.18); border-top-color: rgba(255, 255, 255, 0.18); font-family: -apple-system, BlinkMacSystemFont, "Segoe WPC", "Segoe UI", HelveticaNeue-Light, Ubuntu, "Droid Sans", sans-serif; font-weight: normal; line-height: 1.2; padding-bottom: 0.3em; position: relative;">
Yak 2 - mount namespace error aka unshare(CLONE_NEWNS) permission aka the wrong yak</h1>
<h1 class="code-line" data-line="0" style="border-bottom: 1px solid rgba(255, 255, 255, 0.18); border-left-color: rgba(255, 255, 255, 0.18); border-right-color: rgba(255, 255, 255, 0.18); border-top-color: rgba(255, 255, 255, 0.18); font-family: -apple-system, BlinkMacSystemFont, "Segoe WPC", "Segoe UI", HelveticaNeue-Light, Ubuntu, "Droid Sans", sans-serif; font-weight: normal; line-height: 1.2; padding-bottom: 0.3em; position: relative;">
<div class="code-line" data-line="45" style="font-size: 14px; position: relative;">
<em>Spoiler</em>: this yak is a red herring</div>
<pre style="background-color: rgba(10, 10, 10, 0.4); border-radius: 3px; font-size: 14px; overflow: auto; padding: 16px; white-space: pre-wrap;"><code class="code-line" data-line="47" style="font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px; position: relative;"><code style="font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px;"># buildah --storage-driver vfs bud -t test .
STEP 1: FROM centos:7
Getting image source signatures
Copying blob sha256:aeb7866da422acc7e93dcf7323f38d7646f6269af33bcdb6647f2094fc4b3bf7
71.24 MiB / 71.24 MiB [====================================================] 4s
Copying config sha256:75835a67d1341bdc7f4cc4ed9fa1631a7d7b6998e9327272afea342d90c4ab6d
2.13 KiB / 2.13 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
STEP 2: RUN uptime
error running container: error creating new mount namespace for [/bin/sh -c uptime]: operation not permitted
error building at step {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] Command:run Args:[uptime] Flags:[] Attrs:map[] Message:RUN uptime Original:RUN uptime}: exit status 1
</code></code></pre>
<div class="code-line" data-line="62" style="font-size: 14px; position: relative;">
strace to the rescue</div>
<pre style="background-color: rgba(10, 10, 10, 0.4); border-radius: 3px; font-size: 14px; overflow: auto; padding: 16px; white-space: pre-wrap;"><code class="code-line" data-line="64" style="font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px; position: relative;"><code style="font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px;">unshare(CLONE_NEWNS) = -1 EPERM (Operation not permitted)
</code></code></pre>
<div class="code-line" data-line="68" style="font-size: 14px; position: relative;">
After some googling I found that centos/rhel kernels have user namespace disabled by default and need to have a kernel parameter set to get this working.</div>
<div class="code-line" data-line="70" style="font-size: 14px; position: relative;">
We can enable this by running on the host</div>
<pre style="background-color: rgba(10, 10, 10, 0.4); border-radius: 3px; font-size: 14px; overflow: auto; padding: 16px; white-space: pre-wrap;"><code class="code-line" data-line="71" style="font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px; position: relative;"><code style="font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px;">sudo grubby --args="namespace.unpriv_enable=1 user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"
</code></code></pre>
<div class="code-line" data-line="74" style="font-size: 14px; position: relative;">
And also set the maximum number of user namespaces that any user in the current user namespace may create by running</div>
<pre style="background-color: rgba(10, 10, 10, 0.4); border-radius: 3px; font-size: 14px; overflow: auto; padding: 16px; white-space: pre-wrap;"><code class="code-line" data-line="76" style="font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px; position: relative;"><code style="font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px;">echo "user.max_user_namespaces=15000" >> /etc/sysctl.conf
</code></code></pre>
<div class="code-line" data-line="80" style="font-size: 14px; position: relative;">
Now we can reboot the server</div>
<div class="code-line" data-line="82" style="font-size: 14px; position: relative;">
And come to the conclusion that it still doesn't work.</div>
</h1>
<h1 class="code-line" data-line="84" id="yak-3---outdated-buildah-version" style="border-bottom: 1px solid rgba(255, 255, 255, 0.18); border-left-color: rgba(255, 255, 255, 0.18); border-right-color: rgba(255, 255, 255, 0.18); border-top-color: rgba(255, 255, 255, 0.18); font-family: -apple-system, BlinkMacSystemFont, "Segoe WPC", "Segoe UI", HelveticaNeue-Light, Ubuntu, "Droid Sans", sans-serif; font-weight: normal; line-height: 1.2; padding-bottom: 0.3em; position: relative;">
Yak 3 - outdated buildah version</h1>
<h1 class="code-line" data-line="0" style="border-bottom: 1px solid rgba(255, 255, 255, 0.18); border-left-color: rgba(255, 255, 255, 0.18); border-right-color: rgba(255, 255, 255, 0.18); border-top-color: rgba(255, 255, 255, 0.18); font-family: -apple-system, BlinkMacSystemFont, "Segoe WPC", "Segoe UI", HelveticaNeue-Light, Ubuntu, "Droid Sans", sans-serif; font-weight: normal; line-height: 1.2; padding-bottom: 0.3em; position: relative;">
<div class="code-line" data-line="86" style="font-size: 14px; position: relative;">
Thanks to the #buildah channel on freenode, I found out that the problem of yak 2 was actually an outdated buildah version.<br />
Centos has only a buildah 1.2 rpm, but 1.4 or higher was needed so I'd have to build my own.</div>
<div class="code-line" data-line="86" style="font-size: 14px; position: relative;">
<br /></div>
<div class="code-line" data-line="89" style="font-size: 14px; position: relative;">
You can have this pleasure too with the following script containing a modified RPM spec.</div>
<div class="code-line" data-line="89" style="font-size: 14px; position: relative;">
<br /></div>
<div class="code-line" data-line="91" style="font-size: 14px; position: relative;">
Run a new centos:7 container</div>
<pre style="background-color: rgba(10, 10, 10, 0.4); border-radius: 3px; font-size: 14px; overflow: auto; padding: 16px; white-space: pre-wrap;"><code class="code-line" data-line="93" style="font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px; position: relative;"><code style="font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px;"># docker run -ti -v /tmp:/tmp centos:7 /bin/bash
</code></code></pre>
<div class="code-line" data-line="96" style="font-size: 14px; position: relative;">
and run following commands in the container:</div>
<pre style="background-color: rgba(10, 10, 10, 0.4); border-radius: 3px; font-size: 14px; overflow: auto; padding: 16px; white-space: pre-wrap;"><code class="code-line" data-line="98" style="font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px; position: relative;"><code style="font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px;">yum -y group install development
yum -y install wget
cd /root/rpmbuild/SOURCES
wget "https://github.com/containers/buildah/tarball/608fa843cce45e7ee58ccb71a90297b645a984d3" -O buildah-608fa84.tar.gz
tar zxvf buildah-608fa84.tar.gz
mv containers-buildah-608fa84 buildah-608fa843cce45e7ee58ccb71a90297b645a984d3
tar zcvf buildah-608fa84.tar.gz buildah-608fa843cce45e7ee58ccb71a90297b645a984d3
rm -rf buildah-608fa843cce45e7ee58ccb71a90297b645a984d3
cd ../SPECS
wget https://gist.githubusercontent.com/42wim/848fba2ed2d64d457f56eeebef0e85a2/raw/bb3ad3c524529ed921626fb077b8ff78a56783fc/buildah.spec -O buildah.spec
yum-builddep -y buildah.spec
rpmbuild -ba buildah.spec
</code></code></pre>
<div class="code-line" data-line="113" style="font-size: 14px; position: relative;">
This will give you your RPMs</div>
<pre style="background-color: rgba(10, 10, 10, 0.4); border-radius: 3px; font-size: 14px; overflow: auto; padding: 16px; white-space: pre-wrap;"><code class="code-line" data-line="115" style="font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px; position: relative;"><code style="font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px;">Wrote: /root/rpmbuild/SRPMS/buildah-1.4-1.git608fa84.el7.centos.src.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/buildah-1.4-1.git608fa84.el7.centos.x86_64.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/buildah-debuginfo-1.4-1.git608fa84.el7.centos.x86_64.rpm
</code></code></pre>
</h1>
<h1 class="code-line" data-line="121" id="yak-4---proc-mount-error" style="border-bottom: 1px solid rgba(255, 255, 255, 0.18); border-left-color: rgba(255, 255, 255, 0.18); border-right-color: rgba(255, 255, 255, 0.18); border-top-color: rgba(255, 255, 255, 0.18); font-family: -apple-system, BlinkMacSystemFont, "Segoe WPC", "Segoe UI", HelveticaNeue-Light, Ubuntu, "Droid Sans", sans-serif; font-weight: normal; line-height: 1.2; padding-bottom: 0.3em; position: relative;">
Yak 4 - proc mount error</h1>
<h1 class="code-line" data-line="0" style="border-bottom: 1px solid rgba(255, 255, 255, 0.18); border-left-color: rgba(255, 255, 255, 0.18); border-right-color: rgba(255, 255, 255, 0.18); border-top-color: rgba(255, 255, 255, 0.18); font-family: -apple-system, BlinkMacSystemFont, "Segoe WPC", "Segoe UI", HelveticaNeue-Light, Ubuntu, "Droid Sans", sans-serif; font-weight: normal; line-height: 1.2; padding-bottom: 0.3em; position: relative;">
<div class="code-line" data-line="122" style="font-size: 14px; position: relative;">
Progress, a new error when running buildah 1.4!</div>
<pre style="background-color: rgba(10, 10, 10, 0.4); border-radius: 3px; font-size: 14px; overflow: auto; padding: 16px; white-space: pre-wrap;"><code class="code-line" data-line="124" style="font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px; position: relative;"><code style="font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px;"># buildah --storage-driver vfs bud -t test .
STEP 1: FROM centos:7
Getting image source signatures
Copying blob sha256:205941c9c2d103bcdff0bc72d8836e0ffc4573ec0e6e524ec1a59606062a289f
71.25 MiB / 71.25 MiB [====================================================] 4s
Copying config sha256:e26dc8af6a3b1856b9f4a893d5b51855c02dfe3b9cec58a4e55002036528c669
2.14 KiB / 2.14 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
STEP 2: RUN uptime
container_linux.go:336: starting container process caused "process_linux.go:399: container init caused \"rootfs_linux.go:58: mounting \\\"/proc\\\" to rootfs \\\"/tmp/buildah596035765/mnt/rootfs\\\" at \\\"/proc\\\" caused \\\"operation not permitted\\\"\""
error running container: error creating container for [/bin/sh -c uptime]: : exit status 1
error building at step {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] Command:run Args:[uptime] Flags:[] Attrs:map[] Message:RUN uptime Original:RUN uptime}: exit status 1
ERRO[0012] exit status 1
</code></code></pre>
<div class="code-line" data-line="141" style="font-size: 14px; position: relative;">
Again thanks to #buildah channel, I found out that running <code style="font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px;">--isolation chroot</code> would solve it.</div>
</h1>
<h1 class="code-line" data-line="143" id="victory" style="border-bottom: 1px solid rgba(255, 255, 255, 0.18); border-left-color: rgba(255, 255, 255, 0.18); border-right-color: rgba(255, 255, 255, 0.18); border-top-color: rgba(255, 255, 255, 0.18); font-family: -apple-system, BlinkMacSystemFont, "Segoe WPC", "Segoe UI", HelveticaNeue-Light, Ubuntu, "Droid Sans", sans-serif; font-weight: normal; line-height: 1.2; padding-bottom: 0.3em; position: relative;">
Victory!</h1>
<h1 class="code-line" data-line="0" style="border-bottom: 1px solid rgba(255, 255, 255, 0.18); border-left-color: rgba(255, 255, 255, 0.18); border-right-color: rgba(255, 255, 255, 0.18); border-top-color: rgba(255, 255, 255, 0.18); font-family: -apple-system, BlinkMacSystemFont, "Segoe WPC", "Segoe UI", HelveticaNeue-Light, Ubuntu, "Droid Sans", sans-serif; font-weight: normal; line-height: 1.2; padding-bottom: 0.3em; position: relative;">
<div class="code-line" data-line="145" style="font-size: 14px; position: relative;">
Finally it works, we have an image created by buildah running in an unprivileged container.</div>
<pre style="background-color: rgba(10, 10, 10, 0.4); border-radius: 3px; font-size: 14px; overflow: auto; padding: 16px; white-space: pre-wrap;"><code class="code-line" data-line="147" style="font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px; position: relative;"><code style="font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px;"># buildah --storage-driver vfs bud --isolation chroot -t test .
STEP 1: FROM centos:7
STEP 2: RUN uptime
21:30:55 up 32 min, 0 users, load average: 0.39, 0.12, 0.08
STEP 3: COMMIT containers-storage:[vfs@/var/lib/containers/storage+/var/run/containers/storage]localhost/test:latest
Getting image source signatures
Skipping fetch of repeat blob sha256:f972d139738dfcd1519fd2461815651336ee25a8b54c358834c50af094bb262f
Skipping fetch of repeat blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
Copying config sha256:26e3b2177f9e9db1bdc8f49083d09dbb980a99ed4e606f4dc45b79ca865588ce
1.17 KiB / 1.17 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
--> 26e3b2177f9e9db1bdc8f49083d09dbb980a99ed4e606f4dc45b79ca865588ce
</code></code></pre>
<div class="code-line" data-line="163" style="font-size: 14px; position: relative;">
But after testing a new yak appears.</div>
</h1>
<h1 class="code-line" data-line="165" id="yak-5---a-lot-of-diskspace" style="border-bottom: 1px solid rgba(255, 255, 255, 0.18); border-left-color: rgba(255, 255, 255, 0.18); border-right-color: rgba(255, 255, 255, 0.18); border-top-color: rgba(255, 255, 255, 0.18); font-family: -apple-system, BlinkMacSystemFont, "Segoe WPC", "Segoe UI", HelveticaNeue-Light, Ubuntu, "Droid Sans", sans-serif; font-weight: normal; line-height: 1.2; padding-bottom: 0.3em; position: relative;">
Yak 5 - a lot of diskspace</h1>
<h1 class="code-line" data-line="0" style="border-bottom: 1px solid rgba(255, 255, 255, 0.18); border-left-color: rgba(255, 255, 255, 0.18); border-right-color: rgba(255, 255, 255, 0.18); border-top-color: rgba(255, 255, 255, 0.18); font-family: -apple-system, BlinkMacSystemFont, "Segoe WPC", "Segoe UI", HelveticaNeue-Light, Ubuntu, "Droid Sans", sans-serif; font-weight: normal; line-height: 1.2; padding-bottom: 0.3em; position: relative;">
<div class="code-line" data-line="167" style="font-size: 14px; position: relative;">
This one is related to yak 1, because we're using the <code style="font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px;">vfs</code> storage driver, it uses the disk not very space efficient (according to <a href="https://docs.docker.com/storage/storagedriver/vfs-driver/" style="text-decoration-line: none;" title="https://docs.docker.com/storage/storagedriver/vfs-driver/">https://docs.docker.com/storage/storagedriver/vfs-driver/</a>) a more complicated docker build uses gigabytes of disk when using the <code style="font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px;">vfs</code> storage driver compared to the <code style="font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px;">overlay</code> driver.</div>
<div class="code-line" data-line="167" style="font-size: 14px; position: relative;">
<br /></div>
<div class="code-line" data-line="169" style="font-size: 14px; position: relative;">
To run with the overlay driver we need access to the mount call which means we have to run our docker container with <code style="font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px;">CAP_SYS_ADMIN</code> which is unfortunate.</div>
<div class="code-line" data-line="169" style="font-size: 14px; position: relative;">
<br /></div>
<pre style="background-color: rgba(10, 10, 10, 0.4); border-radius: 3px; font-size: 14px; overflow: auto; padding: 16px; white-space: pre-wrap;"><code class="code-line" data-line="171" style="font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px; position: relative;"><code style="font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px;"># docker run --rm --add-cap SYS_ADMIN -ti centos:7 /bin/bash
</code></code></pre>
</h1>
<h1 class="code-line" data-line="175" id="conclusion" style="border-bottom: 1px solid rgba(255, 255, 255, 0.18); border-left-color: rgba(255, 255, 255, 0.18); border-right-color: rgba(255, 255, 255, 0.18); border-top-color: rgba(255, 255, 255, 0.18); font-family: -apple-system, BlinkMacSystemFont, "Segoe WPC", "Segoe UI", HelveticaNeue-Light, Ubuntu, "Droid Sans", sans-serif; font-weight: normal; line-height: 1.2; padding-bottom: 0.3em; position: relative;">
Conclusion</h1>
<h1 class="code-line" data-line="0" style="border-bottom: 1px solid rgba(255, 255, 255, 0.18); border-left-color: rgba(255, 255, 255, 0.18); border-right-color: rgba(255, 255, 255, 0.18); border-top-color: rgba(255, 255, 255, 0.18); font-family: -apple-system, BlinkMacSystemFont, "Segoe WPC", "Segoe UI", HelveticaNeue-Light, Ubuntu, "Droid Sans", sans-serif; font-weight: normal; line-height: 1.2; padding-bottom: 0.3em; position: relative;">
<div class="code-line" data-line="177" style="font-size: 14px; position: relative;">
It's possible to run buildah in an unprivileged container but only using the vfs storage driver, but beware of the disk usage when building images!<br />
<br /></div>
<div class="code-line" data-line="179" style="font-size: 14px; position: relative;">
Interesting links:</div>
<ul style="font-size: 14px;">
<li class="code-line" data-line="180" style="position: relative;"><span style="color: black;"><a href="https://github.com/containers/fuse-overlayfs" style="text-decoration-line: none;" title="https://github.com/containers/fuse-overlayfs">https://github.com/containers/fuse-overlayfs</a> this will probably fix the <code style="font-family: Menlo, Monaco, Consolas, "Droid Sans Mono", "Courier New", monospace, "Droid Sans Fallback"; line-height: 19px;">CAP_SYS_ADMIN</code> issue for overlay</span></li>
<li class="code-line" data-line="181" style="position: relative;"><span style="color: black;"><a href="https://kinvolk.io/blog/2018/04/towards-unprivileged-container-builds/" style="text-decoration-line: none;" title="https://kinvolk.io/blog/2018/04/towards-unprivileged-container-builds/">https://kinvolk.io/blog/2018/04/towards-unprivileged-container-builds/</a> an overview of what's the problem and what's getting fixed regarding to unprivileged builds, must read!</span></li>
<li class="code-line" data-line="182" style="position: relative;"><a href="https://buildah.io/" style="text-decoration-line: none;" title="https://buildah.io/">https://buildah.io</a> everything about buildah</li>
</ul>
</h1>
42wimhttp://www.blogger.com/profile/03140786286473099167noreply@blogger.com2tag:blogger.com,1999:blog-2503925492290059.post-65483783941110834832015-09-28T01:08:00.000+02:002015-09-28T01:10:42.996+02:00How to create an IPv6-only consul cluster with docker<div class="separator" style="clear: both; text-align: center;">
</div>
<h2>
Why?</h2>
<ul>
<li>we're using docker to run consul (and registrator and our services) in, and IPv6 makes this easier (no NAT => better performance)</li>
<li>it's easier to maintain one stack</li>
<li>consul is known to give issues with NAT and docker (<a href="https://github.com/docker/docker/issues/8795">https://github.com/docker/docker/issues/8795</a>)</li>
<li>IPv4 is legacy and obsolete ;-)</li>
</ul>
Consul 0.5.2 has some issues running such a setup, but if you're building consul from master (which includes some fixes (see <a href="https://github.com/hashicorp/consul/commits?author=42wim">https://github.com/hashicorp/consul/commits?author=42wim</a>) it will work fine.<br />
<br />
<b>Issues to be aware of:</b><br />
<br />
<ul>
<li>the IPv4 version of consul listens by default on private address ranges, when using IPv6 you'll be running on 'public' addresses. So be sure you're firewalling those from the internet.</li>
<li>If you're using consul recursive powers, you'll also need IPv6 dns recursors. (e.g. google's 2001:4860:4860::8888)</li>
<li>Not IPv6 related, but for extra stability, enable leave_on_terminate.</li>
<li>Also not Ipv6 related, but I've noticed that the default LAN settings for consul can be a bit too strict when running on vmware hosts. This <a href="https://github.com/42wim/consul/commit/b20ddeb7310f0a2eda182436170316de5665bb7a" target="_blank">patch</a> increase the probetimeout to 2 seconds (instead of 500msec)</li>
</ul>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://snag.gy/bY9tm.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://snag.gy/bY9tm.jpg" height="318" width="640" /></a></div>
<br />
<br />
<h2>
Consul extra configuration server and client</h2>
<div>
Extra settings below necessary for the consul server and client agent setup</div>
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>
Configuration:
{
"recursor": "[2001:4860:4860::8888]",
"leave_on_terminate": true,
"client_addr": "::",
"addresses": { "http": "::"}
}
</code>
</pre>
<h2>
Consul server setup</h2>
The consul server are running as a docker host mode container (which means, they share the same network namespace as the host).<br />
<br />
The reason here is that we need a fixed IPv6 address for the servers because we're forwarding our dns requests to those servers. (ofcourse with some extra work we could make a script that dynamically update our dns forwards to the dynamic IP address).<br />
<br />
Our server has multiple IPv6 addresses so we'll have to add a -advertise and -bind flag<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>consul agent -server -advertise 2001:db8::1 -bind 2001:db8::1 -bootstrap-expect 3 -retry-join [2001:db8::1]:8301 -retry-join [2001:db8::2]:8301 -retry-join [2001:db8::3]:8301</code>
</pre>
<br />
Using <a href="https://github.com/gliderlabs/docker-consul" target="_blank">consul-docker</a> as our consul docker container (for client and server)<br />
<br />
<h2>
Consul client setup </h2>
You'll need to cherry-pick this PR into your local build: <a href="https://github.com/hashicorp/consul/pull/1219">https://github.com/hashicorp/consul/pull/1219</a>.<br />
The IPv6 address in the docker container will be random and we want to bind to the IPv6 address.<br />
This patch looks for the first 'public' IPv6 address and uses this address to advertise.<br />
<div>
<br /></div>
So we start the client with:<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>consul agent -bind :: -join consul.service.consul
</code></pre>
<br />
<b>Gotcha's here:</b><br />
bind :: actually binds to IPv4 and IPv6 addresses in the container, but because we advertise the IPv6 address the IPv4 address won't be used.<br />
<br />
<h2>
Other software</h2>
<div>
<h4>
Registrator</h4>
We also use <a href="https://github.com/gliderlabs/registrator" target="_blank">registrator</a> to register our services in consul. So every time a container starts or stops, registrator handles the consul service registration process.</div>
<div>
<br />
Also for registrator some extra fixes are needed to have IPv6 support. (not yet merged, see <a href="https://github.com/gliderlabs/registrator/pull/229">https://github.com/gliderlabs/registrator/pull/229</a>)<br />
<div>
<br /></div>
</div>
<div>
Because we're running consul on IPv6 this means registrator also needs to connect to the IPv6 address.<br />
<br /></div>
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>registrator consul://server1.node.consul:8500
</code></pre>
<div>
<br /></div>
<div>
Registrator then can register other services that are running on the docker host, like e.g elasticsearch.</div>
<div>
<br />
<h4>
Registrator-netfilter</h4>
</div>
<div>
Besides main registrator we also run <a href="https://github.com/42wim/registrator-netfilter" target="_blank">registrator-netfilter</a> which automatically firewalls the IPv6 services in the container. The containers are no longer NATted but directly accessible, so they need to be firewalled.<br />
<br /></div>
<div>
<h4>
Docker</h4>
</div>
<div>
A /64 is allocated for docker and a /80 is given to each docker host, running with the switches<br />
<br /></div>
<div>
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>--ipv6=true --fixed-cidr-v6=2001:db8::/80
</code></pre>
</div>
<div>
<br /></div>
<h4>
Elasticsearch</h4>
ES is also run ipv6 only, using registrator, registrator-netfilter and consul.<br />
You can find the relevant commands to give to docker below:<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>docker run --net bridge -e SERVICE_NAME=es -e SERVICE_9200_TAGS=http-data
-e SERVICE_9300_TAGS=transport-data -e SERVICE_9200_IPV6=tcp -e SERVICE_9300_IPV6=tcp
-e ADVERTISE_IPV6=yes
</code></pre>
<br />42wimhttp://www.blogger.com/profile/03140786286473099167noreply@blogger.com3tag:blogger.com,1999:blog-2503925492290059.post-30403060571379347162015-02-10T23:58:00.001+01:002015-02-10T23:58:10.610+01:00tmux memory usage on linux<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://photos1.blogger.com/x/blogger/781/3347/1600/49630/Shanty14balloonblowuplady.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://photos1.blogger.com/x/blogger/781/3347/1600/49630/Shanty14balloonblowuplady.jpg" height="240" width="320" /></a></div>
<br />
So a while ago I switched from screen to tmux. My reason for switching was that GNU screen didn't work in my docker containers and tmux did ;-)<br />
<div>
<br /></div>
<div>
All was well for a few months and I was replacing screen with tmux everywhere. It did have some other niceties besides working in containers and seem to do its job.<br />
<br />
Until<br />
<br /></div>
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
wim 1660 1.3 12.8 135056 131404 ? Ss 2014 722:46 tmux -u
</code>
</pre>
<br />
Notice anything special above ? Compare it with screen.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
wim 29595 0.0 4.5 48784 46116 ? Ss 2014 3:49 SCREEN -c mscreen
</code>
</pre>
<br />
The tmux session has 8 open windows and 10000 history limit. (set -g history-limit 10000)<br />
The screen session has 39 open windows and 10000 history limit (defscrollback 5000)<br />
<br />
So, tmux seems to be using an awful lot of memory. Two times more than screen, for a 'lighter' session setup.<br />
<br />
A quick google showed that other people were having the <a href="http://stackoverflow.com/questions/23535630/tmux-using-1g-ram-even-after-clearing-all-scrollback-history" target="_blank">same issues</a><br />
<br />
My first thought was, 'memoryleak', so I checked the code, but everything seemed to be free'd correctly.<br />
<br />
I joined the #tmux channel on freenode for some help and got told that it's a specific glibc (linux) issue. Although the memory was free'd, Glibc wasn't releasing it back to the OS.<br />
<br />
But you could force it by using <a href="http://man7.org/linux/man-pages/man3/malloc_trim.3.html" target="_blank">malloc_trim</a>(0). And maybe you could use <a href="http://man7.org/linux/man-pages/man3/mallopt.3.html" target="_blank">specific glibc environment variables</a> to control memory allocation behaviour to also emulate malloc_trim().<br />
<br />
Too much time googling and testing was wasted, I couldn't get it too work, the memory wasn't getting released back to the OS.<br />
<br />
So I made a small <a href="https://github.com/42wim/tmux/commit/429fc7a7498ca39c5210b26d4e541e8d6969db3a" target="_blank">patch</a> to tmux which<br />
- calls malloc_trim(0) when a window gets destroyed<br />
- also free's memory when you clear your history manually in a window (and also call malloc_trim())<br />
<br />
The patch works for me but YMMV<br />
<br />
I tried to get this patch into upstream tmux, but was told: 'It's up to glibc to decide how malloc works'.<br />
<br />
PS: if you set history-limit 0, tmux actually uses less memory than screen (and doesn't grow), but ofcourse you don't have a scrollback ;-)<br />
<br />
<div>
<br /></div>
42wimhttp://www.blogger.com/profile/03140786286473099167noreply@blogger.com3tag:blogger.com,1999:blog-2503925492290059.post-1886595050311216412015-01-24T00:45:00.000+01:002015-01-24T00:51:44.365+01:00Rancid 3.2 alpha + git<br />
Rancid lovers rejoice, a 3.2 alpha version is <a href="http://www.shrubbery.net/pipermail/rancid-discuss/2014-December/007972.html" target="_blank">released</a> with (at least) 2 interesting features.<br />
<br />
- <b>Git support</b>: based on the patch by <a href="https://github.com/jcollie/rancid" target="_blank">jcollie</a>.<br />
<br />
But with a 'small' difference, not one repository for all the groups, but a repository per group.<br />
Maybe fine if your starting from scratch, but for my situation I like the one repository setup of the original patch.<br />
<br />
You can find the latest version with the original setup of one repository for everything, together with some other minor patches on <a href="https://github.com/42wim/rancid/commits/mypatches3.1.99">https://github.com/42wim/rancid/commits/mypatches3.1.99</a><br />
<br />
- <b>WLC support</b>: Now you can backup your Cisco Wireless Lan Controllers configuration out of the box. One patch less to maintain. Hurrah!<br />
<br />
I'm running Rancid in a Docker setup, so upgrading and testing was quite easy.<br />
No issues found yet with this version.<br />
<br />42wimhttp://www.blogger.com/profile/03140786286473099167noreply@blogger.com0tag:blogger.com,1999:blog-2503925492290059.post-27717236126062218912014-02-25T00:16:00.000+01:002014-02-25T00:16:16.461+01:00Circumventing IPv6 feature parity: drop AAAA to specific IPsUnless you've been living under a rock, you'll be aware that IPv6 usage has been increasing.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://snag.gy/0AnqO.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://snag.gy/0AnqO.jpg" height="200" width="320" /></a></div>
<br />
Yes, it even has come to this: mere mortals can use it at home. The audacity!<br />
<br />
Unfortunately not all vendors (if any?) have feature parity, in our case a specific VPN product doesn't support IPv6.<br />
The client will only receive an IPv4 address from the VPN server.<br />
<br />
When the user at home starts it's VPN and asks for an internal resource (which also has an IPv6 address), it will try to connect to this resource using the IPv6 from his provider (he didn't receive one from the VPN server) which doesn't work, because this specific resource is firewalled for outside addresses.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://snag.gy/0RvFr.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://snag.gy/0RvFr.jpg" height="504" width="640" /></a></div>
<br />
Luckily the user has to use our DNS server to look up records (forced to do so by the vpn client)<br />
Luckily we're using <a href="http://www.powerdns.com/">PowerDNS recursor</a> which has support for LUA scripting which can modify DNS responses.<br />
<br />
The script below gives normal answers to every host not coming from 10.100.0.0/15 or 10.0.0.1/32. Otherwise if the answer contains an AAAA, drop it, and return the rest.<br />
<div>
<br /></div>
<br />
<script src="https://gist.github.com/42wim/9198691.js"></script>More information about LUA scripting for PowerDNS can be found here: <a href="http://doc.powerdns.com/html/recursor-scripting.html">http://doc.powerdns.com/html/recursor-scripting.html</a>42wimhttp://www.blogger.com/profile/03140786286473099167noreply@blogger.com0tag:blogger.com,1999:blog-2503925492290059.post-24466646748365717742013-12-01T00:06:00.003+01:002013-12-01T00:07:21.626+01:00Winter is comingWhich means I have some more time, expect some IPv6 and Cisco related posts in the near future.<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://i.imgflip.com/54wnv.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://i.imgflip.com/54wnv.jpg" height="235" width="320" /></a></div>
<br />42wimhttp://www.blogger.com/profile/03140786286473099167noreply@blogger.com0tag:blogger.com,1999:blog-2503925492290059.post-73169551195097694672013-05-13T23:20:00.001+02:002013-08-27T22:40:48.458+02:00Compiling your own PuTTY-CAC with EID support<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
So we've got electronic ID's, (smartcards) but except for doing our taxes we're not using them so much.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://blog.negonation.com/es/wp-content/uploads/2008/05/eid-belgium-front-medium.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="207" src="http://blog.negonation.com/es/wp-content/uploads/2008/05/eid-belgium-front-medium.png" width="320" /></a></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Now u<span style="font-size: 11pt;">nder linux there are options to use them for SSH authentication, but these days I'm mostly using Putty on Windows, so I wanted it to work with this client. </span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-size: 11pt;"><br /></span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-size: 11pt;">After some searches I found a possible candidate: Putty-cac : </span><a href="http://www.risacher.org/putty-cac/">http://www.risacher.org/putty-cac/</a></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
It works with CAPI, the military uses it, it's opensource and based on Putty. Seems like a win-win-win-win. And for once it also is :-)</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
So I downloaded the source from <a href="https://github.com/risacher/putty-cac/archive/master.zip">https://github.com/risacher/putty-cac/archive/master.zip</a></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
compared it with the official putty source from http://svn.tartarus.org/sgt/putty/ to see if nothing suspicious was added to the code. There wasn't, so I could safely build the binary myself.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11pt; margin: 0in;">
<span style="font-size: 11pt;">I remembered that Visual Studio Express was a free C++ compiler from Microsoft, so i download version 2010</span></div>
<div style="font-family: Calibri; font-size: 11pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11pt; margin: 0in;">
So now just open the project and press build right? Wrong! The project was made in Visual studio 6 and apparently you can not convert from visual studio 6 to visual studio 2010. According to the internets you need to first install Visual studio 2008, convert there, save it, open it in Visual studio 2010, convert, save and build.</div>
<div style="font-size: 11pt; margin: 0in;">
<br /></div>
<br />
<div style="font-family: Calibri; font-size: 11pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11pt; margin: 0in;">
Here is an overview for those that want to do this:</div>
<div style="font-family: Calibri; font-size: 11pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11pt; margin: 0in;">
Start visual C++ 2008</div>
<div style="font-family: Calibri; font-size: 11pt; margin: 0in;">
Open Project - c:\temp\putty-cac-master\windows\MSVC\putty.dsw</div>
<div style="font-family: Calibri; font-size: 11pt; margin: 0in;">
Convert and open project</div>
<div style="font-family: Calibri; font-size: 11pt; margin: 0in;">
Choose File - Save All</div>
<div style="font-family: Calibri; font-size: 11pt; margin: 0in;">
Start visual C++ 2010 (and close 2008 ;-)</div>
<div style="font-family: Calibri; font-size: 11pt; margin: 0in;">
Open Project - c:\temp\putty-cac-master\windows\MSVC\putty.sln</div>
<div style="font-family: Calibri; font-size: 11pt; margin: 0in;">
You'll get a wizard: Next - Next - Finish</div>
<div style="font-family: Calibri; font-size: 11pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11pt; margin: 0in;">
Now when you try to build it, it won't. You'll need to add a define </div>
<div style="font-family: Calibri; font-size: 11pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11pt; margin: 0in;">
Open c:\temp\putty-cac-master\windows\winstuff.sh</div>
<div style="font-family: Calibri; font-size: 11pt; margin: 0in;">
Add #define SECURITY_WIN32 at the top of the file</div>
<div style="font-family: Calibri; font-size: 11pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11pt; margin: 0in;">
If you compile now you'll get linking errors. You'll need to add 'sc.c' and 'capi.c' to the 'source files'</div>
<div style="font-family: Calibri; font-size: 11pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11pt; margin: 0in;">
Now you're finally ready to build your binary. Press build and e<span style="font-size: 11pt;">njoy your own build putty.</span></div>
<div style="font-family: Calibri; font-size: 11pt; margin: 0in;">
<br /></div>
<div style="margin: 0in;">
<span style="font-family: Calibri; font-size: 11pt;">To actually use your EID with this, just follow the CAPI instructions on </span><span style="font-family: Calibri;"><span style="font-size: 15px;">http://www.risacher.org/putty-cac/</span></span></div>
<br />
<br />
<div style="font-family: Calibri; margin: 0in;">
<b>Stuff you'll need to get this working:</b></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Microsoft Windows
SDK for Windows 7 and .NET Framework 4</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<a href="http://www.microsoft.com/en-us/download/details.aspx?id=8279">http://www.microsoft.com/en-us/download/details.aspx?id=8279</a></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Visual 2008 express</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<a href="http://download.microsoft.com/download/E/8/E/E8EEB394-7F42-4963-A2D8-29559B738298/VS2008ExpressWithSP1ENUX1504728.iso">http://download.microsoft.com/download/E/8/E/E8EEB394-7F42-4963-A2D8-29559B738298/VS2008ExpressWithSP1ENUX1504728.iso</a></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Visual 2010 express</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<a href="http://go.microsoft.com/?linkid=9709949">http://go.microsoft.com/?linkid=9709949</a></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-size: 11pt;">PuTTY CAC source</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<a href="https://github.com/risacher/putty-cac/archive/master.zip">https://github.com/risacher/putty-cac/archive/master.zip</a></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Good luck!</div>
42wimhttp://www.blogger.com/profile/03140786286473099167noreply@blogger.com2tag:blogger.com,1999:blog-2503925492290059.post-8767281387828003432013-04-03T00:40:00.001+02:002013-05-13T23:23:21.516+02:00DHCPv6: ISC MAC logging and Cisco relay agent configuration<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h2>
<span style="font-family: inherit; font-size: large;">DHCPv6 server MAC address logging</span></h2>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
One of the
differences between DHCPv6 and DHCPv4 is that it uses DUID as an identifier
instead of a MAC address.<br />
<span style="font-size: 11pt;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjC6ew2vW3JViD2G_tlXBOGyr7P0PWLfaB8wf0cS8bcPaVlY2-4oLxFef5sSBielo8kNTWEa9197Bd0ei7mQTx8MccgMfyqIZ8FSazOh9y9qYXDjaLyQjEH9ZZr_hwJJknapUm2dA3IoA/s1600/whynomac.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjC6ew2vW3JViD2G_tlXBOGyr7P0PWLfaB8wf0cS8bcPaVlY2-4oLxFef5sSBielo8kNTWEa9197Bd0ei7mQTx8MccgMfyqIZ8FSazOh9y9qYXDjaLyQjEH9ZZr_hwJJknapUm2dA3IoA/s320/whynomac.png" height="240" width="320" /></a></div>
<span style="font-size: 11pt;"><br /></span>
<span style="font-size: 11pt;">As you probably
know, DUID's are for the majority of OS (ie Windows) based on a timestamp
suffixed by a MAC address</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-size: 11pt;">For some of our
internal systems we use a MAC as an identifier, for now we will also be needing this for IPv6. The default ISC DHCPv6
daemon isn't logging a MAC address by default. One way to have it print out is
by adding this to your DHCP config.</span><br />
<span style="font-size: 11pt;"><br /></span></div>
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>option dhcp6.macaddr code 193 = string;
option dhcp6.leased-address code 194 = string;
option dhcp6.macaddr = binary-to-ascii(16, 8, ":", suffix(option dhcp6.client-id, 6));
option dhcp6.leased-address = binary-to-ascii(16,16, ":", substring(suffix(option dhcp6.ia-na, 24),0,16));
log (info, concat ("Lease for ",config-option dhcp6.leased-address, " leased to ", config-option dhcp6.macaddr));</code>
</pre>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br />
Above code will only
work for DUID-LLT and DUID-LL (so not DUID-EN, but I don't know anyone using
this at the moment)</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
More info over DUID on <a href="http://tools.ietf.org/html/rfc6355">http://tools.ietf.org/html/rfc6355</a></div>
<div style="font-family: "Courier New"; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Other great blogpost
about DHCPv6 at <a href="http://ipv6friday.org/blog/2011/12/dhcpv6/">http://ipv6friday.org/blog/2011/12/dhcpv6/</a></div>
<div style="font-family: "Courier New"; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: "Courier New"; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-size: 11pt; margin: 0in;">
<h2>
<span style="font-family: inherit; font-size: large;"><br /></span></h2>
<h2>
<span style="font-family: inherit; font-size: large;">DHCPv6 relay configuration on Cisco equipment</span></h2>
<div>
<span style="font-family: Calibri; font-size: 11pt;">Overview of the configuration we're using on our routers.</span></div>
<div>
<br /></div>
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
(1) We're using
FE80:: as our IPv6 default gateway everywhere, seems to work for me for now ;) any best practices for this? (<b>update: because of issues with linux and fe80:: (linux responds to fe80:: if it's specified on any link, this has now changed to FE80::1)</b><br />
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
(2) Asking the hosts
on the subnet to not do SLAAC please, we're asking you nicely.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
(3) Letting the
hosts know we're managing the config, that they must use DHCPv6.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
(4) Just to be sure
also set this flag, tell them to use DHCPv6 not only for getting an IP but also
for getting e.g. DNS servers</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
(5) We're the boss
on this subnet.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
(6) relay DHCPv6
requests to our server and use (7) the loopback as a source for this.<br />
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
We're using
link-local addresses for routing, so if we don't specifiy an source-interface,
the relay agents would try to use a link-local address which obviously can not
be routed.<br />
<br /></div>
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>Interface vlan42
ipv6 address FE80::1 link-local (1)
ipv6 address 2001:0db8:100:4200::1/64
ipv6 nd prefix default 2592000 604800 no-autoconfig (2)
ipv6 nd managed-config-flag (3)
ipv6 nd other-config-flag (4)
ipv6 nd router-preference High (5)
ipv6 dhcp relay destination 2001:0db8:0:40::547:1 (6)
ipv6 dhcp relay source-interface Loopback0 (7)
interface Loopback0
ipv6 address 2001:0db8:300::63/128</code>
</pre>
42wimhttp://www.blogger.com/profile/03140786286473099167noreply@blogger.com3tag:blogger.com,1999:blog-2503925492290059.post-71547155359274470352013-02-26T23:09:00.000+01:002013-02-26T23:10:17.304+01:00IPv6 logging and Cisco NCS<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
So you want to
deploy IPv6 on your wireless network. </div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
You want to use SLAAC and you want
logging of those SLAAC addresses.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Straight from the
horses mouth:</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Q: What are IPv6 private addresses and why are they
important to track?</span> </div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">A:</span> Private (also known as temporary) addresses
are randomly generated by the client when SLAAC address assignment is in use.
These addresses are often rotated at a frequency of a day or so, as to prevent
host traceability that would come from using the same host postfix (last 64
bits) at all times. It is important to track these private addresses for
auditing purposes such as tracing copyright infringement. <span style="font-weight: bold; text-decoration: underline;">Cisco NCS records all IPv6
addresses in use by each client and historically logs them each time the client
roams or establishes a new session.</span> These records can be configured at
NCS to be held for up to a year.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="color: #595959; font-family: Calibri; font-size: 9.0pt; margin: 0in;">
Pasted
from <<a href="http://www.cisco.com/en/US/products/ps10315/products_tech_note09186a0080bae506.shtml#faqs">http://www.cisco.com/en/US/products/ps10315/products_tech_note09186a0080bae506.shtml#faqs</a>>
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
BUT Cisco NCS or PI doesn't
make it easy for you, it keeps the addresses in the database somewhere, but as
soon as the client has disassociated you can't search for the IPv6 address
anymore. <span style="font-size: 11pt;">Disassociated IPv4 addresses
can be searched though.</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://i.qkme.me/3t5g76.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="270" src="http://i.qkme.me/3t5g76.jpg" width="400" /></a></div>
<div style="margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
So far for IPv6
parity.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h3>
<span style="font-size: large;">Radius to the rescue ?</span><span style="font-size: 11pt;"> </span></h3>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
But wait, we've got
radius accounting, right? You'll see the Framed-IP-Address attribute and this
will show you the IPv6 address, right ? RIGHT ?</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
As we all know, IPv6
is a very new protocol (only about 15 years old), so of course there isn't support for
IPv6 in the Framed-IP-Address attribute. There is a draft proposing
Framed-IPv6-Address (also very new, only 3 years old).</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
(See <a href="http://tools.ietf.org/html/draft-ietf-radext-ipv6-access-16" style="font-size: 11pt;">http://tools.ietf.org/html/draft-ietf-radext-ipv6-access-16</a> for more info)</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
So, no searching, no
IPv6 address logging by Radius.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
So far for IPv6 parity again..</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
But according to the
documentation (see above) cisco is recording those addresses somewhere ...</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h3>
<span style="font-size: large;">Reports to the rescue!</span></h3>
<div style="font-family: Calibri; margin: 0in;">
<span style="font-size: 11pt;">The workaround is
reports:</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Go to the report
launch pad - Client - Client Sessions</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Create a new report:</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
-
report by SSID (or your own favorite source)</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
-
reporting criteria (all SSIDs)</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
-
reporting period select last 7 days</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Customize the
report, where you can find the most important data field: "Global
Unique", this will show you the IPv6 address. Now you can schedule this
report weekly and you've got weekly CSV files containing all the necessary
information of the users.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
If anyone got a better workaround please share!</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzmWRgpAUl3Rxrdol2bqjo32svMNeZJmLQqB8bxM4Le-FohAE86NfIGbQc8uJlm_bya5RzoUCDbrFV6xDsx16Ef6Ykc3DWdCZocVgA2IaDtsDV6DjTVfkukGBqUj_sS4-8jvnKAvWv-w/s1600/ncs1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="464" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzmWRgpAUl3Rxrdol2bqjo32svMNeZJmLQqB8bxM4Le-FohAE86NfIGbQc8uJlm_bya5RzoUCDbrFV6xDsx16Ef6Ykc3DWdCZocVgA2IaDtsDV6DjTVfkukGBqUj_sS4-8jvnKAvWv-w/s640/ncs1.png" width="640" /></a></div>
<div style="margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvhdUvqb7JWxbPrYgtlXof9noCvI0A_Fdjb5VeFA2G67CTzrUkX7bs5u2dbX64eJXWHfSzVRWBNZ7frQd6XTZgEHYQ1pE_EIxbkZY-OrEDaTloXRvKDa7W-DR83OBWI5WzvkuQIIgdjQ/s1600/ncs2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="163" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvhdUvqb7JWxbPrYgtlXof9noCvI0A_Fdjb5VeFA2G67CTzrUkX7bs5u2dbX64eJXWHfSzVRWBNZ7frQd6XTZgEHYQ1pE_EIxbkZY-OrEDaTloXRvKDa7W-DR83OBWI5WzvkuQIIgdjQ/s640/ncs2.png" width="640" /></a></div>
<div style="margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
42wimhttp://www.blogger.com/profile/03140786286473099167noreply@blogger.com0tag:blogger.com,1999:blog-2503925492290059.post-5839154456452983982013-02-07T00:09:00.000+01:002013-02-07T00:12:02.981+01:00Cisco Prime Infrastructure NCS 1.2 and the Oracle issueContinuing <a href="http://blog.42.be/2013/01/increasing-diskspace-on-cisco-prime.html">my story</a> about NCS diskspace increase.<br />
<br />
A few weeks everything went fine, until one day the webserver or the shell wasn't accessible anymore.<br />
In the vmware server console we saw that all of the memory was in use (8GB). So we add some extra memory and rebooted the system. (At that time I didn't made the correlation with increasing the disk space a few weeks ago)<br />
<br />
This didn't solve the problem.<br />
<br />
But I could log in and memory usage seemed normal. First I went looking in /var/log , you can skip this step, nothing of value can be found here.<br />
<br />
The interesting directories are /opt/CSCOlumos/logs/<br />
<br />
Last file modified was hm-0-0.log. Containing:<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>ERROR [system] [HealthMonitorServer] HealthMonitorServer.initHealthMonitor: initHealthMonitor(): can not start DB
INFO [database] [Thread-17] OracleSchemaUtil dbServerUp(): wcs errorCode = 1034</code>
</pre>
<br />
(Other logfiles were also giving oracle errors)<br />
Closing in on the oracle DB.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://i.qkme.me/3svrsj.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://i.qkme.me/3svrsj.jpg" height="320" width="283" /></a></div>
<br />
<br />
Next target /opt/oracle (disclaimer: I know nothing about Oracle), after some searching I discovered the logfile /opt/oracle/diag/rdbms/wcs/wcs/trace/alert_wcs.log which seemed interesting.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>ORA-19815: WARNING: db_recovery_file_dest_size of 107374182400 bytes is 100.00% used, and has 0 remaining bytes available.
ORA-19809: limit exceeded for recovery files
ORA-16038: log 1 sequence# 1018 cannot be archived
ORACLE Instance wcs - Archival Error
ARCH: Archival stopped, error occurred. Will continue retrying</code>
</pre>
<div>
<br /></div>
This didn't look good. Apparently Oracle has it own internal idea of how big a disk is and how much free space is available. Because we still had 130Gb available.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>/dev/mapper/smosvg-optvol
249204396 97436636 138909244 42% /opt
</code></pre>
<br />
With the help of google and a colleague with the necessary oracle knowledge. This was the solution (increasing the recovery db size):<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>[root@ncs oracle]# su - oracle
[oracle@ncs ~]$ ls
base coracleenv dbPasswd.pwd oracleenv oraInventory templates utils
[oracle@ncs ~]$ . oracleenv
[oracle@ncs ~]$ sqlplus '/as sysdba'
SQL*Plus: Release 11.2.0.2.0 Production
Copyright (c) 1982, 2010, Oracle. All rights reserved.
Connected to an idle instance.
SQL> startup nomount
ORACLE instance started.
Total System Global Area 4275781632 bytes
Fixed Size 2233336 bytes
Variable Size 2986347528 bytes
Database Buffers 1275068416 bytes
Redo Buffers 12132352 bytes
SQL> show parameter db_recovery_file_dest_size
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
db_recovery_file_dest_size big integer 100G
SQL> alter system set db_recovery_file_dest_size=120G scope=both;
System altered.
SQL> show parameter db_recovery_file_dest_size;
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
db_recovery_file_dest_size big integer 120G
SQL> shutdown
ORA-01507: database not mounted
ORACLE instance shut down.
SQL> quit
</code></pre>
<br />
Still the error, but now some free space to do something<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>ORA-19815: WARNING: db_recovery_file_dest_size of 128849018880 bytes is 85.07% used, and has 19241095680 remaining bytes available.
</code></pre>
<br />
Next ran ncs cleanup on the NCS CLI<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>ncs/admin# ncs cleanup
===================================================
Starting Cleanup:
===================================================
Removing all files in backup staging directory
Removing all Matlab core related files
Removing all older log files
Cleaning older archive logs
Cleaning database backup and all archive logs
Cleaning database
Stopping database
Starting database
Starting database clean
Completed database clean
Stopping database
===================================================
Completed Cleanup
===================================================
ncs/admin#</code>
</pre>
<br />
And rebooted the system.<br />
<br />
Oracle DB came up normally and NCS was happily running again.42wimhttp://www.blogger.com/profile/03140786286473099167noreply@blogger.com13tag:blogger.com,1999:blog-2503925492290059.post-55076594595695802302013-01-21T22:51:00.002+01:002013-01-21T23:02:06.543+01:00New features added to WPTVdroidNew features recently added to WPTVdroid:<br />
<br />
Channel guide (with current and next running show) added when you click long on the selected channel.<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwHHIpMUE-Dz3u1kaXmU6lcm1ve50sn-VJO6pbeHkgvmAIG9c-QNv0TyFmQ9L6tA8FmcXuMzUDM2QdovSsteiswXtz6wQKBSunKtXPydZlHSiQqoobqYAmJi6aDrIbNb8HVjez5PCYdQ/s1600/2013-01-21-22-37-58.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="177" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwHHIpMUE-Dz3u1kaXmU6lcm1ve50sn-VJO6pbeHkgvmAIG9c-QNv0TyFmQ9L6tA8FmcXuMzUDM2QdovSsteiswXtz6wQKBSunKtXPydZlHSiQqoobqYAmJi6aDrIbNb8HVjez5PCYdQ/s1600/2013-01-21-22-37-58.png" width="320" /></a></div>
<br />
<br />
Channel preview, go to menu and select "Channels preview", this will briefly connect to every channel and will show you a screenshot of the current program. Just click on the screenshot to go to the selected channel.<br />
(sometimes the screenshot will not be very clear/show artifacts.. haven't found a solution for this yet)<br />
(this feature adds about 10MB to the WPTVdroid installation, if you don't have that much space don't use this version)<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEifJ-6FxKVYdDNvgML5dtRQdQyLD1ayVZDC4A14HXX1fASolpHcXf7KqYpPoee4vFpzylU9hNkyym5fKVS_t_QK5R2oaWz6Z8aiDS5cqKqukGv804MK5_QeO2XL3_6NFssKOuA4KYD9dQ/s1600/2013-01-21-22-40-47.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="177" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEifJ-6FxKVYdDNvgML5dtRQdQyLD1ayVZDC4A14HXX1fASolpHcXf7KqYpPoee4vFpzylU9hNkyym5fKVS_t_QK5R2oaWz6Z8aiDS5cqKqukGv804MK5_QeO2XL3_6NFssKOuA4KYD9dQ/s1600/2013-01-21-22-40-47.png" width="320" /></a></div>
<br />
<br />
Downloads on <a href="http://blog.42.be/p/weepeetv.html">http://blog.42.be/p/weepeetv.html</a><br />
<br />
Any new features/ideas you would like to have added? Leave a comment or contact me @42wim<br />
<br />
<br />42wimhttp://www.blogger.com/profile/03140786286473099167noreply@blogger.com4tag:blogger.com,1999:blog-2503925492290059.post-15307303362065923412013-01-15T22:39:00.001+01:002013-01-15T22:39:14.483+01:00Increasing diskspace on Cisco Prime Infrastructure NCS 1.2 "the hard way?"Somewhere last year PI NCS 1.2 was constantly complaining about disk full errors, although the the disk was only 62% used.<br />
<br />
Apparently the complaining starts above 60% because it needs to be able to make a temporary backup on the same disk as it is running the database.<br />
<br />
Our datacenter guy increased the disk (200GB) in vmware by increasing it by 100GB.<br />
<br />
I wrongly thought this would have fixed the problem and that the system would automatically resize everything.<br />
<br />
I was wrong.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://cdn.memegenerator.net/instances/250x250/33430387.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://cdn.memegenerator.net/instances/250x250/33430387.jpg" /></a></div>
<br />
<br />
So after running root_enable it was possible to login as root (it's just running a linux beneath it)<br />
<br />
Below you can see the layout of the system with 62% in use for /opt where the database and temporary backups are located.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/smosvg-rootvol
1967952 477860 1388512 26% /
/dev/mapper/smosvg-tmpvol
1967952 36460 1829912 2% /tmp
/dev/sda3 988116 17764 919348 2% /storedconfig
/dev/mapper/smosvg-recvol
95195 5664 84616 7% /recovery
/dev/mapper/smosvg-home
95195 5668 84612 7% /home
/dev/mapper/smosvg-optvol
147630356 86745492 53264668 62% /opt
/dev/mapper/smosvg-usrvol
5935604 904020 4725204 17% /usr
/dev/mapper/smosvg-varvol
1967952 90372 1776000 5% /var
/dev/mapper/smosvg-storeddatavol
3967680 74320 3688560 2% /storeddata
/dev/mapper/smosvg-altrootvol
95195 5664 84616 7% /altroot
/dev/mapper/smosvg-localdiskvol
29583412 204132 27852292 1% /localdisk
/dev/sda1 101086 12713 83154 14% /boot
tmpfs 4021728 2044912 1976816 51% /dev/shm
</code></pre>
<div>
<br /></div>
How to fix this manually.<br />
So we run fdisk, and we see that the extra 100GB is detected (/dev/sda 322.1GB)<br />
We then add another (4) primary partition which consists of the missing 100GB<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed rgb(153, 153, 153); overflow: auto; padding: 5px; width: 100%;"><code><span style="font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace;"><span style="font-size: 12px; line-height: 14px;"># fdisk /dev/sda
The number of cylinders for this disk is set to 39162.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Command (m for help): p
Disk /dev/sda: 322.1 GB, 322122547200 bytes
255 heads, 63 sectors/track, 39162 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 25368 203664037+ 8e Linux LVM
/dev/sda3 25369 25495 1020127+ 83 Linux
Command (m for help): n
Command action
e extended
p primary partition (1-4)
</span></span><div>
<span style="font-size: 12px; line-height: 14px;">p
Selected partition 4
First sector (409577175-629145599, default 409577175):
Using default value 409577175
Last sector or +size or +sizeM or +sizeK (409577175-629145599, default 629145599):
Using default value 629145599
Command (m for help): p
Disk /dev/sda: 322.1 GB, 322122547200 bytes
255 heads, 63 sectors/track, 39162 cylinders, total 629145600 sectors
Units = sectors of 1 * 512 = 512 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 63 208844 104391 83 Linux
/dev/sda2 208845 407536919 203664037+ 8e Linux LVM
/dev/sda3 407536920 409577174 1020127+ 83 Linux
/dev/sda4 409577175 629145599 109784212+ 83 Linux</span><span style="font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace;"><span style="font-size: 12px; line-height: 14px;">
</span></span></div>
</code></pre>
<br />
Also tag it 8e to be a linux LVM filesystem.<br />
And write the partition table <scary><br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>Command (m for help): t
Partition number (1-4): 4
Hex code (type L to list codes): 8e
Changed system type of partition 4 to 8e (Linux LVM)
Command (m for help): p
Disk /dev/sda: 322.1 GB, 322122547200 bytes
255 heads, 63 sectors/track, 39162 cylinders, total 629145600 sectors
Units = sectors of 1 * 512 = 512 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 63 208844 104391 83 Linux
/dev/sda2 208845 407536919 203664037+ 8e Linux LVM
/dev/sda3 407536920 409577174 1020127+ 83 Linux
/dev/sda4 409577175 629145599 109784212+ 8e Linux LVM
Command (m for help): v
62 unallocated sectors
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.
</code></pre>
<br />
Now reboot the system.<br />
When it boots up, the partition can be used.<br />
Use pvcreate to create the new physical volume.<br />
vgdisplay will still show the old 194.22GB because it's not added yet to smosvg.
<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code># pvcreate /dev/sda4
Physical volume "/dev/sda4" successfully created
# vgdisplay
--- Volume group ---
VG Name smosvg
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 12
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 11
Open LV 11
Max PV 0
Cur PV 1
Act PV 1
VG Size 194.22 GB
PE Size 32.00 MB
Total PE 6215
Alloc PE / Size 6215 / 194.22 GB
Free PE / Size 0 / 0
VG UUID vJcFNu-qPIP-uoKY-KiIu-wKCd-Jj7l-e8cLaV
</code></pre>
<br />
pvdisplay will show the 2 volumes.
<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code># pvdisplay
--- Physical volume ---
PV Name /dev/sda2
VG Name smosvg
PV Size 194.23 GB / not usable 10.66 MB
Allocatable yes (but full)
PE Size (KByte) 32768
Total PE 6215
Free PE 0
Allocated PE 6215
PV UUID ugQzvR-HsdX-cK8u-Sylm-NYAm-FXFX-YXkb6R
--- Physical volume ---
PV Name /dev/sda4
VG Name smosvg
PV Size 104.70 GB / not usable 11.15 MB
Allocatable yes
PE Size (KByte) 32768
Total PE 3350
Free PE 3350
Allocated PE 0
PV UUID pEPFVV-AI3e-HLfo-fy5i-TyEN-eRCp-BJiXPS
</code></pre>
<br />
Now add this volume to /opt and extend it by 50Gb
And use resize2fs so that the kernel nows the size changed.
<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code># lvextend /dev/mapper/smosvg-optvol /dev/sda4
# lvextend -L +50G /dev/mapper/smosvg-optvol
Extending logical volume optvol to 195.34 GB
Logical volume optvol successfully resized
# resize2fs /dev/mapper/smosvg-optvol
resize2fs 1.39 (29-May-2006)
Filesystem at /dev/mapper/smosvg-optvol is mounted on /opt; on-line resizing required
Performing an on-line resize of /dev/mapper/smosvg-optvol to 51208192 (4k) blocks.
The filesystem on /dev/mapper/smosvg-optvol is now 51208192 blocks long.
</code></pre>
<br />
A df will noshow 47% in use for /opt<br />
The annoying popups are now gone.
<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code># df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/smosvg-rootvol
1967952 477876 1388496 26% /
/dev/mapper/smosvg-tmpvol
1967952 36460 1829912 2% /tmp
/dev/sda3 988116 17764 919348 2% /storedconfig
/dev/mapper/smosvg-recvol
95195 5664 84616 7% /recovery
/dev/mapper/smosvg-home
95195 5668 84612 7% /home
/dev/mapper/smosvg-optvol
198417376 88100224 100077796 47% /opt
/dev/mapper/smosvg-usrvol
5935604 904020 4725204 17% /usr
/dev/mapper/smosvg-varvol
1967952 88744 1777628 5% /var
/dev/mapper/smosvg-storeddatavol
3967680 74320 3688560 2% /storeddata
/dev/mapper/smosvg-altrootvol
95195 5664 84616 7% /altroot
/dev/mapper/smosvg-localdiskvol
29583412 204132 27852292 1% /localdisk
/dev/sda1 101086 12713 83154 14% /boot
tmpfs 4021728 2044888 1976840 51% /dev/shm
</code></pre>
<br />
It seems like I fixed everything and that I was in the clear.<br />
<br />
Which it was for a couple of weeks until ... to be continued in a later post.
<br />
<br />
<br />
<br />42wimhttp://www.blogger.com/profile/03140786286473099167noreply@blogger.com15tag:blogger.com,1999:blog-2503925492290059.post-4735833786093768572013-01-10T22:38:00.000+01:002013-01-15T22:40:02.122+01:00Simple WeepeeTV recorder<br />
Created the record.pl script, with this simple script you can record WeepeeTV streams.<br />
<br />
Unfortunately the WeepeeTV streams aren't that stable and ffmpeg (which is used to record) can't append recordings, so sometimes your recordings may not be complete.<br />
<h2>
How it works</h2>
Basically it will create a small shell script in /tmp which will be queued as an at job
at the time you specified. This script will be executed at the specified time and
will be calling ffmpeg to save the stream to a .mkv file in the outputdirectory you've specified.<br />
<h2>
Config</h2>
To run the script change the variables on top of record.pl<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>
<span class="c1">#make sure your ffmpeg supports https</span>
<span class="k">my</span> <span class="nv">$FFMPEG</span><span class="o">=</span><span class="s">"/usr/bin/ffmpeg-weepeetv"</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$ATCMD</span><span class="o">=</span><span class="s">"/usr/bin/at"</span><span class="p">;</span>
<span class="c1">#locations of xml source file</span>
<span class="k">my</span> <span class="nv">$XMLFILE</span><span class="o">=</span><span class="s">"wptv.xml"</span><span class="p">;</span>
<span class="c1">#where to write the recordings</span>
<span class="k">my</span> <span class="nv">$OUTPUTDIR</span><span class="o">=</span><span class="s">"/opt/recordings"</span><span class="p">;</span>
<span class="c1">#installation directory</span>
<span class="k">my</span> <span class="nv">$PATH</span><span class="o">=</span><span class="s">"/home/wim/wptvscraper"</span><span class="p">;</span>
</code>
</pre>
<h2>
Usage</h2>
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>
record.pl --start=<time date> --duration <hh:mm:ss> --desc <description> --channel <name>
e.g. perl record.pl --start=15:00 tomorrow --duration 00:30:00 --desc "crappy show" --channel een
records 30 minutes starting tomorrow at 15h on channel 'een'
<time date> can be anything 'at' supports, see 'man at'
Other options:
--channels (shows available channels)</code></pre>
<br />
<br />
You can find the code on <a href="https://github.com/42wim/wptvscraper">https://github.com/42wim/wptvscraper</a><br />
<br />42wimhttp://www.blogger.com/profile/03140786286473099167noreply@blogger.com10tag:blogger.com,1999:blog-2503925492290059.post-157772540207973622013-01-05T15:45:00.001+01:002013-01-15T22:40:29.051+01:00WeepeeTV and VLC (2)<br />
Created createwptvxml.pl and weepeetv.lua (for use with vlc)<br />
<br />
createwptvxml.pl is a simple screenscaper perl script <i>(yes it's written in Perl and not in a cool language like Python or Ruby)</i> which logs in on the
WeepeeTV site and creates an XML containing the necessary m3u8 stream URLS.<br />
<br />
This XML can be used to feed other applications on your local network
e.g. for use with VLC (see weepeetv.lua)<br />
The xml file format is as follows:<br />
<div class="highlight">
<pre><span class="cp"><?xml version='1.0'?></span>
<span class="nt"><items></span>
<span class="nt"><item></span>
<span class="nt"><title></span>channel<span class="nt"></title></span>
<span class="nt"><thumb></span>https://weepee.tv/img/channels/channel.jpg<span class="nt"></thumb></span>
<span class="nt"><h264></span>https://weepeetv.my-stream.eu/channel/uuid/channeluuid/stream.m3u8<span class="nt"></h264></span>
<span class="nt"></item></span>
<span class="nt"></items></span>
</pre>
</div>
<br />
To run the script change the variables on top of createwptvmxl.pl
<br />
<div class="highlight">
<pre><span class="c1">## change these variables</span>
<span class="k">my</span> <span class="nv">$ACCOUNT</span><span class="o">=</span><span class="s">"uxxxxxx"</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$PASSWORD</span><span class="o">=</span><span class="s">"secret"</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$CURL</span><span class="o">=</span><span class="s">"/usr/bin/curl"</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$XMLFILE</span><span class="o">=</span><span class="s">"wptv.xml"</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">@sort</span><span class="o">=</span><span class="p">(</span><span class="s">"een"</span><span class="p">,</span><span class="s">"canvas"</span><span class="p">,</span><span class="s">"bbc1"</span><span class="p">,</span><span class="s">"bbc2"</span><span class="p">,</span><span class="s">"acht"</span><span class="p">,</span><span class="s">"vtm"</span><span class="p">,</span><span class="s">"2be"</span><span class="p">,</span><span class="s">"vitaya"</span><span class="p">,</span><span class="s">"jim"</span><span class="p">,</span><span class="s">"ketnet"</span><span class="p">,</span><span class="s">"bbcentertainment"</span><span class="p">,</span><span class="s">"kanaalz"</span><span class="p">,</span><span class="s">"vtmKzoom</span><span class="p">,</span><span class="s">"tvllogosmall"</span><span class="p">,</span><span class="s">"livetv"</span><span class="p">);</span></pre>
</div>
<br />
You can find the code on <a href="https://github.com/42wim/wptvscraper">github</a>
<br />
<br />42wimhttp://www.blogger.com/profile/03140786286473099167noreply@blogger.com2tag:blogger.com,1999:blog-2503925492290059.post-77708023590976345622013-01-04T20:55:00.000+01:002013-01-15T22:40:50.858+01:00Android: things learned.<span style="font-size: large;">Things learned this past week developing Android apps:</span><br />
<br />
- <b>Cookies: </b>A cookie isn't just a cookie, there are different RFC's which are apparently not implemented everywhere. More info on <a href="http://developer.android.com/reference/org/apache/http/client/params/CookiePolicy.html">Cookiepolicy</a> and in this <a href="http://goo.gl/Z1lS0">Google search</a> cries of developers trying to workaround broken cookies. Didn't really found a solution using the SDK, so went native and used good old <a href="http://curl.haxx.se/">curl</a><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://cdn.memegenerator.net/instances/250x250/32903489.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://cdn.memegenerator.net/instances/250x250/32903489.jpg" /></a></div>
<br />
<br />
- <b>SSL: </b>Using self-signed certificates on Android is a PITA. (This is good for security ofcourse, but bad if you want to test something on your own development server). After googling and trying out a lot of different solutions, this one was the best/easiest: <a href="https://github.com/Elbandi/Nagroid/blob/master/src/de/schoar/android/helper/http/SSLSelfSigned.java">SSLSelfSigned.java from Nagroid</a><br />
<br />
- <b>Android ports</b>: <a href="http://dan.drown.org/android/">http://dan.drown.org/android/</a> is a cool site if you want your android more unixy.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://i.imgur.com/lPm3a.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://i.imgur.com/lPm3a.jpg" height="240" width="320" /></a></div>
<br />
<br />
- <b>waitFor(): </b>Don't forget Runtime.getRuntime().waitFor() when running a lot of native apps sequentially :)<br />
<br />
<b>- namespaces: </b>Java packages can't have numbers only in the subpackages, e.g be.42.myapp. Booh.<br />
<br />
- <b>DNS: </b>Everything is a fscking DNS problem! :-) DNS resolving for native apps only works when linked to Bionic libc. This doesn't seem to be the case for the stunnel from stunnel.org.<br />
<br />
<br />42wimhttp://www.blogger.com/profile/03140786286473099167noreply@blogger.com0tag:blogger.com,1999:blog-2503925492290059.post-41941279084495661372013-01-03T01:09:00.002+01:002013-01-15T22:41:10.002+01:00Unofficial WeepeeTV android app (beta)Found a workaround for the cookie problem by using a native app (<a href="http://curl.haxx.se/">Curl</a>).<br />
<br />
This allowed me to do the screenscraping with curl and the parsing on android itself.<br />
<br />
For the backend I picked <a href="https://sites.google.com/site/mxvpen/">MXplayer</a>, the nice thing about it is that it has HLS support, so you'll be able to stream WeepeeTV on older Android versions too. - I got it running on my HTC Wildfire S (although very slowly, no hardware decoding).<br />
<br />
Of course this also allows you to have full screen and you're able to change your aspect ratio.<br />
<br />
<i>(The not so nice thing about MXplayer is that it doesn't support https urls, so I included <a href="https://www.stunnel.org/index.html">stunnel</a> to do the https offloading)</i><br />
<br />
Some screenshots when running on my 7" ICS tablet.<br />
<br />
<table>
<tbody>
<tr><td><div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5QEk8SXsWQsoFqIDsnLPwxz2Hsqt-kc2fW84INqSVFLvfYP4fo76CSC6H9tRbJAmxyAvsxW10w6zzSjn4sNPM3wGBrJbJ_DkoYaoyyACEe1B-EFtkv8Auk04YIX7kHNP5qb5-4Im7IQ/s1600/2013-01-03-00-44-55.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5QEk8SXsWQsoFqIDsnLPwxz2Hsqt-kc2fW84INqSVFLvfYP4fo76CSC6H9tRbJAmxyAvsxW10w6zzSjn4sNPM3wGBrJbJ_DkoYaoyyACEe1B-EFtkv8Auk04YIX7kHNP5qb5-4Im7IQ/s1600/2013-01-03-00-44-55.png" height="400" width="251" /></a></div>
</td>
<td><div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEixAfloBM09-Lm6ngzVnVbXoO1_vOLnCPdnN3bXKkqUM74sa7K1iFtdrxmZ9Zx7CDazk_VeUj1GZSoMhqpg_RH0viBoA8semZ6TaZW8t5_-FTGDK2b03QinzzcVoyMfJPc9VlOkesgAUA/s1600/2013-01-03-00-51-20.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEixAfloBM09-Lm6ngzVnVbXoO1_vOLnCPdnN3bXKkqUM74sa7K1iFtdrxmZ9Zx7CDazk_VeUj1GZSoMhqpg_RH0viBoA8semZ6TaZW8t5_-FTGDK2b03QinzzcVoyMfJPc9VlOkesgAUA/s1600/2013-01-03-00-51-20.png" height="221" width="400" /></a></div>
</td></tr>
</tbody></table>
So there still has to be done some polishing, like decent error handling ;-)<br />
<div>
But for now it works for me.</div>
<div>
<br /></div>
<div>
<b>If anyone is interested in beta-testing this contact me on twitter (@42wim)</b></div>
42wimhttp://www.blogger.com/profile/03140786286473099167noreply@blogger.com8tag:blogger.com,1999:blog-2503925492290059.post-68686688457047320872012-12-28T03:44:00.003+01:002013-01-15T22:41:20.873+01:00Invalid cookies on Android & WeepeeTVI'm trying to build an android application that logs in on the weepeetv website and does the parsing/screenscraping directly on android.<br />
<br />
Unfortunately, the weepeetv site is setting cookies with an incorrect Expire date.<br />
<br />
Android does not like this.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>W/ResponseProcessCookies(20386): Invalid cookie header: "Set-Cookie: snip; expires=; path=/;
isSecure=false". Unable to parse expires attribute:
</code></pre>
<br />
<div style="text-align: left;">
I've been trying to use different http implementations: HttpClient versus HttpURLConnection but to no avail.
</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
Tried the different CookiePolicies on <a href="http://developer.android.com/reference/org/apache/http/client/params/CookiePolicy.html">http://developer.android.com/reference/org/apache/http/client/params/CookiePolicy.html</a> </div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
Nothing works.</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEitCukoJxzZIVW5RlJQBNtkkOKKRl5iDE6EAxmjmssV1QR_RDO8mE200gDfa00EM3v1zYojOoWiRg3y2vicQsOFZgPgDhgrRoVnBa1TPqz5uZmuFDZej3rI1ecCHSwTzBMPIupAerIMZw/s1600/3scy3p.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEitCukoJxzZIVW5RlJQBNtkkOKKRl5iDE6EAxmjmssV1QR_RDO8mE200gDfa00EM3v1zYojOoWiRg3y2vicQsOFZgPgDhgrRoVnBa1TPqz5uZmuFDZej3rI1ecCHSwTzBMPIupAerIMZw/s320/3scy3p.jpg" height="240" width="320" /></a></div>
<br />
<br />
So the only solution I found is doing the parsing on Linux, running a webserver with the parsed XML file and providing this to the android application. (or compiling curl for android, but this is too much work for now :)<br />
<br />
This works.<br />
<br />
<div class="separator" style="clear: both; text-align: left;">
<a href="https://pbs.twimg.com/media/A-4zvV4CYAED-5I.png:large" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="https://pbs.twimg.com/media/A-4zvV4CYAED-5I.png:large" width="187" /></a></div>
<br />
The next challenge is actually playing those streaming files. (hint: <b>https</b>)<br />
<br />
<br />
<br />42wimhttp://www.blogger.com/profile/03140786286473099167noreply@blogger.com0tag:blogger.com,1999:blog-2503925492290059.post-46991036967800924352012-12-27T02:44:00.000+01:002013-01-15T22:41:33.698+01:00WeepeeTV and VLC<div class="separator" style="clear: both; text-align: left;">
Started testdriving WeepeeTV this week, bought a subscription for a month (for now).</div>
<div class="separator" style="clear: both; text-align: left;">
You can watch tv (SD quality / H264 encoded) using flash (cpu intensive) or HLS.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Unfortunately there's no API available.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
But with some creative screenscraping and parsing a lot of fun stuff can be done. </div>
<div class="separator" style="clear: both; text-align: left;">
One example below is a VLC plugin which uses my own xml file (created by screenscraping) as a source.</div>
<div class="separator" style="clear: both; text-align: left;">
Very easy way to switch channels without clicking too much or using a browser.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEif30cIWXBsKA8IiOL6k0LUgvJCcZauzJPYFE6gy-L0FN4IaFB9f6BntbjH1i5TxKDINYMFqHr_d5EGrUAtcrGz-NeVXzYNyYeKx8WOjF0t6cQ4yMMYX7nr6ZAgOhMN49T9CGhOc9ndJg/s1600/vlcaddon.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEif30cIWXBsKA8IiOL6k0LUgvJCcZauzJPYFE6gy-L0FN4IaFB9f6BntbjH1i5TxKDINYMFqHr_d5EGrUAtcrGz-NeVXzYNyYeKx8WOjF0t6cQ4yMMYX7nr6ZAgOhMN49T9CGhOc9ndJg/s640/vlcaddon.png" height="427" width="640" /></a></div>
<br />
I will put my xml creating script on github, so you can use it too :-)<br />
<br />
To use this lua script put it in the lua/sd directory and call it weepeetv.lua<br />
Ofcourse you need to change the parse_url to your own server or local file.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>--SD_Description=WeePee TV
--[[
Authors: 42wim
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston MA 02110-1301, USA.
--]]
require "simplexml"
function descriptor()
return { title="WeePee TV" }
end
function main()
local tree = simplexml.parse_url("http://192.168.x.x:8080/input.xml")
for _, items in ipairs( tree.children ) do
simplexml.add_name_maps(items)
local url = vlc.strings.resolve_xml_special_chars( items.children_map['h264'][1].children[1] )
local title = vlc.strings.resolve_xml_special_chars( items.children_map['title'][1].children[1] )
local arturl = vlc.strings.resolve_xml_special_chars( items.children_map['thumb'][1].children[1] )
vlc.sd.add_item( { path = url, title = title , arturl = arturl } )
end
end
</code></pre>
42wimhttp://www.blogger.com/profile/03140786286473099167noreply@blogger.com7tag:blogger.com,1999:blog-2503925492290059.post-14170713689416212432012-10-21T23:44:00.001+02:002012-12-27T02:45:59.529+01:00<img src="http://cdn.memegenerator.net/instances/250x250/28745714.jpg" /><br />
<div>
<br /></div>
<div>
Sums up my experience with vendor$ helpdesks lately</div>
42wimhttp://www.blogger.com/profile/03140786286473099167noreply@blogger.com0tag:blogger.com,1999:blog-2503925492290059.post-50996150119407759692012-10-21T23:37:00.003+02:002012-10-21T23:37:27.322+02:00First postFirst blog post, this blog will be mostly about technical stuff and rants ;-)42wimhttp://www.blogger.com/profile/03140786286473099167noreply@blogger.com0