Poorly
this post was submitted on 07 Sep 2023
43 points (92.2% liked)
Selfhosted
40313 readers
385 users here now
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
-
Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
-
No spam posting.
-
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
-
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
-
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
-
No trolling.
Resources:
- selfh.st Newsletter and index of selfhosted software and apps
- awesome-selfhosted software
- awesome-sysadmin resources
- Self-Hosted Podcast from Jupiter Broadcasting
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
founded 1 year ago
MODERATORS
What does your economic status have to do with creating docker images?
My code ain't the only thing that's broke
For the littering part, just type crontab -e and add the following line:
@daily docker system prune -a -f
Careful this will also delete your unused volumes (not attached to a running container because it is stopped for whatever reason counts as unused). For this reason alone, always use bind mounts for volumes you care about.
Yes.
All my self hosted containers are bound to some volume (since they require reading settings or databases).
as a user with root permission or as root ?
You shouldn't need sudo to run docker, just can create a docker group and add your user to it. This will give you the steps on how to run docker without sudo.
Edit: as pointed out below, please make sure that you're comfortable with giving these permissions to the user you're adding to the docker group.
run docker without sudo.
Doing that, you effectively give the user account root access without password
docker run --volume /etc:/host_etc debian /bin/bash -> can read/write anything below the host's /etc directory, including shadow file, etc.
True.
But I assume OP was already running docker from that user, so they are comfortable with those permissions.
Maybe should have made it clearer. Added to my other post. Thanks!
Genuinely curious, what would the advantages be?
Also, what if the Linux distro does not have systemd?
The chances I am going to manage a linux distro without systemd are low, but some systems (arch for example) don't have cron out of the box.
Not that big of a deal since it's easy to translate them all, but that's one of the reasons why I default to systemd/timer units.
I was just making a meme dude. Personally, I like systemd, it's more complicated to learn, I ended up reading books to really learn it properly. There's 100% nothing wrong with cron.
One of the reasons I like timers is journalctl integration. I can see everything in one place. Small thing.
I use Gitea and a Runner to build Docker images from the projects in the git repo. Since I'm lazy and only have one machine, I just run the runner on the target machine and mount the docker socket.
BTW: If you manage to "litter your system with docker related files" you fundamentally mis-used Docker. That's exactly what Docker is supposed to prevent.
Self hosting your own CI/CD is the key for OP. Littering is solved too because litter is only a problem on long running servers, which is an anti-pattern in a CI/CD environment.
I already have Forgejo (soft-fork of Gitea) in a Docker container. I guess I need to check how I can access that exact same Docker server where itself is hosted …
With littering I mean several docker dotfiles and dotdirectories in the user’s home directory and other system-wide locations. When I installed Docker on my local computer it created various images, containers, and volumes when created an image.
This is what I want to prevent. Neither do I want nor do I need a fully-featured Docker environment on my local computer.
Maybe you should read up a bit about how docker works, you seem to misunderstand a lot here.
For example the "various images" are kind of the point of docker. Images are layered, and each layer is its own image, so you might end up with 3 or 4 images despite only building one image.
This is something you can't really prevent. It's just how docker works.
Anyway, you can mount the docker socket into a container, and using that socket you can then build an image within the running container. That's essentially how most ci/cd systems work.
You could maybe look into podman and buildah, as far as I know, these can build images without a running docker daemon. That might be a tad "cleaner", but comes with other problems (like no caching).
I have no problem with Docker creating several images and containers and volumes for building a single-image application. The problem is that it does not clean up afterwards and leaves me with multiple things I don’t need for anything else.
I also don’t care about caching or any “magic” stuff. I just ideally want to run one command (or script doing it for me) to build an image resulting in just this one image without any other traces left. … I just like a clean environment and the build process ideally being self-contained.
But I’ll look into your suggestions, thanks!
I seriously don't understand what leftovers you're talking about.
You essentially have a Dockerfile that describes how you want to build your image, you run docker build with the path of your Dockerfile and the path of the context, and the rest is completely up to you. Docker does not leave that many traces around - only the built images within docker itself, but as I said, that's the point of building them.
You can even export the image into a tar file and run docker prune afterwards, that should only leave the exported tar file.
When I built an image last time there were several unused other images with just hashes as names and two unused volumes, also multiple cache files and other files in the user’s home directory in various subfolders.
It's very possible they weren't unused.
Docker builds their images out of layers, and all the layers are used during runtime!:
https://sweetcode.io/understanding-docker-image-layers/
The idea is that you can essentially change PARTS of an image, without rebuilding it entirely, which saves space and bandwidth.
Do you mean that you want to build the docker image on one computer, export it to a different computer where it's going to run, and there shouldn't be any traces of the build process on the first computer? Perhaps it's possible with the --output option.. Otherwise you could write a small script which combines the commands for docker build, export to file, delete local image, and clean up the system.
I want to export the image to my repository/registry and then use it somewhere else. I also don't want to set up a complete docker environment with all the "magic" things. Just build an image and upload it.
Gitlab has a great set of CI tools for deploying docker images, and includes an internal registry of images automatically tied to your repo and available in CI.
I build, configure, and deploy them with nix flakes for maximum reproducibility. It’s the way you should be doing it for archival purposes. With this tech, you can rebuild any docker image identically to today’s in 100 years.
https://youtu.be/0uixRE8xlbY?si=NIIFyzRhXDmcU8Kh
and here’s a link to a blog post, showing how to create a docker image and rust dev environment.
https://johns.codes/blog/rust-enviorment-and-docker-build-with-nix-flakes
I knew you were going to mention nix before reading you post.
::Robert Redford nodding gif::
For local testing: build and run tests on whatever computer I'm developing on.
For deployment: I have a self hosted gitlab instance in a kubernetes cluster. It comes with a registry all setup. Push the project, let the cicd pipeline build, test, and deploy through staging into prod.
Nix + dockerTools.
Doesn't even need docker, and if buitt with flakes I don't even have to checkout the repo.
I use drone CI. you can also use woodpecker which is a community fork of drone CI. https://github.com/woodpecker-ci/woodpecker
VM with a docker build environment.
As for "littering", a simple docker system prune -f after a build gets rid of most of it.
I use podman, and the standalone tool “buildah” can build images from dockerfiles, and the tool “skopeo” can upload it to an image repository.
I use portainer, and when I deploy an image, I write a short bash script for it.
- stop the image if running
- pull the image
- run the image
This lets me easily do updates. I have a script for each image I run, it's less than a dozen. They're all from public repositories.
Docker, Jenkins, Docker-in-Docker (dind)
Nowadays, I build them locally, and upload stable releases to registry. I have in the last used GitHub runners to do it, but building locally is just easier and faster for testing.
pycharm + selfhosted docker registry
For development, I have a single image per project tagged "dev" running locally in WSL that I overwrite over and over again.
For real builds, I use pipelines on my Azure DevOps server to build the image on an agent using a remote buildkit container and push it in my internal repository. All 3 components hosted in the same kubernetes cluster.
For private project free gitlab account. At work self hosted gitlab.
For public projects, I use github build pipelines.
For private, I use ansible.
Oh hey.
I've done this in a ton of different ways.
Manually, viis GitLab CI/CD, CI/CD with Kaniko.
My current favourite though is Kubler; I did a write-up for Lemmy a little while ago: https://lemmy.srcfiles.zip/post/32334