Wednesday, December 23, 2015

HOWTO STRUCTURE MANY PROJECTS IN A SINGLE REPOSITORY - PART 3

Overview

At this point in time you have a active GIT Server that is working well for your team. You and your team have been creating projects that can be centrally shared and viewed by other team members. You can even browse the source using a web browser.

New projects can be initially setup and then pushed onto the server pretty easily as well.

But now you are running into a problem. As the number of projects increases, so does the list of projects on the central GIT Server. This is pretty manageable at a client level since they clone only what is needed but on the server, the flat directory structure is confusing and hard to manage.

The goal of this HOWTO is to show the process for cleaning up your directory structure on the GIT server to allow some form of logical grouping. 

In my example, I just want to group together by purpose allowing a new tools area. The way things are grouped does not matter and can be done any way you choose. The technique remains the same.

Fine Print

I am not a security expert nor am I a complete GIT expert. This HOWTO does its best to get you up and running with GIT so your team can get to work.

Assumptions

I am doing this using 

  • Ubuntu Linux 15.10
  • GIT based on the Ubuntu distribution version
  • GITWEB based on the Ubuntu distribution version
  • PERL based on the Ubuntu distribution version
  • GITOLITE based on a git clone from github.com
  • You already have the Ubuntu distribution version of Apache2 installed and running.

For this document a project called mr_line_number will be moved using uuklanger as an administrator.

My Ideal Setup

Part Description
Server Name atomserver
Access Method SSH
GITWEB http://atomserver/gitweb/
User Management GITOLITE
REPO Base /opt/data/git/repositories/
GIT User git
GIT User Home /home/git/

The Scenario

Some time ago, mr_line_number was added to the central GIT server to allow other developers to enjoy the benefits of this tool. However, tools like this are now mixed in with a long list of other projects where some are production code.

To help avoid confusion, it has been decided to split the projects up into logical grouping

  • Products
  • Tools
  • Support

The venerable mr_line_number will be the first to be moved to the new tools area. All users of this will need re-pointed so their remote note uses the new location.

High Level Procedure

To make this work, the following will need done

  1. Create a directory for the new logical grouping and move mr_line_number into it
  2. Set the gitolite permission/config to add tools to the affected project
  3. Update the projects.list to add the new relative path
  4. Provide the command to update the remote of each user affected.

Once done, pushing or pulling changes from a client to the new remote location will be seamless. There is no need to “re-clone” the project.

Server Side Move of Project

As the git user, the mr_line_number project will need to be moved into a new folder called tools

$ sudo su - git
$ cd /opt/data/git/repositories/
$ mkdir tools
$ mv mr_line_number tools

Once completed, you will have one less project in your repositories base directory. The project will now be in tools.

Setup Gitolite Re-Point to the new project location

As the user who administers gitolite, a small change will need to be made to re-point gitolite and those who have permissions to see it, to a the new tools directory.

This assumes that the gitolite setup followed PART 1 of this series.

$ cd $HOME/Admin/git/gitolite-admin/conf
$ vi gitolite.conf
$ cat gitolite.conf 
repo gitolite-admin
    RW+     =   uuklanger

repo testing
    RW+     =   @all

repo tools/mr_line_number 
    RW+     =   uuklanger
    R       =   gitweb
$ git commit -m “Moving mr_line_number into the tools directory” gitolite.conf
$ git push origin master

In front the line mr_line_number was changed to be tools/mr_line_number. The changes are then committed and pushed.

IF YOU FORGET THIS STEP or do it wrong, you will see a new repository created in the repository base directory that is empty. During push, you should not see any message about something getting initialized or created.

Once completed, you should only see mr_line_number in <GIT_REPO_BASE>/tools/.

projects.list Update

Back as the git user, the projects.list will need updated to point to the new tools directory.

$ sudo su - git
$ vi projects.list
$ cat projects.list
tools/mr_line_number.git
testing.git

Now GITWEB will know where to look for mr_line_number among other projects.

Checks so far

If you visit the server’s URL for gitweb, you will see tools/mr_line_number as a link you can click on vs just mr_line_number

Click on the new link and confirm you can navigate the tree and see everything expected.

There is a possibility that Apache may need restarted if you do not see tools in front of mr_line_number.

Update the end users to re-point their remote to the new location

Each and every remote user who currently has this project cloned will still be pointed to the old location. If they try to push changes, it will not go very well for them. 

The good part is that re-pointing each remote is very easy and does not require re-clone or other scary stuff.

This is done on the client’s system that points the remote updated above.

$ cd work/mr_line_number
$ git remote -v
origin git@atomserver:mr_line_number.git (fetch)
origin git@atomserver:mr_line_number.git (push)
$ git remote set-url origin git@atomserver:tools/mr_line_number.git
$ git remote -v 
origin git@atomserver:tools/mr_line_number.git (fetch)
origin git@atomserver:tools/mr_line_number.git (push)

Once completed, the next git push origin master will successfully flow to the new location on atomserver

The changes pushed should be visible in GITWEB as well.

New Clone

If a user new to this tool decides to clone it after this operation is completed they will need to ensure the path includes tools. Assuming that this user keeps their clones in a work directory…

$ cd work
$ git clone git@atomserver:tools/mr_line_number.git
$ git remote -v
origin git@atomserver:tools/mr_line_number.git (fetch)
origin git@atomserver:tools/mr_line_number.git (push)

Once completed, the project should be cloned normally and the remote should be set as expected.

CONCLUSION

Following the above general technique, it should be possible for a GIT administrator to be able to re-structure project repos logically for easier navigation and maintenance. 

Created 23-December–2015 by uuklanger

Sunday, December 13, 2015

HOWTO PUSH A PROJECT TO A REMOTE GIT REPOSITORY - PART 2

Overview

Using GIT for core work like

  • Creating a new local repo
  • Adding files to it
  • Committing your changes
  • Looking at the history

Is reasonably straight forward and requires some reading and practice. The problem is, what do you do if you want to push your code to a server that other people can access. Imagine that you started a project local on your workstation and are now ready to share it with your team or other interested parties.

The goal of this document is to show how to take a local GIT repo and push it to a remote location. For this to work, you will need

  • A GIT repo that is remote from your local home directory. That does not mean on a separate physical server but it needs to be a repo accessable through SSH or HTTP using something like GITOLITE.
  • Your client side already has permissions to GIT on the target server.
  • Permissions of RW+ to the new git repo. This allows you full admin to the repository
  • A project that you are willing to push.

Fine Print

I am not a security expert nor am I a complete GIT expert. This HOWTO does its best to get you up and running with GIT so your team can get to work.

Assumptions

I am doing this using 

  • Ubuntu Linux 15.10
  • GIT based on the Ubuntu distribution version
  • GITWEB based on the Ubuntu distribution version
  • PERL based on the Ubuntu distribution version
  • GITOLITE based on a git clone from github.com
  • You already have the Ubuntu distribution version of Apache2 installed and running.

For this document a project called mr_line_number will be added using uuklanger as an administrator.

My Ideal Setup

Part Description
Server Name atomserver
Access Method SSH
GITWEB http://atomserver/gitweb/
User Management GITOLITE
REPO Base /opt/data/git/repositories/
GIT User git
GIT User Home /home/git/

Permission on the Server Side

For a push to any server, you will need two things. 

  • Permissions to access the server.
  • Administrative permissions to create a new repo

The first is assumed where your public key has been added to the server and you can see projects like “testing.git”.

To add permissions 

  1. Connect to the server as your GIT administrator
  2. Navigate to the gitolite-admin project which is located in ~/Admin/git/ for my configuration
  3. Add the new project mr_line_number and uuklanger with RW+ permissions. Also add gitweb with R permissions.
  4. git commit the changes
  5. git push the changes to the main server
$ cd $HOME/Admin/git/gitolite-admin/conf
$ vi gitolite.conf
$ cat gitolite.conf 
repo gitolite-admin
    RW+     =   uuklanger

repo testing
    RW+     =   @all

repo mr_line_number 
    RW+     =   uuklanger
    R       =   gitweb
$ git commit -m “adding mr_line_number” gitolite.conf
$ git push

Once completed, mr_line_number will be a new project for git to manage through gitolite.

GITWEB Project List Configuration

It is likely that you will want your new project to be navigatable using GITWEB for quick searches/reviews. For this to work, the GITWEB projects.list will need to be configured. This will need to be done on your remote server.

$ sudo su - git
$ vi $HOME/projects.list
$ cat $HOME/projects.list
testing.git
mr_line_number.git

Once done you should have your new project visible in projects.list file for the “git” user.

Add the New Project to the Remote Server

Based on the status of your project mr_line_number, everything should be ready to go where 

  1. The project is managed by GIT (locally for now)
  2. All has been added/staged in GIT
  3. All code has been committed to GIT

What remains is simply the push to the remote repository. The following is assumed.

  • git is the SSH user with access to the GIT repositories
  • atomserver is the target remote server
  • mr_line_number.git is the name of the project on the remote server
  • ~/work/mr_line_number (without the .git) is the name of the local workspace directory.
$ cd ~/work/mr_line_number
$ git remote add origin git@atomserver:mr_line_number.git
$ git remote -v
$ git push -u origin master

Since this was configured earlier, if you visit your gitweb site, you should now see mr_line_number in the list of GIT managed projects.

The parameter of origin is the remote name. You can name it something else as well. After the remote is added, you can view it in -v verbose mode. 

In the future you may add other remotes for the project. In that case you will just change origin to something else like review if (for example) you have a team that reviews changes.

Where the -u tells git to use this server for upstream requests.

Getting New Project onto remote client

Once central to a team, it it likely that someone else may want to gain access to this project. For a user to do so, they will need to clone the project from GIT. In this case they want the project in their local system where they already have permissions to access GIT and this GIT project (via the gitolite.conf edited earlier but will need updated for this new user following the same procedure).

This user wants the project checked out into their work directory.

$ cd $HOME/work
$ git clone git@atomserver:mr_line_number.git

Once completed, you will see a new $HOME/work/mr_line_number/ with the expected contents.

Making and committing changes

If the user makes a change that are ready for the world they can be pushed to the central GIT

$ git push origin master

Note that the “-u” that tells git to use this server for upstream requests is missing. This is because the code was originally pulled from a central server so GIT already knows this. You can, however, change this if origin needs to become something else.

Conclusions

Using git it is possible to have a full source control system that you can use for managing end-to-end development cycles. When you are ready to share, you can push your changes to a central server for all to enjoy. 

What is nice about a DVCS is that you can do a lot of local experimentation in branches without impacting other users. 

This HOWTO will hopefully show how to push changes to a central location once the code is tablized and ready for the real world.

Created 20-November–2015 by uuklanger

Updated 23-December–2015 by uuklanger

HOWTO SETUP GIT WITH GITOLITE AND GITWEB - PART 1

HOWTO SETUP GIT WITH GITOLITE AND GITWEB - PART 1

Overview

GIT is a pretty powerful version control system, however, unlike Subversion; it is more of a pain to setup for a collaborative team. The goal of this HOWTO is to show the steps involved in setting up 

  • Base GIT install
  • GITOLITE for multi user management/administration
  • GITWEB for an HTTP GIT Repo Browser

Fine Print

I am not a security expert nor am I a complete GIT expert. This HOWTO does its best to get you up and running with GIT so your team can get to work.

Help

I found this completely awesome article to be the best resource for covering 90% of what I wrote this to smooth things out for my specific setup. If my procedure does not make sense, fall back on this one.

Assumptions

I am doing this using 

  • Ubuntu Linux 15.10
  • GIT based on the Ubuntu distribution version
  • GITWEB based on the Ubuntu distribution version
  • PERL based on the Ubuntu distribution version
  • GITOLITE based on a git clone from github.com
  • You already have the Ubuntu distribution version of Apache2 installed and running.

My Ideal Setup

Part Description
Server Name atomserver
Access Method SSH
GITWEB http://atomserver/gitweb/
User Management GITOLITE
REPO Base /opt/data/git/repositories/
GIT User git
GIT User Home /home/git/

GIT

The process for installing GIT involves installing a package and creating a user that will internally manage GIT once exposed to external systems.

Install GIT

$ sudo apt-get install git 

Create a GIT user with NO Password

This user will be the user associated with the GIT REPO and will proxy between external clients and the central REPO. For this reason, it needs to be in the SSH group.

$ sudo adduser \ 
--system \ 
--shell /bin/bash \ 
--gecos 'git version control' \ 
--group \ 
--disabled-password \ 
--home /home/git \ 
git 
$ sudo adduser git ssh 

Create Location for REPO

We don’t really want to store the world’s code in a home directory so next we want to create a location that we can consider to be repo base. This assume you already have an /opt/data/ directory owned by someone.

$ sudo mkdir /opt/data/git
$ sudo mkdir /opt/data/git/repositories
$ sudo chown -R git:git /opt/data/git

Later on this repositories directory will be pointed to by a default directory in /home/git.

Checks

Ensure you can become GIT and have a /home/git directory that git owns.

$ sudo su -l git 
$ pwd
$ ls -l /home
$ ls -l /opt/data

If all goes well you will see a git folder within /home/ that is owned by git and is in the git group. Same with /opt/data/

The foundation part is now done. Next step will allow your team to see the repo.

GITOLITE

GITOLITE is a proxy between GIT users and your managed GIT REPO. It support a great deal of things, however, all I need is SSH access into my REPOs. 

Install

It is completely possible to get GITOLITE from yum or apt-get but you would be insane to do this since there are a few versions and the older version are a bit of a nightmare to setup based on my experience. The best approach I found in my research was to simply use the official one right off of github.

Clone GITOLITE from GITHUB

From the git user’s home directory, clone GITOLITE and then create a bin directory that will link an executable later in this process.

$ sudo su -l git 
$ cd /home/git 
$ git clone git://github.com/sitaramc/gitolite 
$ mkdir $HOME/bin 

Once done, you will see a new gitolite directory and bin directory. 

Launch the GITOLIATE Setup

GITOLITE configures a few things and then creates a symlink from ./bin/ to the gitolite install directory

$ sudo su - git
$ $ cd $HOME
$ gitolite/install -ln 
$ ls -l bin/gitolite 

Once done, you will see a bin/gitolite symlink.

SYMLINK the main repositories to your GITOLITE HOME

Earlier a directory was created to house the git repositories. This was put outside of the git users home to allow more flexibility in moving it around. To ensure GITOLITE can see it, a symlink will be needed.

$ sudo su - git
$ cd $HOME
$ ln -s /opt/data/git/repositories/ repositories

Once completed, you will now have a new symlinked folder in the git user’s home directory for repositories.

Create RSA Key for the GIT Adminitrator

An initial RSA key will be needed for the main GIT administrator to work properly. This should be a local user on the server although other remote users can be added later on.

If you are using linux or an OS that already has .ssh installed, you will probably have $HOME/.ssh directory with an id_rsa.pub file. 

If not, create an RSA key as yourself, using the following command. When asked for a passphrase, just press enter. 

$ ssh-keygen -t rsa

Using your personal user (aka uuklanger in my case), copy it into the /tmp directory so it can be picked up by git and stored in the GITOLITE keydir.

$ cp ~/.ssh/id_rsa.pub /tmp/uuklanger.pub
$ sudo chown git:git /tmp/uuklanger.pub

Once completed, you will have your first Administrator GIT user key ready to go. This is the first step needed before GIT administration is possible.

Initialize gitolite_admin project

In order to be able to administer the GIT site, an administrator will need to be setup. The user will be identifed by their public key. In this case, uuklanger

$ sudo su - git
$ cd $HOME
$ bin/gitolite setup -pk /tmp/uuklanger.pub 
$ rm /tmp/uuklanger.pub

When the gitolite setup is run, you will see various output showing things being created and initialized. 

Once completed, you will have the first official user configured in GITOLITE as an administrator. 

Administrator gitolite-admin clone

The git user .gitolite area should not be touched. It will need to be managed by your new administrator where changes can be pushed to the main gitolite repository.

For this to work, you will need your own copy of the gitolite-admin project. To get this you will need to operate as the git administrator for this server.

$ mkdir $HOME/Admin
$ mkdir $HOME/Admin/git 
$ cd $HOME/Admin/git
$ git config --global user.name “UU KLanger” 
$ git config --global user.email “uuklanger@sandwich.net“ 
$ git config --list 
$ git clone git@localhost:gitolite-admin.git 

Once compelted, you should have a ~/Admin/git/gitolite-admin/ folder with conf/ keydir/ in it. This is where GIT will be managed.

The ~git/.gitolite/ folder should never be touched unless there is a very good reason to

If you look at your ~/Admin/git/gitolite-admin/conf/gitolite.conf file you should see something like

repo gitolite-admin
    RW+     =   uuklanger

repo testing
    RW+     =   @all

Where uuklanger is my administrator for this GIT server.

Prepare for GITOLITE Users

It is very likely that you will access GIT from more then one machine as yourelf. However, GITOLITE indentifies you by your RSA key. To manage this, you can nest subdirectories for each platform that will connect to GITOLITE. For now, localhost is a good start.

$ cd $HOME/Admin/git/gitolite-admin/keydir
$ mkdir local
$ mkdir macbookpro

In this directory you can put all user keys that are local to this server. If you create a remote user create a directory for that client’s keys. GITOLITE will recurse the tree to validate the key.

User Setup

In order for gitolite to work securely, you will need an SSH key per user. The public key will be needed and can be easily created if it is not already in the $HOME/.ssh/. 

Each user will need to submit a public key to the administrator who will place it in the ~/Admin/git/gitolite-admin/keydir/ directory. If a subdirectories were created per client, the key will need to be moved there and set with 750 permissions.

For a client to create a key, they can follow the same process used before, run the following command being sure to press enter when asked for a password.

$ ssh-keygen -t rsa

id_rsa.pub file will just need to be sent to the local admin who can put it in the keydirdirectory. Say we need to add “newguy” to git who put the key in the /tmp directory

$ cd ~/Admin/git/gitolite-admin/keydir
$ cp /tmp/newguy.pub .
$ chmod 750 newguy.pub
$ git add newguy.pub 
$ cd ~/Desktop/gitolite-admin/conf
$ git add gitolite.conf
$ git commit -m “Adding new guy”
$ cd ..
$ git push origin master 
$ sudo rm /tmp/newguy.pub

As project are added, gotlite.conf can be edited to give permissions to users for a given project. It is even possible to give other users admin permissions on gitolite-admin.

Checks

The git push origin master command will commit the changes into gitolite. The changes should be visible in ~git/.gitolite/.

You should have something like this for permissions

uuklanger@atomserver:/home/git$ ls -al
total 48
drwxrwxr-x 2 git  git  4096 Aug 15 15:14 bin
drwxrwxr-x 6 git  git  4096 Aug 15 15:12 gitolite
drwx------ 6 git  git  4096 Aug 15 22:00 .gitolite
-rw------- 1 git  git  6998 Aug 15 15:28 .gitolite.rc
-rwxrwxr-x 1 git  git    12 Aug 24 08:24 projects.list
lrwxrwxrwx 1 git  git    27 Aug 15 17:53 repositories -> /opt/data/git/repositories/
drwx------ 2 git  git  4096 Aug 24 08:24 .ssh

to ensure that outside processes can see.

GITWEB

In order to allow someone to browse a GIT repo without a full GIT install, GITWEB has been created. This tool lets you look around and navigate your source tree.

For this to work you will need apche2 installed with cgi support enabled.

Install

Running UBUNTU, gitweb can be installed using apt

$ sudo apt-get install gitweb

This will provide the core install in /usr/share/gitweb/ along with 

  • /etc/gitweb.conf
  • /etc/apache2/conf-{available, enabled}/gitweb.conf

With the base setup, the configuration is needed next.

Configuration of Apache

Edit /etc/apache2/conf-available/gitweb.conf

Alias /gitweb /usr/share/gitweb

<Directory /usr/share/gitweb>
    SetEnv  GITWEB_CONFIG  /etc/gitweb.conf
    DirectoryIndex gitweb.cgi
    Allow from all
    AllowOverride all
    Order allow,deny
    Options +ExecCGI +FollowSymLinks
    AddHandler cgi-script .cgi
    <Files gitweb.cgi>
      SetHandler cgi-script
    </Files>
    RewriteEngine on
    RewriteRule ^[a-zA-Z0-9_-]+.git/?(\?.)?$ /gitweb.cgi%{REQUESTURI} [L,PT]
</Directory>

Ensure that this configuration file is symlinked in ../conf-enabled/gitweb.conf

Configuration of gitweb

Edit /etc/gitweb.conf to to look as follows. Ensure that the extra attention is paid to $projectroot

# path to git projects (<project>.git)
$projectroot = "/home/git/repositories/";

# directory to use for temp files
$git_temp = "/tmp";

# target of the home link on top of all pages
#$home_link = $my_uri || "/";

# html text to include at home page
#$home_text = "indextext.html";

# file with project list; by default, simply scan the projectroot dir.
#$projects_list = $projectroot;
$projects_list = "/home/git/projects.list";

# stylesheet to use
@stylesheets = ("/static/gitweb.css");

# javascript code for gitweb
$javascript = "/static/gitweb.js";

# logo to use
$logo = "/static/git-logo.png";

# the 'favicon'
$favicon = "/static/git-favicon.png";

# git-diff-tree(1) options to use for generated patches
#@diff_opts = ("-M");
@diff_opts = ();

In addition, as the user git, the .gitolite.rc file will need to be updated to change the UMASK.

$ sudo su - git
$ cd $HOME
$ vi .gitolite.rc

Near the top of the file, there will be a UMASK configuration. The default 0077 but should be set to 0027.

Once done, the gitweb.conf file will be pointing to the correct locations for things like the project root and list.

Visible Repos using projects.list

The repos that are visible can be managed in the the git user home.

An example project list

$ sudo su - git
$ cd $HOME
$ cat projects.list
testing.git

As you can see, we only have one project in this list. If you need to add more then simply have a single line project.

$ sudo su - git
$ cd $HOME
$ vi projects.list

Once completed, you should have a projects.list that works.

Project Repositories Permissions

The directory where the repositories are stored will need their permissions confirmed. Best way is to force them to be as you wish.

$ sudo chmod +R 750 /opt/data/git/repositories

When new projects are added, this level of permissions will be set for each new *.git created automatically based on the UMASK set earlier. To ensure that apache2 can see the repos, www-data will needed added to the git group.

$ sudo adduser www-data git

To test to see if gitweb works, try the following where we are assuming gitweb is installed /usr/share/gitweb/ …

$ sudo su - git
$ /usr/share/gitweb/gitweb.cgi

If everything works well, you should see your projects listed in the returned HTML. 

Since Apache runs as a different user, the only true way to confirm this works is using Apache. For that to happen all the configuraiton will need activated. 

Activation

After all configuration is completed apache will need to be restarted.

$ sudo service apache2 restart

Apache should restart without any errors. 

Checks

Once restarted it should be possible to navigate to 

http://HOSTNAME/cgi-bin/gitweb/

or 

http://HOSTNAME/gitweb/

If you get back your project list, then you are in and should be able to fully navigate your code trees based on the repository permissions (r-x for world) and what is stored in the projects.list file for the git user.

Conclusions

Based on this you will hopefully have a base to start with for GIT. If you do not understand why a step was done above, research it to ensure it is understood and agreed. 

Created 20-November–2015 by uuklanger

Updated 06-December–2015 by uuklanger

Updated 23-December–2015 by uuklanger