Wednesday, December 23, 2015

HOWTO STRUCTURE MANY PROJECTS IN A SINGLE REPOSITORY - PART 3

Overview

At this point in time you have a active GIT Server that is working well for your team. You and your team have been creating projects that can be centrally shared and viewed by other team members. You can even browse the source using a web browser.

New projects can be initially setup and then pushed onto the server pretty easily as well.

But now you are running into a problem. As the number of projects increases, so does the list of projects on the central GIT Server. This is pretty manageable at a client level since they clone only what is needed but on the server, the flat directory structure is confusing and hard to manage.

The goal of this HOWTO is to show the process for cleaning up your directory structure on the GIT server to allow some form of logical grouping. 

In my example, I just want to group together by purpose allowing a new tools area. The way things are grouped does not matter and can be done any way you choose. The technique remains the same.

Fine Print

I am not a security expert nor am I a complete GIT expert. This HOWTO does its best to get you up and running with GIT so your team can get to work.

Assumptions

I am doing this using 

  • Ubuntu Linux 15.10
  • GIT based on the Ubuntu distribution version
  • GITWEB based on the Ubuntu distribution version
  • PERL based on the Ubuntu distribution version
  • GITOLITE based on a git clone from github.com
  • You already have the Ubuntu distribution version of Apache2 installed and running.

For this document a project called mr_line_number will be moved using uuklanger as an administrator.

My Ideal Setup

Part Description
Server Name atomserver
Access Method SSH
GITWEB http://atomserver/gitweb/
User Management GITOLITE
REPO Base /opt/data/git/repositories/
GIT User git
GIT User Home /home/git/

The Scenario

Some time ago, mr_line_number was added to the central GIT server to allow other developers to enjoy the benefits of this tool. However, tools like this are now mixed in with a long list of other projects where some are production code.

To help avoid confusion, it has been decided to split the projects up into logical grouping

  • Products
  • Tools
  • Support

The venerable mr_line_number will be the first to be moved to the new tools area. All users of this will need re-pointed so their remote note uses the new location.

High Level Procedure

To make this work, the following will need done

  1. Create a directory for the new logical grouping and move mr_line_number into it
  2. Set the gitolite permission/config to add tools to the affected project
  3. Update the projects.list to add the new relative path
  4. Provide the command to update the remote of each user affected.

Once done, pushing or pulling changes from a client to the new remote location will be seamless. There is no need to “re-clone” the project.

Server Side Move of Project

As the git user, the mr_line_number project will need to be moved into a new folder called tools

$ sudo su - git
$ cd /opt/data/git/repositories/
$ mkdir tools
$ mv mr_line_number tools

Once completed, you will have one less project in your repositories base directory. The project will now be in tools.

Setup Gitolite Re-Point to the new project location

As the user who administers gitolite, a small change will need to be made to re-point gitolite and those who have permissions to see it, to a the new tools directory.

This assumes that the gitolite setup followed PART 1 of this series.

$ cd $HOME/Admin/git/gitolite-admin/conf
$ vi gitolite.conf
$ cat gitolite.conf 
repo gitolite-admin
    RW+     =   uuklanger

repo testing
    RW+     =   @all

repo tools/mr_line_number 
    RW+     =   uuklanger
    R       =   gitweb
$ git commit -m “Moving mr_line_number into the tools directory” gitolite.conf
$ git push origin master

In front the line mr_line_number was changed to be tools/mr_line_number. The changes are then committed and pushed.

IF YOU FORGET THIS STEP or do it wrong, you will see a new repository created in the repository base directory that is empty. During push, you should not see any message about something getting initialized or created.

Once completed, you should only see mr_line_number in <GIT_REPO_BASE>/tools/.

projects.list Update

Back as the git user, the projects.list will need updated to point to the new tools directory.

$ sudo su - git
$ vi projects.list
$ cat projects.list
tools/mr_line_number.git
testing.git

Now GITWEB will know where to look for mr_line_number among other projects.

Checks so far

If you visit the server’s URL for gitweb, you will see tools/mr_line_number as a link you can click on vs just mr_line_number

Click on the new link and confirm you can navigate the tree and see everything expected.

There is a possibility that Apache may need restarted if you do not see tools in front of mr_line_number.

Update the end users to re-point their remote to the new location

Each and every remote user who currently has this project cloned will still be pointed to the old location. If they try to push changes, it will not go very well for them. 

The good part is that re-pointing each remote is very easy and does not require re-clone or other scary stuff.

This is done on the client’s system that points the remote updated above.

$ cd work/mr_line_number
$ git remote -v
origin git@atomserver:mr_line_number.git (fetch)
origin git@atomserver:mr_line_number.git (push)
$ git remote set-url origin git@atomserver:tools/mr_line_number.git
$ git remote -v 
origin git@atomserver:tools/mr_line_number.git (fetch)
origin git@atomserver:tools/mr_line_number.git (push)

Once completed, the next git push origin master will successfully flow to the new location on atomserver

The changes pushed should be visible in GITWEB as well.

New Clone

If a user new to this tool decides to clone it after this operation is completed they will need to ensure the path includes tools. Assuming that this user keeps their clones in a work directory…

$ cd work
$ git clone git@atomserver:tools/mr_line_number.git
$ git remote -v
origin git@atomserver:tools/mr_line_number.git (fetch)
origin git@atomserver:tools/mr_line_number.git (push)

Once completed, the project should be cloned normally and the remote should be set as expected.

CONCLUSION

Following the above general technique, it should be possible for a GIT administrator to be able to re-structure project repos logically for easier navigation and maintenance. 

Created 23-December–2015 by uuklanger

Sunday, December 13, 2015

HOWTO PUSH A PROJECT TO A REMOTE GIT REPOSITORY - PART 2

Overview

Using GIT for core work like

  • Creating a new local repo
  • Adding files to it
  • Committing your changes
  • Looking at the history

Is reasonably straight forward and requires some reading and practice. The problem is, what do you do if you want to push your code to a server that other people can access. Imagine that you started a project local on your workstation and are now ready to share it with your team or other interested parties.

The goal of this document is to show how to take a local GIT repo and push it to a remote location. For this to work, you will need

  • A GIT repo that is remote from your local home directory. That does not mean on a separate physical server but it needs to be a repo accessable through SSH or HTTP using something like GITOLITE.
  • Your client side already has permissions to GIT on the target server.
  • Permissions of RW+ to the new git repo. This allows you full admin to the repository
  • A project that you are willing to push.

Fine Print

I am not a security expert nor am I a complete GIT expert. This HOWTO does its best to get you up and running with GIT so your team can get to work.

Assumptions

I am doing this using 

  • Ubuntu Linux 15.10
  • GIT based on the Ubuntu distribution version
  • GITWEB based on the Ubuntu distribution version
  • PERL based on the Ubuntu distribution version
  • GITOLITE based on a git clone from github.com
  • You already have the Ubuntu distribution version of Apache2 installed and running.

For this document a project called mr_line_number will be added using uuklanger as an administrator.

My Ideal Setup

Part Description
Server Name atomserver
Access Method SSH
GITWEB http://atomserver/gitweb/
User Management GITOLITE
REPO Base /opt/data/git/repositories/
GIT User git
GIT User Home /home/git/

Permission on the Server Side

For a push to any server, you will need two things. 

  • Permissions to access the server.
  • Administrative permissions to create a new repo

The first is assumed where your public key has been added to the server and you can see projects like “testing.git”.

To add permissions 

  1. Connect to the server as your GIT administrator
  2. Navigate to the gitolite-admin project which is located in ~/Admin/git/ for my configuration
  3. Add the new project mr_line_number and uuklanger with RW+ permissions. Also add gitweb with R permissions.
  4. git commit the changes
  5. git push the changes to the main server
$ cd $HOME/Admin/git/gitolite-admin/conf
$ vi gitolite.conf
$ cat gitolite.conf 
repo gitolite-admin
    RW+     =   uuklanger

repo testing
    RW+     =   @all

repo mr_line_number 
    RW+     =   uuklanger
    R       =   gitweb
$ git commit -m “adding mr_line_number” gitolite.conf
$ git push

Once completed, mr_line_number will be a new project for git to manage through gitolite.

GITWEB Project List Configuration

It is likely that you will want your new project to be navigatable using GITWEB for quick searches/reviews. For this to work, the GITWEB projects.list will need to be configured. This will need to be done on your remote server.

$ sudo su - git
$ vi $HOME/projects.list
$ cat $HOME/projects.list
testing.git
mr_line_number.git

Once done you should have your new project visible in projects.list file for the “git” user.

Add the New Project to the Remote Server

Based on the status of your project mr_line_number, everything should be ready to go where 

  1. The project is managed by GIT (locally for now)
  2. All has been added/staged in GIT
  3. All code has been committed to GIT

What remains is simply the push to the remote repository. The following is assumed.

  • git is the SSH user with access to the GIT repositories
  • atomserver is the target remote server
  • mr_line_number.git is the name of the project on the remote server
  • ~/work/mr_line_number (without the .git) is the name of the local workspace directory.
$ cd ~/work/mr_line_number
$ git remote add origin git@atomserver:mr_line_number.git
$ git remote -v
$ git push -u origin master

Since this was configured earlier, if you visit your gitweb site, you should now see mr_line_number in the list of GIT managed projects.

The parameter of origin is the remote name. You can name it something else as well. After the remote is added, you can view it in -v verbose mode. 

In the future you may add other remotes for the project. In that case you will just change origin to something else like review if (for example) you have a team that reviews changes.

Where the -u tells git to use this server for upstream requests.

Getting New Project onto remote client

Once central to a team, it it likely that someone else may want to gain access to this project. For a user to do so, they will need to clone the project from GIT. In this case they want the project in their local system where they already have permissions to access GIT and this GIT project (via the gitolite.conf edited earlier but will need updated for this new user following the same procedure).

This user wants the project checked out into their work directory.

$ cd $HOME/work
$ git clone git@atomserver:mr_line_number.git

Once completed, you will see a new $HOME/work/mr_line_number/ with the expected contents.

Making and committing changes

If the user makes a change that are ready for the world they can be pushed to the central GIT

$ git push origin master

Note that the “-u” that tells git to use this server for upstream requests is missing. This is because the code was originally pulled from a central server so GIT already knows this. You can, however, change this if origin needs to become something else.

Conclusions

Using git it is possible to have a full source control system that you can use for managing end-to-end development cycles. When you are ready to share, you can push your changes to a central server for all to enjoy. 

What is nice about a DVCS is that you can do a lot of local experimentation in branches without impacting other users. 

This HOWTO will hopefully show how to push changes to a central location once the code is tablized and ready for the real world.

Created 20-November–2015 by uuklanger

Updated 23-December–2015 by uuklanger

HOWTO SETUP GIT WITH GITOLITE AND GITWEB - PART 1

HOWTO SETUP GIT WITH GITOLITE AND GITWEB - PART 1

Overview

GIT is a pretty powerful version control system, however, unlike Subversion; it is more of a pain to setup for a collaborative team. The goal of this HOWTO is to show the steps involved in setting up 

  • Base GIT install
  • GITOLITE for multi user management/administration
  • GITWEB for an HTTP GIT Repo Browser

Fine Print

I am not a security expert nor am I a complete GIT expert. This HOWTO does its best to get you up and running with GIT so your team can get to work.

Help

I found this completely awesome article to be the best resource for covering 90% of what I wrote this to smooth things out for my specific setup. If my procedure does not make sense, fall back on this one.

Assumptions

I am doing this using 

  • Ubuntu Linux 15.10
  • GIT based on the Ubuntu distribution version
  • GITWEB based on the Ubuntu distribution version
  • PERL based on the Ubuntu distribution version
  • GITOLITE based on a git clone from github.com
  • You already have the Ubuntu distribution version of Apache2 installed and running.

My Ideal Setup

Part Description
Server Name atomserver
Access Method SSH
GITWEB http://atomserver/gitweb/
User Management GITOLITE
REPO Base /opt/data/git/repositories/
GIT User git
GIT User Home /home/git/

GIT

The process for installing GIT involves installing a package and creating a user that will internally manage GIT once exposed to external systems.

Install GIT

$ sudo apt-get install git 

Create a GIT user with NO Password

This user will be the user associated with the GIT REPO and will proxy between external clients and the central REPO. For this reason, it needs to be in the SSH group.

$ sudo adduser \ 
--system \ 
--shell /bin/bash \ 
--gecos 'git version control' \ 
--group \ 
--disabled-password \ 
--home /home/git \ 
git 
$ sudo adduser git ssh 

Create Location for REPO

We don’t really want to store the world’s code in a home directory so next we want to create a location that we can consider to be repo base. This assume you already have an /opt/data/ directory owned by someone.

$ sudo mkdir /opt/data/git
$ sudo mkdir /opt/data/git/repositories
$ sudo chown -R git:git /opt/data/git

Later on this repositories directory will be pointed to by a default directory in /home/git.

Checks

Ensure you can become GIT and have a /home/git directory that git owns.

$ sudo su -l git 
$ pwd
$ ls -l /home
$ ls -l /opt/data

If all goes well you will see a git folder within /home/ that is owned by git and is in the git group. Same with /opt/data/

The foundation part is now done. Next step will allow your team to see the repo.

GITOLITE

GITOLITE is a proxy between GIT users and your managed GIT REPO. It support a great deal of things, however, all I need is SSH access into my REPOs. 

Install

It is completely possible to get GITOLITE from yum or apt-get but you would be insane to do this since there are a few versions and the older version are a bit of a nightmare to setup based on my experience. The best approach I found in my research was to simply use the official one right off of github.

Clone GITOLITE from GITHUB

From the git user’s home directory, clone GITOLITE and then create a bin directory that will link an executable later in this process.

$ sudo su -l git 
$ cd /home/git 
$ git clone git://github.com/sitaramc/gitolite 
$ mkdir $HOME/bin 

Once done, you will see a new gitolite directory and bin directory. 

Launch the GITOLIATE Setup

GITOLITE configures a few things and then creates a symlink from ./bin/ to the gitolite install directory

$ sudo su - git
$ $ cd $HOME
$ gitolite/install -ln 
$ ls -l bin/gitolite 

Once done, you will see a bin/gitolite symlink.

SYMLINK the main repositories to your GITOLITE HOME

Earlier a directory was created to house the git repositories. This was put outside of the git users home to allow more flexibility in moving it around. To ensure GITOLITE can see it, a symlink will be needed.

$ sudo su - git
$ cd $HOME
$ ln -s /opt/data/git/repositories/ repositories

Once completed, you will now have a new symlinked folder in the git user’s home directory for repositories.

Create RSA Key for the GIT Adminitrator

An initial RSA key will be needed for the main GIT administrator to work properly. This should be a local user on the server although other remote users can be added later on.

If you are using linux or an OS that already has .ssh installed, you will probably have $HOME/.ssh directory with an id_rsa.pub file. 

If not, create an RSA key as yourself, using the following command. When asked for a passphrase, just press enter. 

$ ssh-keygen -t rsa

Using your personal user (aka uuklanger in my case), copy it into the /tmp directory so it can be picked up by git and stored in the GITOLITE keydir.

$ cp ~/.ssh/id_rsa.pub /tmp/uuklanger.pub
$ sudo chown git:git /tmp/uuklanger.pub

Once completed, you will have your first Administrator GIT user key ready to go. This is the first step needed before GIT administration is possible.

Initialize gitolite_admin project

In order to be able to administer the GIT site, an administrator will need to be setup. The user will be identifed by their public key. In this case, uuklanger

$ sudo su - git
$ cd $HOME
$ bin/gitolite setup -pk /tmp/uuklanger.pub 
$ rm /tmp/uuklanger.pub

When the gitolite setup is run, you will see various output showing things being created and initialized. 

Once completed, you will have the first official user configured in GITOLITE as an administrator. 

Administrator gitolite-admin clone

The git user .gitolite area should not be touched. It will need to be managed by your new administrator where changes can be pushed to the main gitolite repository.

For this to work, you will need your own copy of the gitolite-admin project. To get this you will need to operate as the git administrator for this server.

$ mkdir $HOME/Admin
$ mkdir $HOME/Admin/git 
$ cd $HOME/Admin/git
$ git config --global user.name “UU KLanger” 
$ git config --global user.email “uuklanger@sandwich.net“ 
$ git config --list 
$ git clone git@localhost:gitolite-admin.git 

Once compelted, you should have a ~/Admin/git/gitolite-admin/ folder with conf/ keydir/ in it. This is where GIT will be managed.

The ~git/.gitolite/ folder should never be touched unless there is a very good reason to

If you look at your ~/Admin/git/gitolite-admin/conf/gitolite.conf file you should see something like

repo gitolite-admin
    RW+     =   uuklanger

repo testing
    RW+     =   @all

Where uuklanger is my administrator for this GIT server.

Prepare for GITOLITE Users

It is very likely that you will access GIT from more then one machine as yourelf. However, GITOLITE indentifies you by your RSA key. To manage this, you can nest subdirectories for each platform that will connect to GITOLITE. For now, localhost is a good start.

$ cd $HOME/Admin/git/gitolite-admin/keydir
$ mkdir local
$ mkdir macbookpro

In this directory you can put all user keys that are local to this server. If you create a remote user create a directory for that client’s keys. GITOLITE will recurse the tree to validate the key.

User Setup

In order for gitolite to work securely, you will need an SSH key per user. The public key will be needed and can be easily created if it is not already in the $HOME/.ssh/. 

Each user will need to submit a public key to the administrator who will place it in the ~/Admin/git/gitolite-admin/keydir/ directory. If a subdirectories were created per client, the key will need to be moved there and set with 750 permissions.

For a client to create a key, they can follow the same process used before, run the following command being sure to press enter when asked for a password.

$ ssh-keygen -t rsa

id_rsa.pub file will just need to be sent to the local admin who can put it in the keydirdirectory. Say we need to add “newguy” to git who put the key in the /tmp directory

$ cd ~/Admin/git/gitolite-admin/keydir
$ cp /tmp/newguy.pub .
$ chmod 750 newguy.pub
$ git add newguy.pub 
$ cd ~/Desktop/gitolite-admin/conf
$ git add gitolite.conf
$ git commit -m “Adding new guy”
$ cd ..
$ git push origin master 
$ sudo rm /tmp/newguy.pub

As project are added, gotlite.conf can be edited to give permissions to users for a given project. It is even possible to give other users admin permissions on gitolite-admin.

Checks

The git push origin master command will commit the changes into gitolite. The changes should be visible in ~git/.gitolite/.

You should have something like this for permissions

uuklanger@atomserver:/home/git$ ls -al
total 48
drwxrwxr-x 2 git  git  4096 Aug 15 15:14 bin
drwxrwxr-x 6 git  git  4096 Aug 15 15:12 gitolite
drwx------ 6 git  git  4096 Aug 15 22:00 .gitolite
-rw------- 1 git  git  6998 Aug 15 15:28 .gitolite.rc
-rwxrwxr-x 1 git  git    12 Aug 24 08:24 projects.list
lrwxrwxrwx 1 git  git    27 Aug 15 17:53 repositories -> /opt/data/git/repositories/
drwx------ 2 git  git  4096 Aug 24 08:24 .ssh

to ensure that outside processes can see.

GITWEB

In order to allow someone to browse a GIT repo without a full GIT install, GITWEB has been created. This tool lets you look around and navigate your source tree.

For this to work you will need apche2 installed with cgi support enabled.

Install

Running UBUNTU, gitweb can be installed using apt

$ sudo apt-get install gitweb

This will provide the core install in /usr/share/gitweb/ along with 

  • /etc/gitweb.conf
  • /etc/apache2/conf-{available, enabled}/gitweb.conf

With the base setup, the configuration is needed next.

Configuration of Apache

Edit /etc/apache2/conf-available/gitweb.conf

Alias /gitweb /usr/share/gitweb

<Directory /usr/share/gitweb>
    SetEnv  GITWEB_CONFIG  /etc/gitweb.conf
    DirectoryIndex gitweb.cgi
    Allow from all
    AllowOverride all
    Order allow,deny
    Options +ExecCGI +FollowSymLinks
    AddHandler cgi-script .cgi
    <Files gitweb.cgi>
      SetHandler cgi-script
    </Files>
    RewriteEngine on
    RewriteRule ^[a-zA-Z0-9_-]+.git/?(\?.)?$ /gitweb.cgi%{REQUESTURI} [L,PT]
</Directory>

Ensure that this configuration file is symlinked in ../conf-enabled/gitweb.conf

Configuration of gitweb

Edit /etc/gitweb.conf to to look as follows. Ensure that the extra attention is paid to $projectroot

# path to git projects (<project>.git)
$projectroot = "/home/git/repositories/";

# directory to use for temp files
$git_temp = "/tmp";

# target of the home link on top of all pages
#$home_link = $my_uri || "/";

# html text to include at home page
#$home_text = "indextext.html";

# file with project list; by default, simply scan the projectroot dir.
#$projects_list = $projectroot;
$projects_list = "/home/git/projects.list";

# stylesheet to use
@stylesheets = ("/static/gitweb.css");

# javascript code for gitweb
$javascript = "/static/gitweb.js";

# logo to use
$logo = "/static/git-logo.png";

# the 'favicon'
$favicon = "/static/git-favicon.png";

# git-diff-tree(1) options to use for generated patches
#@diff_opts = ("-M");
@diff_opts = ();

In addition, as the user git, the .gitolite.rc file will need to be updated to change the UMASK.

$ sudo su - git
$ cd $HOME
$ vi .gitolite.rc

Near the top of the file, there will be a UMASK configuration. The default 0077 but should be set to 0027.

Once done, the gitweb.conf file will be pointing to the correct locations for things like the project root and list.

Visible Repos using projects.list

The repos that are visible can be managed in the the git user home.

An example project list

$ sudo su - git
$ cd $HOME
$ cat projects.list
testing.git

As you can see, we only have one project in this list. If you need to add more then simply have a single line project.

$ sudo su - git
$ cd $HOME
$ vi projects.list

Once completed, you should have a projects.list that works.

Project Repositories Permissions

The directory where the repositories are stored will need their permissions confirmed. Best way is to force them to be as you wish.

$ sudo chmod +R 750 /opt/data/git/repositories

When new projects are added, this level of permissions will be set for each new *.git created automatically based on the UMASK set earlier. To ensure that apache2 can see the repos, www-data will needed added to the git group.

$ sudo adduser www-data git

To test to see if gitweb works, try the following where we are assuming gitweb is installed /usr/share/gitweb/ …

$ sudo su - git
$ /usr/share/gitweb/gitweb.cgi

If everything works well, you should see your projects listed in the returned HTML. 

Since Apache runs as a different user, the only true way to confirm this works is using Apache. For that to happen all the configuraiton will need activated. 

Activation

After all configuration is completed apache will need to be restarted.

$ sudo service apache2 restart

Apache should restart without any errors. 

Checks

Once restarted it should be possible to navigate to 

http://HOSTNAME/cgi-bin/gitweb/

or 

http://HOSTNAME/gitweb/

If you get back your project list, then you are in and should be able to fully navigate your code trees based on the repository permissions (r-x for world) and what is stored in the projects.list file for the git user.

Conclusions

Based on this you will hopefully have a base to start with for GIT. If you do not understand why a step was done above, research it to ensure it is understood and agreed. 

Created 20-November–2015 by uuklanger

Updated 06-December–2015 by uuklanger

Updated 23-December–2015 by uuklanger

Friday, March 13, 2015

HOWTO Batch Language Date and Time

Overview

Batch language is super basic but there is one trick BATCH supports for getting the date and time. Looking through many different blogs I was able to assemble this HOWTO.

I use this syntax for a little script that I wrote that to backup/archive various coding directories I have on my workstation. I use Windows Backups but I find I trust a simple BAT file that creates date/time stamped 7z files. I wish Windows had a “time machine” like facility but use what you have vs fuss.
To create a proper date/time stamp you can use the following technique.

Syntax/Technique

If you include something like this in your Windows BAT file, you will get a date and time that can be applied to a filename or similar.
set hr=%time:~0,2%
if "%hr:~0,1%" equ " " set hr=0%hr:~1,1%
set FNAME=ARC_%date:~-4,4%%date:~-10,2%%date:~-7,2%_%hr%%time:~3,2%%time:~6,2%.zip
echo %FNAME%
… you will get a result like this
C:\Users\k.langer>echo %FNAME%
ARC_20150209_115340.zip

Conclusion

You can use this trick when automating backups or similar.
[For More Information http://stackoverflow.com/questions/7727114/batch-command-date-and-time-in-file-name]

HOWTO Setup a PiGlow with PyGlow on a Raspberry Pi B Plus

Overview

There are many articles and blogs out there that show how to use a PiGlow LED GPIO board with a Raspberry Pi but most do not cover everything. Some only describe how to set it up with hardware before the B+ and others get you going but fail on permissions.
My hope is to show all the steps needed to let you run Python code with a PiGlow and the PyGlow.py module.

Security

I am not a security genius, nor do I view this project as security driven. You will need to decide what changes below are reasonable for you. For me, I want the PiGlow to work and be easy to work with. This is the process I used to get there.

What you need

  1. Raspberry Pi B+
  2. Rasparian Wheezy
  3. PiGlow module

kernel setup

For this to work, there will need to be some kernel level changes. This simply tells the kernel to load some required modules.

/boot/config.txt

In order for the i2c module to work, it needs to be enabled. Add or uncomment the following to your config.txt file.
dtparam=i2c1=on

/etc/modules

Ensure that the following two lines are in your modules file.
i2c-dev
i2c-bcm2708

/etc/modprobe.d/raspi-blacklist.conf

Ensure that the following modules are not blacklisted by ensuring these lines are commented out.

# blacklist spi-bcm2708
# blacklist i2c-bcm2708

Reboot

At this point, the kernel will have what it needs to allow you to work with your PiGlow. To activate the new configuration, you need to restart your Raspberry Pi.

Setting up PyGlow Python Module

With the kernel prepped and ready, the next phase is to install the python library needed to interact with your PiGlow. For a Raspberry Pi, I suggest creating a library location where this and other common libraries can be stored for shared use across users and applications. On my RPi, I have them in /opt/lib/python If this path does not exist, create it or something like it. Ensure you can get to it using a normal users and chmod as needed.

Setup a Library area in /etc/bash.bashrc

To ensure all users on the platform see this new library location, it will need to be added to the PYTHONPATH. That can be done either in each users .bashrc file or globally in the /etc/bash.bashrc. Just add the following line.
export PYTHONPATH=/opt/lib/python
For the full approach to work, it will need to be added globally. Within a user’s .bashrc, they can append to this path with other personal lib or module path.

Install PyGlow Module from GIT

To get the lastest PyGlow.py module, run the following commands.
cd /opt/lib/python/
git clone https://github.com/benleb/PyGlow.git

Git should already be installed on your RPi. If not or you don’t like git. You can find a ZIP of the same thing using a few Google searches.
Once completed you should have a PyGlow sub directory within /opt/lib/python/. If you read other documentation, you will be told that you can simply run (or at least attempt to run)
cd PyGlow/examples/
sudo python test.py

But when you do, you get an error that PyGlow cannot be found. This is beause by default, PYTHONPATH is not seen by sudo. This takes us to our last phase of setting this up.

sudo Setup for Python Users

Add PYTHONPATH to /etc/sudoers

To tell sudo that you want to keep PYTHONPATH as an environment variable. Simply edit /etc/sudoers and add (or uncomment) the following line.
Defaults    env_keep += "PYTHONPATH"
Now anything in the global Python path will be available to sudo. I would assume you can have multiple lines for each ENV you want to keep.

PyGlow Error for RPi B+

So, all excited, you re-run the test script

cd $PYTHONPATH
cd PyGlow/examples/
sudo python test.py

Now a new error appears complaining about a revision issue or something like that.

The fix for the PyGlow.py Error


The code for the PyGlow (at the time this was written) was not updated for the B+ (Revision #3) GRUMBLE, but the fix is easy. Edit /opt/lib/python/PyGlow/PyGlow.py and search for the following code….

 # Check what Raspberry Pi version we got
 if rpi.RPI_REVISION == 1:
     i2c_bus = 0
 elif rpi.RPI_REVISION == 2:
     i2c_bus = 1
 else:
     raise PyGlowException(
         self, "Unknown Raspberry Pi hardware revision: %s" %
         (rpi.RPI_REVISION))
 
 
 After the above elif, add the following elif that.  It should be before the else 
 
  elif rpi.RPI_REVISION >= 3:
     i2c_bus = 1


It basically says that any revision after and including 3 should use the #1 bus for ic2. Or at least I suppose.

Testing

Now you re-run, more causiously, the following command.

cd $PYTHONPATH
cd PyGlow/examples/
sudo python test.py

Based on my experience, you should have success. In fact all the example scripts should now work.

Next Steps

There are a lot of groups in linux and it is possible that vs using sudo to run your code you can simply add your user to a group that allows i2c and other modules to work. I’ll have to research that next to see if I can give a user permissions.

This was not much of a goal for me since this device is experimental so sudo is not a worry for me to run. For eample, I like to run the sudo python cpu.py to let the lights show me how much CPU I am using. python-psutil and psutils are needed for that to work so sudo apt-get install python-psutil psutils before running that comand.

Conclusions

With a few steps you now have a working PiGlow with support for PyGlow.py. This same approach should work for other GPIO type components.

Thursday, February 5, 2015

HOWTO Create an SVN Branch from a Tag

Overview

Bugs happen and sometimes you need to create a patch. Best way to do this using SVN is to take an existing tagged version of your code (which you hopefully do before each build) and create a branch off of it.

Approach

For this to happen you need to know which tag to use and what you want to call the branch. In the example below, I am using 20141205_Version_R8–1–0–07 as my tag. Within this location is an SVN copy of what was in trunk at the time the tag was created.

Look at the contents of the tag

To see the contents of the tag, use the following command
> svn list http://svn.homeworld.lan/repos/python/tags/20141205_Version_R8-1-0-07/
You will get a listing similar to what you would see in trunk.

Create the Branch

To create the branch all you will do is tell SVN to copy the contents from the tag into a new (non-existing) target folder. In this case the folder within branches is called 20150205_R8–1–1_Patch. Remember that this folder will be created and filled by SVN automatically so the name of the target folder is all you need to provide.
> svn copy http://svn.homeworld.lan/repos/python/tags/20141205_Version_R8-1-0-07/ \
    http://svn.homeworld.lan/repos/python/branches/20150205_R8-1-1_Patch \
    -m "Creating a branch off of tag 20141205_Version_R8-1-0-07"

Committed revision 630.
Once completed you will see a revision number (630 in this case)

Look at the contents of the new branch

To confirm that the branch worked as you would expect, you can get a listing like you did for the tag.
> svn list http://svn.homeworld.lan/repos/python/branches/20150205_R8-1-1_Patch

Getting to Work

Now that this is done you can checkout the code from the branch into a working directory and get to work solving the worlds problems (certainly not your own).
> mkdir 20150205_R8-1-1_Patch
> cd 20150205_R8-1-1_Patch
> svn checkout http://svn.homeworld.lan/repos/python/branches/20150205_R8-1-1_Patch ./

Conclusion

Using a few commands you can easily create a branch from a tag. If you do not happen to have a tag available, you can branch off a specific revision of your trunk using the revision command line option. But hopefully you will create tags as you go for each build you create or for each release pushed to SQA or any testing group.

HOWTO Create lockfile logic using BATCH language

Overview

The goal of this HOWTO is to show how to manage a lock file in batch language. This is handy for scheduled processes.

Example

@echo off
REM
REM Example of howto use a lockfile in BATCH language
REM

:INIT
CD \BatchJobs
ECHO Current Working Directory
CD

:START
IF EXIST NIGHTLYRUN.LOCK GOTO SKIP_WORK
ECHO Turning on Lock
ECHO ""LOCK"" > NIGHTLYRUN.LOCK
REM Do work here. this is where your program or process goes. 
ECHO Turning off Lock
DEL NIGHTLYRUN.LOCK

GOTO DONE

:SKIP_WORK
ECHO ""This process is already running..""

:DONE

Conclusion

Using this approach you can prevent multiple instances of something running. A possible enhancement is something that can sense a stale lock if there is a crash or something. however this should help for most situations.

HOWTO Python Setup UNICODE Logging

Overview

Python does a very good job handling UNICODE for most modules, however, logging is not one of then. To get around this is easy but a special technique is required.

Common Approach

Normally with logging you pass a file to the logger and then write to it as you need though the logger interface. This approach is fast but does not allow the setting of the file encoding.
import logging
logging.basicConfig(
    level=logging.DEBUG,
    filename=filename,
    filemode="w")

formatter = logging.Formatter('%(asctime)-15s: %(levelname)-7s - %(message)s')
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
ch.setFormatter(formatter)

log = logging.getLogger("MAIN")
log.addHandler(ch)
_logger = log

Improved Approach

To get around this and to ensure you control the formatting, a file stream will need to be created first and then that stream can be passed to logger.
import logging
import logging.handlers
log_fh = open("/logs/my_unicode.log", "w", encoding="utf-8")
logging.basicConfig(
    level=logging.DEBUG,
    format=config.FORMAT)

formatter = logging.Formatter('%(asctime)-15s: %(levelname)-7s - %(message)s')
ch = logging.StreamHandler(log_fh)
ch.setLevel(logging.DEBUG)
ch.setFormatter(formatter)

log = logging.getLogger("MAIN")
log.addHandler(ch)
The log_fh holds the handle to the UTF–8 stream which is passed to the logger StreamHandler. Once set, text will be properly encoded as it is passed to the logger.

Conclusion

Using a slightly altered technique it should be possible to write anything to a python log using the build in logger module. There is no reason to use this technique for all logging even if you do not think UNICODE will be used.

Sunday, February 1, 2015

HOWTO Add/Subtract Days or Time

Objective

Show different database techniques for determining the date in the past (or future). The following will show you HOWTO get three days ago based on CURRENT_TIMESTAMP. All will generate something like this.
CurrentTime ThreeDaysAgo
2015–02–01 11:42:04 2015–01–29 11:42:04
Below you will find information MySQL, SQL Server, SQLITE3, and PostgreSQL.

MySQL

Using MySQL, take a timestamp DB property (like CURRENT_TIMESTAMP for “now” or a property from a table). Then add + or - followed by the keyword INTERVAL. Then a number followed by a unit like day if you (for example) want to add or subtract a certain amount of days. The fllowing code subtracts three days from now.
mysql> SELECT CURRENT_TIMESTAMP as CurrentTime,
         CURRENT_TIMESTAMP - INTERVAL 3 day as ThreeDaysAgo;
For More Information

Microsoft SQL Server

Using SQL Server, the end result is the same but they use a function to do the same thing. Using the DateAdd function, you specify a unit, quantity, and DB property. In the following example I use “now” and subtract three days from it.
SELECT CURRENT_TIMESTAMP as CurrentTime,  
    DateAdd(day,-3,CURRENT_TIMESTAMP) as ThreeDaysAgo;

SQLITE3

For SQLITE, you need two wrapper the date time property in the datetime(…) function with a paramter telling it what manipulations you would like to do. In this case, I want to do subtract three days.
sqlite> SELECT CURRENT_TIMESTAMP as CurrentTime, 
datetime(CURRENT_TIMESTAMP, '-3 days') as ThreeDaysAgo
For More Information

PostgreSQL

I have not had a chace to work this out for postgresql yet.
For Information on this point visit

Conclusions

Using the above syntax, you will be able to manipulate dates to add or subtract time as your code requires.