Wednesday, December 23, 2015

HOWTO STRUCTURE MANY PROJECTS IN A SINGLE REPOSITORY - PART 3

Overview

At this point in time you have a active GIT Server that is working well for your team. You and your team have been creating projects that can be centrally shared and viewed by other team members. You can even browse the source using a web browser.

New projects can be initially setup and then pushed onto the server pretty easily as well.

But now you are running into a problem. As the number of projects increases, so does the list of projects on the central GIT Server. This is pretty manageable at a client level since they clone only what is needed but on the server, the flat directory structure is confusing and hard to manage.

The goal of this HOWTO is to show the process for cleaning up your directory structure on the GIT server to allow some form of logical grouping. 

In my example, I just want to group together by purpose allowing a new tools area. The way things are grouped does not matter and can be done any way you choose. The technique remains the same.

Fine Print

I am not a security expert nor am I a complete GIT expert. This HOWTO does its best to get you up and running with GIT so your team can get to work.

Assumptions

I am doing this using 

  • Ubuntu Linux 15.10
  • GIT based on the Ubuntu distribution version
  • GITWEB based on the Ubuntu distribution version
  • PERL based on the Ubuntu distribution version
  • GITOLITE based on a git clone from github.com
  • You already have the Ubuntu distribution version of Apache2 installed and running.

For this document a project called mr_line_number will be moved using uuklanger as an administrator.

My Ideal Setup

Part Description
Server Name atomserver
Access Method SSH
GITWEB http://atomserver/gitweb/
User Management GITOLITE
REPO Base /opt/data/git/repositories/
GIT User git
GIT User Home /home/git/

The Scenario

Some time ago, mr_line_number was added to the central GIT server to allow other developers to enjoy the benefits of this tool. However, tools like this are now mixed in with a long list of other projects where some are production code.

To help avoid confusion, it has been decided to split the projects up into logical grouping

  • Products
  • Tools
  • Support

The venerable mr_line_number will be the first to be moved to the new tools area. All users of this will need re-pointed so their remote note uses the new location.

High Level Procedure

To make this work, the following will need done

  1. Create a directory for the new logical grouping and move mr_line_number into it
  2. Set the gitolite permission/config to add tools to the affected project
  3. Update the projects.list to add the new relative path
  4. Provide the command to update the remote of each user affected.

Once done, pushing or pulling changes from a client to the new remote location will be seamless. There is no need to “re-clone” the project.

Server Side Move of Project

As the git user, the mr_line_number project will need to be moved into a new folder called tools

$ sudo su - git
$ cd /opt/data/git/repositories/
$ mkdir tools
$ mv mr_line_number tools

Once completed, you will have one less project in your repositories base directory. The project will now be in tools.

Setup Gitolite Re-Point to the new project location

As the user who administers gitolite, a small change will need to be made to re-point gitolite and those who have permissions to see it, to a the new tools directory.

This assumes that the gitolite setup followed PART 1 of this series.

$ cd $HOME/Admin/git/gitolite-admin/conf
$ vi gitolite.conf
$ cat gitolite.conf 
repo gitolite-admin
    RW+     =   uuklanger

repo testing
    RW+     =   @all

repo tools/mr_line_number 
    RW+     =   uuklanger
    R       =   gitweb
$ git commit -m “Moving mr_line_number into the tools directory” gitolite.conf
$ git push origin master

In front the line mr_line_number was changed to be tools/mr_line_number. The changes are then committed and pushed.

IF YOU FORGET THIS STEP or do it wrong, you will see a new repository created in the repository base directory that is empty. During push, you should not see any message about something getting initialized or created.

Once completed, you should only see mr_line_number in <GIT_REPO_BASE>/tools/.

projects.list Update

Back as the git user, the projects.list will need updated to point to the new tools directory.

$ sudo su - git
$ vi projects.list
$ cat projects.list
tools/mr_line_number.git
testing.git

Now GITWEB will know where to look for mr_line_number among other projects.

Checks so far

If you visit the server’s URL for gitweb, you will see tools/mr_line_number as a link you can click on vs just mr_line_number

Click on the new link and confirm you can navigate the tree and see everything expected.

There is a possibility that Apache may need restarted if you do not see tools in front of mr_line_number.

Update the end users to re-point their remote to the new location

Each and every remote user who currently has this project cloned will still be pointed to the old location. If they try to push changes, it will not go very well for them. 

The good part is that re-pointing each remote is very easy and does not require re-clone or other scary stuff.

This is done on the client’s system that points the remote updated above.

$ cd work/mr_line_number
$ git remote -v
origin git@atomserver:mr_line_number.git (fetch)
origin git@atomserver:mr_line_number.git (push)
$ git remote set-url origin git@atomserver:tools/mr_line_number.git
$ git remote -v 
origin git@atomserver:tools/mr_line_number.git (fetch)
origin git@atomserver:tools/mr_line_number.git (push)

Once completed, the next git push origin master will successfully flow to the new location on atomserver

The changes pushed should be visible in GITWEB as well.

New Clone

If a user new to this tool decides to clone it after this operation is completed they will need to ensure the path includes tools. Assuming that this user keeps their clones in a work directory…

$ cd work
$ git clone git@atomserver:tools/mr_line_number.git
$ git remote -v
origin git@atomserver:tools/mr_line_number.git (fetch)
origin git@atomserver:tools/mr_line_number.git (push)

Once completed, the project should be cloned normally and the remote should be set as expected.

CONCLUSION

Following the above general technique, it should be possible for a GIT administrator to be able to re-structure project repos logically for easier navigation and maintenance. 

Created 23-December–2015 by uuklanger

Sunday, December 13, 2015

HOWTO PUSH A PROJECT TO A REMOTE GIT REPOSITORY - PART 2

Overview

Using GIT for core work like

  • Creating a new local repo
  • Adding files to it
  • Committing your changes
  • Looking at the history

Is reasonably straight forward and requires some reading and practice. The problem is, what do you do if you want to push your code to a server that other people can access. Imagine that you started a project local on your workstation and are now ready to share it with your team or other interested parties.

The goal of this document is to show how to take a local GIT repo and push it to a remote location. For this to work, you will need

  • A GIT repo that is remote from your local home directory. That does not mean on a separate physical server but it needs to be a repo accessable through SSH or HTTP using something like GITOLITE.
  • Your client side already has permissions to GIT on the target server.
  • Permissions of RW+ to the new git repo. This allows you full admin to the repository
  • A project that you are willing to push.

Fine Print

I am not a security expert nor am I a complete GIT expert. This HOWTO does its best to get you up and running with GIT so your team can get to work.

Assumptions

I am doing this using 

  • Ubuntu Linux 15.10
  • GIT based on the Ubuntu distribution version
  • GITWEB based on the Ubuntu distribution version
  • PERL based on the Ubuntu distribution version
  • GITOLITE based on a git clone from github.com
  • You already have the Ubuntu distribution version of Apache2 installed and running.

For this document a project called mr_line_number will be added using uuklanger as an administrator.

My Ideal Setup

Part Description
Server Name atomserver
Access Method SSH
GITWEB http://atomserver/gitweb/
User Management GITOLITE
REPO Base /opt/data/git/repositories/
GIT User git
GIT User Home /home/git/

Permission on the Server Side

For a push to any server, you will need two things. 

  • Permissions to access the server.
  • Administrative permissions to create a new repo

The first is assumed where your public key has been added to the server and you can see projects like “testing.git”.

To add permissions 

  1. Connect to the server as your GIT administrator
  2. Navigate to the gitolite-admin project which is located in ~/Admin/git/ for my configuration
  3. Add the new project mr_line_number and uuklanger with RW+ permissions. Also add gitweb with R permissions.
  4. git commit the changes
  5. git push the changes to the main server
$ cd $HOME/Admin/git/gitolite-admin/conf
$ vi gitolite.conf
$ cat gitolite.conf 
repo gitolite-admin
    RW+     =   uuklanger

repo testing
    RW+     =   @all

repo mr_line_number 
    RW+     =   uuklanger
    R       =   gitweb
$ git commit -m “adding mr_line_number” gitolite.conf
$ git push

Once completed, mr_line_number will be a new project for git to manage through gitolite.

GITWEB Project List Configuration

It is likely that you will want your new project to be navigatable using GITWEB for quick searches/reviews. For this to work, the GITWEB projects.list will need to be configured. This will need to be done on your remote server.

$ sudo su - git
$ vi $HOME/projects.list
$ cat $HOME/projects.list
testing.git
mr_line_number.git

Once done you should have your new project visible in projects.list file for the “git” user.

Add the New Project to the Remote Server

Based on the status of your project mr_line_number, everything should be ready to go where 

  1. The project is managed by GIT (locally for now)
  2. All has been added/staged in GIT
  3. All code has been committed to GIT

What remains is simply the push to the remote repository. The following is assumed.

  • git is the SSH user with access to the GIT repositories
  • atomserver is the target remote server
  • mr_line_number.git is the name of the project on the remote server
  • ~/work/mr_line_number (without the .git) is the name of the local workspace directory.
$ cd ~/work/mr_line_number
$ git remote add origin git@atomserver:mr_line_number.git
$ git remote -v
$ git push -u origin master

Since this was configured earlier, if you visit your gitweb site, you should now see mr_line_number in the list of GIT managed projects.

The parameter of origin is the remote name. You can name it something else as well. After the remote is added, you can view it in -v verbose mode. 

In the future you may add other remotes for the project. In that case you will just change origin to something else like review if (for example) you have a team that reviews changes.

Where the -u tells git to use this server for upstream requests.

Getting New Project onto remote client

Once central to a team, it it likely that someone else may want to gain access to this project. For a user to do so, they will need to clone the project from GIT. In this case they want the project in their local system where they already have permissions to access GIT and this GIT project (via the gitolite.conf edited earlier but will need updated for this new user following the same procedure).

This user wants the project checked out into their work directory.

$ cd $HOME/work
$ git clone git@atomserver:mr_line_number.git

Once completed, you will see a new $HOME/work/mr_line_number/ with the expected contents.

Making and committing changes

If the user makes a change that are ready for the world they can be pushed to the central GIT

$ git push origin master

Note that the “-u” that tells git to use this server for upstream requests is missing. This is because the code was originally pulled from a central server so GIT already knows this. You can, however, change this if origin needs to become something else.

Conclusions

Using git it is possible to have a full source control system that you can use for managing end-to-end development cycles. When you are ready to share, you can push your changes to a central server for all to enjoy. 

What is nice about a DVCS is that you can do a lot of local experimentation in branches without impacting other users. 

This HOWTO will hopefully show how to push changes to a central location once the code is tablized and ready for the real world.

Created 20-November–2015 by uuklanger

Updated 23-December–2015 by uuklanger

HOWTO SETUP GIT WITH GITOLITE AND GITWEB - PART 1

HOWTO SETUP GIT WITH GITOLITE AND GITWEB - PART 1

Overview

GIT is a pretty powerful version control system, however, unlike Subversion; it is more of a pain to setup for a collaborative team. The goal of this HOWTO is to show the steps involved in setting up 

  • Base GIT install
  • GITOLITE for multi user management/administration
  • GITWEB for an HTTP GIT Repo Browser

Fine Print

I am not a security expert nor am I a complete GIT expert. This HOWTO does its best to get you up and running with GIT so your team can get to work.

Help

I found this completely awesome article to be the best resource for covering 90% of what I wrote this to smooth things out for my specific setup. If my procedure does not make sense, fall back on this one.

Assumptions

I am doing this using 

  • Ubuntu Linux 15.10
  • GIT based on the Ubuntu distribution version
  • GITWEB based on the Ubuntu distribution version
  • PERL based on the Ubuntu distribution version
  • GITOLITE based on a git clone from github.com
  • You already have the Ubuntu distribution version of Apache2 installed and running.

My Ideal Setup

Part Description
Server Name atomserver
Access Method SSH
GITWEB http://atomserver/gitweb/
User Management GITOLITE
REPO Base /opt/data/git/repositories/
GIT User git
GIT User Home /home/git/

GIT

The process for installing GIT involves installing a package and creating a user that will internally manage GIT once exposed to external systems.

Install GIT

$ sudo apt-get install git 

Create a GIT user with NO Password

This user will be the user associated with the GIT REPO and will proxy between external clients and the central REPO. For this reason, it needs to be in the SSH group.

$ sudo adduser \ 
--system \ 
--shell /bin/bash \ 
--gecos 'git version control' \ 
--group \ 
--disabled-password \ 
--home /home/git \ 
git 
$ sudo adduser git ssh 

Create Location for REPO

We don’t really want to store the world’s code in a home directory so next we want to create a location that we can consider to be repo base. This assume you already have an /opt/data/ directory owned by someone.

$ sudo mkdir /opt/data/git
$ sudo mkdir /opt/data/git/repositories
$ sudo chown -R git:git /opt/data/git

Later on this repositories directory will be pointed to by a default directory in /home/git.

Checks

Ensure you can become GIT and have a /home/git directory that git owns.

$ sudo su -l git 
$ pwd
$ ls -l /home
$ ls -l /opt/data

If all goes well you will see a git folder within /home/ that is owned by git and is in the git group. Same with /opt/data/

The foundation part is now done. Next step will allow your team to see the repo.

GITOLITE

GITOLITE is a proxy between GIT users and your managed GIT REPO. It support a great deal of things, however, all I need is SSH access into my REPOs. 

Install

It is completely possible to get GITOLITE from yum or apt-get but you would be insane to do this since there are a few versions and the older version are a bit of a nightmare to setup based on my experience. The best approach I found in my research was to simply use the official one right off of github.

Clone GITOLITE from GITHUB

From the git user’s home directory, clone GITOLITE and then create a bin directory that will link an executable later in this process.

$ sudo su -l git 
$ cd /home/git 
$ git clone git://github.com/sitaramc/gitolite 
$ mkdir $HOME/bin 

Once done, you will see a new gitolite directory and bin directory. 

Launch the GITOLIATE Setup

GITOLITE configures a few things and then creates a symlink from ./bin/ to the gitolite install directory

$ sudo su - git
$ $ cd $HOME
$ gitolite/install -ln 
$ ls -l bin/gitolite 

Once done, you will see a bin/gitolite symlink.

SYMLINK the main repositories to your GITOLITE HOME

Earlier a directory was created to house the git repositories. This was put outside of the git users home to allow more flexibility in moving it around. To ensure GITOLITE can see it, a symlink will be needed.

$ sudo su - git
$ cd $HOME
$ ln -s /opt/data/git/repositories/ repositories

Once completed, you will now have a new symlinked folder in the git user’s home directory for repositories.

Create RSA Key for the GIT Adminitrator

An initial RSA key will be needed for the main GIT administrator to work properly. This should be a local user on the server although other remote users can be added later on.

If you are using linux or an OS that already has .ssh installed, you will probably have $HOME/.ssh directory with an id_rsa.pub file. 

If not, create an RSA key as yourself, using the following command. When asked for a passphrase, just press enter. 

$ ssh-keygen -t rsa

Using your personal user (aka uuklanger in my case), copy it into the /tmp directory so it can be picked up by git and stored in the GITOLITE keydir.

$ cp ~/.ssh/id_rsa.pub /tmp/uuklanger.pub
$ sudo chown git:git /tmp/uuklanger.pub

Once completed, you will have your first Administrator GIT user key ready to go. This is the first step needed before GIT administration is possible.

Initialize gitolite_admin project

In order to be able to administer the GIT site, an administrator will need to be setup. The user will be identifed by their public key. In this case, uuklanger

$ sudo su - git
$ cd $HOME
$ bin/gitolite setup -pk /tmp/uuklanger.pub 
$ rm /tmp/uuklanger.pub

When the gitolite setup is run, you will see various output showing things being created and initialized. 

Once completed, you will have the first official user configured in GITOLITE as an administrator. 

Administrator gitolite-admin clone

The git user .gitolite area should not be touched. It will need to be managed by your new administrator where changes can be pushed to the main gitolite repository.

For this to work, you will need your own copy of the gitolite-admin project. To get this you will need to operate as the git administrator for this server.

$ mkdir $HOME/Admin
$ mkdir $HOME/Admin/git 
$ cd $HOME/Admin/git
$ git config --global user.name “UU KLanger” 
$ git config --global user.email “uuklanger@sandwich.net“ 
$ git config --list 
$ git clone git@localhost:gitolite-admin.git 

Once compelted, you should have a ~/Admin/git/gitolite-admin/ folder with conf/ keydir/ in it. This is where GIT will be managed.

The ~git/.gitolite/ folder should never be touched unless there is a very good reason to

If you look at your ~/Admin/git/gitolite-admin/conf/gitolite.conf file you should see something like

repo gitolite-admin
    RW+     =   uuklanger

repo testing
    RW+     =   @all

Where uuklanger is my administrator for this GIT server.

Prepare for GITOLITE Users

It is very likely that you will access GIT from more then one machine as yourelf. However, GITOLITE indentifies you by your RSA key. To manage this, you can nest subdirectories for each platform that will connect to GITOLITE. For now, localhost is a good start.

$ cd $HOME/Admin/git/gitolite-admin/keydir
$ mkdir local
$ mkdir macbookpro

In this directory you can put all user keys that are local to this server. If you create a remote user create a directory for that client’s keys. GITOLITE will recurse the tree to validate the key.

User Setup

In order for gitolite to work securely, you will need an SSH key per user. The public key will be needed and can be easily created if it is not already in the $HOME/.ssh/. 

Each user will need to submit a public key to the administrator who will place it in the ~/Admin/git/gitolite-admin/keydir/ directory. If a subdirectories were created per client, the key will need to be moved there and set with 750 permissions.

For a client to create a key, they can follow the same process used before, run the following command being sure to press enter when asked for a password.

$ ssh-keygen -t rsa

id_rsa.pub file will just need to be sent to the local admin who can put it in the keydirdirectory. Say we need to add “newguy” to git who put the key in the /tmp directory

$ cd ~/Admin/git/gitolite-admin/keydir
$ cp /tmp/newguy.pub .
$ chmod 750 newguy.pub
$ git add newguy.pub 
$ cd ~/Desktop/gitolite-admin/conf
$ git add gitolite.conf
$ git commit -m “Adding new guy”
$ cd ..
$ git push origin master 
$ sudo rm /tmp/newguy.pub

As project are added, gotlite.conf can be edited to give permissions to users for a given project. It is even possible to give other users admin permissions on gitolite-admin.

Checks

The git push origin master command will commit the changes into gitolite. The changes should be visible in ~git/.gitolite/.

You should have something like this for permissions

uuklanger@atomserver:/home/git$ ls -al
total 48
drwxrwxr-x 2 git  git  4096 Aug 15 15:14 bin
drwxrwxr-x 6 git  git  4096 Aug 15 15:12 gitolite
drwx------ 6 git  git  4096 Aug 15 22:00 .gitolite
-rw------- 1 git  git  6998 Aug 15 15:28 .gitolite.rc
-rwxrwxr-x 1 git  git    12 Aug 24 08:24 projects.list
lrwxrwxrwx 1 git  git    27 Aug 15 17:53 repositories -> /opt/data/git/repositories/
drwx------ 2 git  git  4096 Aug 24 08:24 .ssh

to ensure that outside processes can see.

GITWEB

In order to allow someone to browse a GIT repo without a full GIT install, GITWEB has been created. This tool lets you look around and navigate your source tree.

For this to work you will need apche2 installed with cgi support enabled.

Install

Running UBUNTU, gitweb can be installed using apt

$ sudo apt-get install gitweb

This will provide the core install in /usr/share/gitweb/ along with 

  • /etc/gitweb.conf
  • /etc/apache2/conf-{available, enabled}/gitweb.conf

With the base setup, the configuration is needed next.

Configuration of Apache

Edit /etc/apache2/conf-available/gitweb.conf

Alias /gitweb /usr/share/gitweb

<Directory /usr/share/gitweb>
    SetEnv  GITWEB_CONFIG  /etc/gitweb.conf
    DirectoryIndex gitweb.cgi
    Allow from all
    AllowOverride all
    Order allow,deny
    Options +ExecCGI +FollowSymLinks
    AddHandler cgi-script .cgi
    <Files gitweb.cgi>
      SetHandler cgi-script
    </Files>
    RewriteEngine on
    RewriteRule ^[a-zA-Z0-9_-]+.git/?(\?.)?$ /gitweb.cgi%{REQUESTURI} [L,PT]
</Directory>

Ensure that this configuration file is symlinked in ../conf-enabled/gitweb.conf

Configuration of gitweb

Edit /etc/gitweb.conf to to look as follows. Ensure that the extra attention is paid to $projectroot

# path to git projects (<project>.git)
$projectroot = "/home/git/repositories/";

# directory to use for temp files
$git_temp = "/tmp";

# target of the home link on top of all pages
#$home_link = $my_uri || "/";

# html text to include at home page
#$home_text = "indextext.html";

# file with project list; by default, simply scan the projectroot dir.
#$projects_list = $projectroot;
$projects_list = "/home/git/projects.list";

# stylesheet to use
@stylesheets = ("/static/gitweb.css");

# javascript code for gitweb
$javascript = "/static/gitweb.js";

# logo to use
$logo = "/static/git-logo.png";

# the 'favicon'
$favicon = "/static/git-favicon.png";

# git-diff-tree(1) options to use for generated patches
#@diff_opts = ("-M");
@diff_opts = ();

In addition, as the user git, the .gitolite.rc file will need to be updated to change the UMASK.

$ sudo su - git
$ cd $HOME
$ vi .gitolite.rc

Near the top of the file, there will be a UMASK configuration. The default 0077 but should be set to 0027.

Once done, the gitweb.conf file will be pointing to the correct locations for things like the project root and list.

Visible Repos using projects.list

The repos that are visible can be managed in the the git user home.

An example project list

$ sudo su - git
$ cd $HOME
$ cat projects.list
testing.git

As you can see, we only have one project in this list. If you need to add more then simply have a single line project.

$ sudo su - git
$ cd $HOME
$ vi projects.list

Once completed, you should have a projects.list that works.

Project Repositories Permissions

The directory where the repositories are stored will need their permissions confirmed. Best way is to force them to be as you wish.

$ sudo chmod +R 750 /opt/data/git/repositories

When new projects are added, this level of permissions will be set for each new *.git created automatically based on the UMASK set earlier. To ensure that apache2 can see the repos, www-data will needed added to the git group.

$ sudo adduser www-data git

To test to see if gitweb works, try the following where we are assuming gitweb is installed /usr/share/gitweb/ …

$ sudo su - git
$ /usr/share/gitweb/gitweb.cgi

If everything works well, you should see your projects listed in the returned HTML. 

Since Apache runs as a different user, the only true way to confirm this works is using Apache. For that to happen all the configuraiton will need activated. 

Activation

After all configuration is completed apache will need to be restarted.

$ sudo service apache2 restart

Apache should restart without any errors. 

Checks

Once restarted it should be possible to navigate to 

http://HOSTNAME/cgi-bin/gitweb/

or 

http://HOSTNAME/gitweb/

If you get back your project list, then you are in and should be able to fully navigate your code trees based on the repository permissions (r-x for world) and what is stored in the projects.list file for the git user.

Conclusions

Based on this you will hopefully have a base to start with for GIT. If you do not understand why a step was done above, research it to ensure it is understood and agreed. 

Created 20-November–2015 by uuklanger

Updated 06-December–2015 by uuklanger

Updated 23-December–2015 by uuklanger

Friday, March 13, 2015

HOWTO Batch Language Date and Time

Overview

Batch language is super basic but there is one trick BATCH supports for getting the date and time. Looking through many different blogs I was able to assemble this HOWTO.

I use this syntax for a little script that I wrote that to backup/archive various coding directories I have on my workstation. I use Windows Backups but I find I trust a simple BAT file that creates date/time stamped 7z files. I wish Windows had a “time machine” like facility but use what you have vs fuss.
To create a proper date/time stamp you can use the following technique.

Syntax/Technique

If you include something like this in your Windows BAT file, you will get a date and time that can be applied to a filename or similar.
set hr=%time:~0,2%
if "%hr:~0,1%" equ " " set hr=0%hr:~1,1%
set FNAME=ARC_%date:~-4,4%%date:~-10,2%%date:~-7,2%_%hr%%time:~3,2%%time:~6,2%.zip
echo %FNAME%
… you will get a result like this
C:\Users\k.langer>echo %FNAME%
ARC_20150209_115340.zip

Conclusion

You can use this trick when automating backups or similar.
[For More Information http://stackoverflow.com/questions/7727114/batch-command-date-and-time-in-file-name]

HOWTO Setup a PiGlow with PyGlow on a Raspberry Pi B Plus

Overview

There are many articles and blogs out there that show how to use a PiGlow LED GPIO board with a Raspberry Pi but most do not cover everything. Some only describe how to set it up with hardware before the B+ and others get you going but fail on permissions.
My hope is to show all the steps needed to let you run Python code with a PiGlow and the PyGlow.py module.

Security

I am not a security genius, nor do I view this project as security driven. You will need to decide what changes below are reasonable for you. For me, I want the PiGlow to work and be easy to work with. This is the process I used to get there.

What you need

  1. Raspberry Pi B+
  2. Rasparian Wheezy
  3. PiGlow module

kernel setup

For this to work, there will need to be some kernel level changes. This simply tells the kernel to load some required modules.

/boot/config.txt

In order for the i2c module to work, it needs to be enabled. Add or uncomment the following to your config.txt file.
dtparam=i2c1=on

/etc/modules

Ensure that the following two lines are in your modules file.
i2c-dev
i2c-bcm2708

/etc/modprobe.d/raspi-blacklist.conf

Ensure that the following modules are not blacklisted by ensuring these lines are commented out.

# blacklist spi-bcm2708
# blacklist i2c-bcm2708

Reboot

At this point, the kernel will have what it needs to allow you to work with your PiGlow. To activate the new configuration, you need to restart your Raspberry Pi.

Setting up PyGlow Python Module

With the kernel prepped and ready, the next phase is to install the python library needed to interact with your PiGlow. For a Raspberry Pi, I suggest creating a library location where this and other common libraries can be stored for shared use across users and applications. On my RPi, I have them in /opt/lib/python If this path does not exist, create it or something like it. Ensure you can get to it using a normal users and chmod as needed.

Setup a Library area in /etc/bash.bashrc

To ensure all users on the platform see this new library location, it will need to be added to the PYTHONPATH. That can be done either in each users .bashrc file or globally in the /etc/bash.bashrc. Just add the following line.
export PYTHONPATH=/opt/lib/python
For the full approach to work, it will need to be added globally. Within a user’s .bashrc, they can append to this path with other personal lib or module path.

Install PyGlow Module from GIT

To get the lastest PyGlow.py module, run the following commands.
cd /opt/lib/python/
git clone https://github.com/benleb/PyGlow.git

Git should already be installed on your RPi. If not or you don’t like git. You can find a ZIP of the same thing using a few Google searches.
Once completed you should have a PyGlow sub directory within /opt/lib/python/. If you read other documentation, you will be told that you can simply run (or at least attempt to run)
cd PyGlow/examples/
sudo python test.py

But when you do, you get an error that PyGlow cannot be found. This is beause by default, PYTHONPATH is not seen by sudo. This takes us to our last phase of setting this up.

sudo Setup for Python Users

Add PYTHONPATH to /etc/sudoers

To tell sudo that you want to keep PYTHONPATH as an environment variable. Simply edit /etc/sudoers and add (or uncomment) the following line.
Defaults    env_keep += "PYTHONPATH"
Now anything in the global Python path will be available to sudo. I would assume you can have multiple lines for each ENV you want to keep.

PyGlow Error for RPi B+

So, all excited, you re-run the test script

cd $PYTHONPATH
cd PyGlow/examples/
sudo python test.py

Now a new error appears complaining about a revision issue or something like that.

The fix for the PyGlow.py Error


The code for the PyGlow (at the time this was written) was not updated for the B+ (Revision #3) GRUMBLE, but the fix is easy. Edit /opt/lib/python/PyGlow/PyGlow.py and search for the following code….

 # Check what Raspberry Pi version we got
 if rpi.RPI_REVISION == 1:
     i2c_bus = 0
 elif rpi.RPI_REVISION == 2:
     i2c_bus = 1
 else:
     raise PyGlowException(
         self, "Unknown Raspberry Pi hardware revision: %s" %
         (rpi.RPI_REVISION))
 
 
 After the above elif, add the following elif that.  It should be before the else 
 
  elif rpi.RPI_REVISION >= 3:
     i2c_bus = 1


It basically says that any revision after and including 3 should use the #1 bus for ic2. Or at least I suppose.

Testing

Now you re-run, more causiously, the following command.

cd $PYTHONPATH
cd PyGlow/examples/
sudo python test.py

Based on my experience, you should have success. In fact all the example scripts should now work.

Next Steps

There are a lot of groups in linux and it is possible that vs using sudo to run your code you can simply add your user to a group that allows i2c and other modules to work. I’ll have to research that next to see if I can give a user permissions.

This was not much of a goal for me since this device is experimental so sudo is not a worry for me to run. For eample, I like to run the sudo python cpu.py to let the lights show me how much CPU I am using. python-psutil and psutils are needed for that to work so sudo apt-get install python-psutil psutils before running that comand.

Conclusions

With a few steps you now have a working PiGlow with support for PyGlow.py. This same approach should work for other GPIO type components.

Thursday, February 5, 2015

HOWTO Create an SVN Branch from a Tag

Overview

Bugs happen and sometimes you need to create a patch. Best way to do this using SVN is to take an existing tagged version of your code (which you hopefully do before each build) and create a branch off of it.

Approach

For this to happen you need to know which tag to use and what you want to call the branch. In the example below, I am using 20141205_Version_R8–1–0–07 as my tag. Within this location is an SVN copy of what was in trunk at the time the tag was created.

Look at the contents of the tag

To see the contents of the tag, use the following command
> svn list http://svn.homeworld.lan/repos/python/tags/20141205_Version_R8-1-0-07/
You will get a listing similar to what you would see in trunk.

Create the Branch

To create the branch all you will do is tell SVN to copy the contents from the tag into a new (non-existing) target folder. In this case the folder within branches is called 20150205_R8–1–1_Patch. Remember that this folder will be created and filled by SVN automatically so the name of the target folder is all you need to provide.
> svn copy http://svn.homeworld.lan/repos/python/tags/20141205_Version_R8-1-0-07/ \
    http://svn.homeworld.lan/repos/python/branches/20150205_R8-1-1_Patch \
    -m "Creating a branch off of tag 20141205_Version_R8-1-0-07"

Committed revision 630.
Once completed you will see a revision number (630 in this case)

Look at the contents of the new branch

To confirm that the branch worked as you would expect, you can get a listing like you did for the tag.
> svn list http://svn.homeworld.lan/repos/python/branches/20150205_R8-1-1_Patch

Getting to Work

Now that this is done you can checkout the code from the branch into a working directory and get to work solving the worlds problems (certainly not your own).
> mkdir 20150205_R8-1-1_Patch
> cd 20150205_R8-1-1_Patch
> svn checkout http://svn.homeworld.lan/repos/python/branches/20150205_R8-1-1_Patch ./

Conclusion

Using a few commands you can easily create a branch from a tag. If you do not happen to have a tag available, you can branch off a specific revision of your trunk using the revision command line option. But hopefully you will create tags as you go for each build you create or for each release pushed to SQA or any testing group.

HOWTO Create lockfile logic using BATCH language

Overview

The goal of this HOWTO is to show how to manage a lock file in batch language. This is handy for scheduled processes.

Example

@echo off
REM
REM Example of howto use a lockfile in BATCH language
REM

:INIT
CD \BatchJobs
ECHO Current Working Directory
CD

:START
IF EXIST NIGHTLYRUN.LOCK GOTO SKIP_WORK
ECHO Turning on Lock
ECHO ""LOCK"" > NIGHTLYRUN.LOCK
REM Do work here. this is where your program or process goes. 
ECHO Turning off Lock
DEL NIGHTLYRUN.LOCK

GOTO DONE

:SKIP_WORK
ECHO ""This process is already running..""

:DONE

Conclusion

Using this approach you can prevent multiple instances of something running. A possible enhancement is something that can sense a stale lock if there is a crash or something. however this should help for most situations.

HOWTO Python Setup UNICODE Logging

Overview

Python does a very good job handling UNICODE for most modules, however, logging is not one of then. To get around this is easy but a special technique is required.

Common Approach

Normally with logging you pass a file to the logger and then write to it as you need though the logger interface. This approach is fast but does not allow the setting of the file encoding.
import logging
logging.basicConfig(
    level=logging.DEBUG,
    filename=filename,
    filemode="w")

formatter = logging.Formatter('%(asctime)-15s: %(levelname)-7s - %(message)s')
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
ch.setFormatter(formatter)

log = logging.getLogger("MAIN")
log.addHandler(ch)
_logger = log

Improved Approach

To get around this and to ensure you control the formatting, a file stream will need to be created first and then that stream can be passed to logger.
import logging
import logging.handlers
log_fh = open("/logs/my_unicode.log", "w", encoding="utf-8")
logging.basicConfig(
    level=logging.DEBUG,
    format=config.FORMAT)

formatter = logging.Formatter('%(asctime)-15s: %(levelname)-7s - %(message)s')
ch = logging.StreamHandler(log_fh)
ch.setLevel(logging.DEBUG)
ch.setFormatter(formatter)

log = logging.getLogger("MAIN")
log.addHandler(ch)
The log_fh holds the handle to the UTF–8 stream which is passed to the logger StreamHandler. Once set, text will be properly encoded as it is passed to the logger.

Conclusion

Using a slightly altered technique it should be possible to write anything to a python log using the build in logger module. There is no reason to use this technique for all logging even if you do not think UNICODE will be used.

Sunday, February 1, 2015

HOWTO Add/Subtract Days or Time

Objective

Show different database techniques for determining the date in the past (or future). The following will show you HOWTO get three days ago based on CURRENT_TIMESTAMP. All will generate something like this.
CurrentTime ThreeDaysAgo
2015–02–01 11:42:04 2015–01–29 11:42:04
Below you will find information MySQL, SQL Server, SQLITE3, and PostgreSQL.

MySQL

Using MySQL, take a timestamp DB property (like CURRENT_TIMESTAMP for “now” or a property from a table). Then add + or - followed by the keyword INTERVAL. Then a number followed by a unit like day if you (for example) want to add or subtract a certain amount of days. The fllowing code subtracts three days from now.
mysql> SELECT CURRENT_TIMESTAMP as CurrentTime,
         CURRENT_TIMESTAMP - INTERVAL 3 day as ThreeDaysAgo;
For More Information

Microsoft SQL Server

Using SQL Server, the end result is the same but they use a function to do the same thing. Using the DateAdd function, you specify a unit, quantity, and DB property. In the following example I use “now” and subtract three days from it.
SELECT CURRENT_TIMESTAMP as CurrentTime,  
    DateAdd(day,-3,CURRENT_TIMESTAMP) as ThreeDaysAgo;

SQLITE3

For SQLITE, you need two wrapper the date time property in the datetime(…) function with a paramter telling it what manipulations you would like to do. In this case, I want to do subtract three days.
sqlite> SELECT CURRENT_TIMESTAMP as CurrentTime, 
datetime(CURRENT_TIMESTAMP, '-3 days') as ThreeDaysAgo
For More Information

PostgreSQL

I have not had a chace to work this out for postgresql yet.
For Information on this point visit

Conclusions

Using the above syntax, you will be able to manipulate dates to add or subtract time as your code requires.

Thursday, October 4, 2012

Alex Lifeson as Greatest Guitar Player of All Time

I don't typically do this but I a am a pretty big fan of RUSH.   They have always produce cool and interesting music.

Recently "Guitar World" magazine decided to have a poll for the "Greatest Guitar Player" of all time.  If you have a minutes, hop online and vote for him.  


He has always been underestimated as far as influence and ability.   If you can, please vote.

Tuesday, March 15, 2011

Jetbrains PyCharm Simply Rocks - Initial Impressions

Overview
I have not had time to write a blog about PyCharm yet. With 1.2 released today, I have had only good experiences with it. Since the first 1.0 release, I have worked on 7 different projects using it.
  • New projects
  • Import existing projects
In all cases, my projects
  • Use the standard python unittest
  • Are stored and managed using Subversion.
Unit Test Integration
About as simple as it gets. PyCharm gives you a lot of flexibility. You can flag a folder as a "test" folder. When you right click, you can Mark Directory As "test" which will turn the folder "green".

If you want to execute all tests, simply right click on the folder and Run "Tests in...." menu option.

If there is a single file of tests you want to run, simply right click on the file and Run the file.

In both case, a nice panel that pops up showing you the test progress. There is a filter that can be turned on or off allowing you to see all test results or just the failed ones. Visual queue help you know how well things are going.

Since Python is fast to run/re-run, I get in the habit of adjusting code and re-running my tests. The unit test module lets you do this by simply clicking on a green double triangle icon in the left side of the Run panel. This is very handy.

This feature is clean and simple to use.

Subversion
Importing and checking out code works as expected. As code is developed, PyCharm provides a "Changes" panel that you can view to see what files are modified.

By right clicking on a folder or file, you can Commit your changes. In reverse, to update your local working copy, you can update a file or folder.

The UI provides clear feedback as to what it is doing in both cases, and lets you know what version your working copy is.

Inspections
I did an experiment. How good is the advice PyCharm provides to improve your code? I took an old project I wrote and imported it into PyCharm. The old code was written using emacs and not much else.

I address all of the suggestions provided by the PyCharm's code Inspection. I was surprised by the sense many of the changes suggested. In my experience, the suggestions provided did not break or cause issues in my code.

It is possible to fine tune the suggestions in the settings/preferences. The options provided are very detailed (in my mind).

Python Interpreter
PyCharm lets you configured one or many version of python. Adding a new version is pretty easy. If you don't have a project open, you can set the default interpreter in the settings/preferences. If you have a project open, you can set the interpreter version for the open project.

Final Thoughts
These are just a few items of many productive features within PyCharm. I am very happy with what I have been using so far. In just about all cases, I have not needed to use any of the docs. It just works the way I would expect it to behave.

I want to try Django support next. But that is for a future blog.

Saturday, January 22, 2011

Saturday, October 23, 2010

Tuesday, February 2, 2010

POLL: I program at home/work (Results)

  • 60% *nix/Windows
  • 20% *nix/*nix
  • 20% Windows/*nix
  • 0% Windows/Windows

POLL: My last vacation I worked (Results)

  • 40% 0-1 hours per day
  • 20% 4-5 hours per day
  • 20% 6-7 hours per day
  • 20% What Vacation!
  • 0% 2-3 hours per day

Sunday, January 3, 2010

Sunday, November 8, 2009

Dell Mini 10v Netbook Running Ubuntu

Overview
A few month ago, I started looking for a small portable computer. I ended up targeting netbook class portable computers. I needed something small and reasonably powerful to allow me to work where ever I happened to be. I also needed something that I could put linux on since that is what I prefer over other operating systems. I ended up getting a Dell Mini 10v netbook for use when traveling.

Specifications
I got the Mini 10v version which is configured as following:
  • Intel® Atom Processor® N270 (1.6GHz/533Mhz FSB/512K cache)
  • 10.1" Widescreen Display (1024x600)
  • 1GB DDR2 SDRAM
  • 120G SATA HD
  • Intel Graphics Media Accelerator (GMA) 950
  • 56WHr Lithium-Ion Battery (6-cell)
  • Ubuntu 8.10 Linux
  • Webcam (1.3 megapixel)
  • 3 USB
  • 1 VGA
  • Headphone/Mic
I got it with the default color case which is black. The only upgrade I got was the battery which was bumped from a 3-cell to a 6-cell.

Aside from that, I picked up a cheap-o portable USB mini mouse at a local computer store since I hate touchpads (only option for the mini) and a Caselogic PLS-9Black Ultra Portable 7-Inch - 10-Inch Laptop Sleeve (Black) So I could safely carry my netbook in a separate compartment from my main laptop.

The unit comes in a very small and tidy box with just the things you need.
  • Netbook
  • Power supply
  • CD with OS and such
  • Small getting started guide
After charging the battery fully, I booted it up. The following are my impressions about the Dell Mini 10v.

Screen
The screen is small. Duh? At 10.1" with 1024x600 resolution, it does not provide much heights to work with. Lets just say, any code I develop on it is even more modular and concise then normal since a page on the mini is shorter then on just about anything else.

For python programming (first job it was tasked with) I had no problems. Emacs is lean and well suited for this computer.

For things like Netbeans, you will want to collapse just about everything except the editor. I have not needed to do heavy programming yet but I suspect it is up to the task.

It does support a VGA output. So far I have not been able to get an extended desktop to work. I can, however, pump all video output to an external monitor or projector. Extended desktop would be a serious advantage for someone who wants to connect this to an external monitor when at home or work.

Image Quality
Fantastic. The screen is bright and clear. I have had no issues reading documentation, code, emails, and blogs using it. I like the 10v screen better then my main Latitude E6500.

Performance
The Intel Atom process does a fine job. It is not very beefy but I have not had noticeable waits. I am not a gamer so I cannot speak for that. For business work and casual Internet work, the performance is just fine.

Ubuntu 8.10
I commend Dell for pre-installing Ubuntu 8.10. It is great that they choose hardware that will work for more then just Windows. Ubuntu 8.10 works well out of the box. It doesn't have any "crap ware" that you usually get with a non-Apple computer. I powered it on, answered a few question and was up and running.

However, I upgraded to 9.04 almost right away since it was more mature. To upgraded, I just downloaded the image and flashed it to a pen drive. After booting and following the base instructions, install was a snap. There were absolutely no issues at all upgrading. All hardware was detected and everything came up perfectly. Even the webcam.

When Ubuntu 9.10 came out, I upgraded that day using the standard Ubuntu network upgrade. Again, no issues. A perfect install/update.

UPDATE (2010-01): I have spoken with a few people who have said that a clean install to 9.10 does not go very smooth. If this is the case, your best bet is to get 9.04 and upgrade to 9.10. It is a little more work but should not cause any issues. Hopefully 10.04 in April 2010 will improve the clean install.

Why not Windows XP? Well, after installing all the extra windows applications + virus scanner, I would be dealing with a pretty sluggish computer. With XP, a real time virus scanner is a must. Especially on a system that is designed to be used for travel.

Overall Linux seems better suited to leaner systems. Plus, it has everything I need to be productive out of the box. Linux is just my personal preference.

Battery
The one upgrade I did for my netbook was to get a bigger battery. The 6-cell is about $35 more and well worth it. On a long flight with my wireless off, I had power for just about the entire time. I was quite surprised actually.

To date, I only use my netbook on battery and the battery life is still very good. The battery life will be less with wireless on but it is still very long for a portable computer.

With the 6-cell battery, there is a long round strip that sticks out the bottom of the netbook. At first I hated this since it put a bulge on the back. I got over it once I realized that it helped keep it cool and put the whole thing at a slight angle for better and easier typing. The sleeve I got stretches to accommodate the bulge from the battery.

Keyboard
For something so small, the keyboard is great. It has the extra windows keys between control and alt that work great with Ubuntu. For me, the feel is very good and I can type fast on it.

For Ubuntu, I suggest going into the CMOS setup of the netbook (accessed when you see the dell logo during boot) and set the function keys (F1-F12) to be function versus multi-media. For me that is a personal preference.

Touchpad
Before I talk about the touchpad, I have to confess, I hate all touchpads. They just don't work for me very well. Especially when I am doing graphical programming.

Of all the touchpads inflicted on me, this is probably one of the worst I have used. Dell made a mistake by mixing the buttons and touchpad area. I can use it but others I have used work better. For me, I don't care since I got a cheap mini-mouse that plugs into a USB port. When the mouse is not plugged in, the touchpad does fine.

I wish it came with an inteli-stick option like the latitude or thinkpad series laptops.

Usability
Even being so small, Dell did a good job balancing the size and usability. I can be productive on this unit. I get the benefit of portability and quick boot up times. The keyboard is great and the screen is clear.

Overall Opinion
I am happy with my Dell Mini 10v. Although I bought it for travel, I find I use it more then I expected. It follows me to meetings at work and around the house.

Pros
  • Great battery life
  • Good quality screen
  • Proper keyboard
  • Runs Ubuntu 8.10, 9.04, 9.10 like a champ
  • UPDATE (2010-12): 10.04 and 10.10 like a champ.
  • 3 USB ports.
Cons
  • Touchpad
  • External VGA out could support extended desktop.
Hope this feedback helps.

Saturday, November 7, 2009

Emacs Configuration .emacs file

Overview
The following is my .emacs file for reference.

.emacs Configuration
;; -*- lisp -*-
;;
;; (c)2009 StoKen Software by uuklanger@gmail.com
;;
;; 2009.11.13 - updated for emacs23
;; 2009.11.14 - added settings for font and height/width
;;
;; ============================================================================
;; Setup Keyboard
;; ============================================================================

(global-set-key [home] 'beginning-of-line)
(global-set-key [end] 'end-of-line)
(global-set-key [delete] 'delete-char)
(global-set-key [C-home] 'beginning-of-buffer)
(global-set-key [C-end] 'end-of-buffer)

;; ============================================================================
;; Emacs customize section
;; ============================================================================

; Window Colors
(set-background-color "DarkSlateGray")
(set-foreground-color "Wheat")
(set-cursor-color "Orchid")
(set-mouse-color "Orchid")
(set-default-font "DejaVu Sans Mono-8")
(set-frame-height (selected-frame) 50)
(set-frame-width (selected-frame) 80)

; Turn off toolbar (toggle)
(tool-bar-mode)

; Display time
(display-time)

; Show column number
(column-number-mode t)

; Show line numbers
(line-number-mode t)

; FAQ 34: Highlight regions of text
(transient-mark-mode t)

; FAQ 66: font-lock-mode for all major modes
(global-font-lock-mode t)
(setq font-lock-maximum-decoration 't)

; FAQ 67: Only scroll one line when moving past screen
(setq scroll-step 1)
(setq scroll-conservatively 1)

; Dont show the GNU splash screen
(setq inhibit-startup-message t)

; Make all "yes or no" prompts show "y or n" instead
; (fset 'yes-or-no-p 'y-or-n-p)

;; Enable wheelmouse support by default
(cond (window-system
(mwheel-install)
))

; Ensure we stay in UNIX mode for EOL char
(setq inhibit-eol-conversion t)

; if we compile, ensure ouput scrolls
(setq compilation-scroll-output t)

;; ============================================================================
;; My preferred key bindings
;; ============================================================================

; Set C-g to goto-line
(global-set-key "\C-cg" 'goto-line)

;; ============================================================================
;; CC Mode setup
;; ============================================================================

(defun stroustrup-common-hook ()
(c-set-style "cc-mode")
(setq tab-width 4 indent-tabs-mode nil)
)
(add-hook 'c-mode-common-hook 'stroustrup-common-hook)

;; ============================================================================
;; Perl Mode setup
;; ============================================================================

(defun perl-common-hook ()
(setq tab-width 4 indent-tabs-mode nil)
(setq perl-indent-level 4
perl-font-lock t
perl-electric-lbrace-space nil
perl-electric-parens nil
perl-electric-linefeed t
perl-electric-keywords nil
perl-info-on-command-no-prompt t
perl-clobber-lisp-bindings t
perl-lazy-help-time 1)
)
(add-hook 'perl-mode-hook 'perl-common-hook)

;; ============================================================================
;; Other hooks
;; ============================================================================

; turn on auto fill mode when editing text files

(add-hook 'text-mode-hook 'turn-on-auto-fill)

;; ============================================================================
;; Runs Scripting Languages Program using compile key bindings
;;
;; if you have another lanugage, simply create a ____-run defun and add an if
;; statement to script-run()
;; ============================================================================

(defun python-run ()
"Use compile to run python programs"
(interactive)
(message "Running python for buffer %s." (buffer-name))
(compile (concat "python " (buffer-name)))
)
(defun perl-run ()
"Use compile to run perl programs"
(interactive)
(message "Running perl for buffer %s." (buffer-name))
(compile (concat "perl -w " (buffer-name)))
)
(defun script-run ()
"Use to run a script"
(interactive)
(if (equal (file-name-extension(buffer-name)) "py")
(python-run))
(if (equal (file-name-extension(buffer-name)) "pl")
(perl-run))
)
(global-set-key [f5] 'script-run)

;; ============================================================================
;; Printer settings
;; ============================================================================
;; ============================================================================
;; Fun stuff. Functions you can call up from the command line
;; ============================================================================

(defun ascii-table ()
"Print the ascii table. Based on a defun by Alex Schroeder
"
(interactive)
(switch-to-buffer "*ASCII*")
(erase-buffer)
(insert (format "ASCII characters up to number %d.\n" 254))
(let ((i 0))
(while (< i 254)
(setq i (+ i 1))
(insert (format "%4d 0x%02X %c\n" i i i))))
(beginning-of-buffer))
(defun insert-date ()
"Insert date at point."
(interactive)
(insert (format-time-string "%a %b %e, %Y %l:%M %p")))
;;; Convert DOS file format to Unix
;; look at
;; M-x comint-strip-ctrl-m
;; Command: Strip trailing `^M' characters from the current output group.
;; from: elf@ee.ryerson.ca (Luis Fernandes)
;; 22 May 1997
;;; Usage: M-x dos2unix
;;;
(defun dos2unix ()
"Convert this entire buffer from MS-DOS text file format to UNIX."
(interactive)
(save-excursion
(goto-char (point-min))
(replace-regexp "\r$" "" nil)
(goto-char (1- (point-max)))
(if (looking-at "\C-z")
(delete-char 1))))
;; ============================================================================
;; End of file
;; ============================================================================
Final Thoughts
If you use emacs, some of the settings above may help. I have gathered or created these settings and lisp code

HOWTO: dos2unix in emacs

Overview
I switch between Linux, OSX, and Windows quiet a bit. One problem with that can be the dreaded EOL issue.

For example, if you add #!/usr/bin/env python and have a DOS/Windows formatted file, running it in UNIX right from a shell may not work.

For my development, I prefer to standardize on UNIX format. For the languages I use, I know it will not cause any issues regardless of where they are run.

Requirements
If you are using linux, OSX, or Solaris you will edit your ~/.emacs file. If you are running Windows, you will be editing your %HOME%/_emacs

For this howto, emacs-22 is being used under Ubuntu Linux 9.10. I doubt the OS or version of emacs will make a difference.

Force Unix EOL
To avoid having to deal with EOL on your own code, add the following to your .emacs file.
; Ensure we stay in UNIX mode for EOL char
(setq inhibit-eol-conversion t)
Now emacs will only use UNIX EOL Plus, you will see ^M in your file for each line with a DOS/Windows EOL.

dos2unix Code
I found this code and have been using it. Why re-invent the wheel.
;;; Convert DOS file format to Unix
;; look at
;; M-x comint-strip-ctrl-m
;;   Command: Strip trailing `^M' characters from the current output group.
;; from: elf@ee.ryerson.ca (Luis Fernandes)
;; 22 May 1997
;;; Usage: M-x dos2unix
;;;
(defun dos2unix ()
"Convert this entire buffer from MS-DOS text file format to UNIX."
(interactive)
(save-excursion
 (goto-char (point-min))
 (replace-regexp "\r$" "" nil)
 (goto-char (1- (point-max)))
 (if (looking-at "\C-z")
     (delete-char 1))))
To use, press ESC-x and type dos2unix After pressing return, you will see the magic happen. I suggest making a backup before doing any global changes to anything. All ^M will disappear.

Final Thoughts
Adding this lisp code will help you standardize on UNIX EOL. You will find the full .emacs file here.

HOWTO: setup emacs to run scripts within UI

Overview
Lately, I have gone back to my old friend... emacs. Some would agree that a better and more flexible editor has never been created. I used emacs for many years for C/C++, perl, HTML, and shell programming. Over time, projects and jobs drove me to full IDEs like Netbeans and Visual Studio.

I recently purchased a DELL Mini 10v netbook running (of course) Ubuntu. With the tiny (but beautiful and clear) screen, big IDEs are just not practical. Even with all panels minimized there is still only a tiny amount of space.

For my last trip, I needed to develop some code to hack a pile data I was emailed right before I got on the plane... stat! Thank ___ for having a plane friendly computer like a netbook.

I have become dependent on all the work heavy IDEs automatically do. Do I really know who to code or does the IDE guide me to a completed program? Plus, data hacking is better for scripting languages like python or perl. On taxi I thought it over and made my call. Welcome back to my tool belt emacs.

Once we hit cruising altitude, I booted up Ubuntu 9.04, fired up emacs and started coding. There was one problem. I have been so used to simply pressing F6 to run my program. I wanted to do the same thing in emacs. Why be limited?

The following tells you howto setup emacs, for this type of behavior, using just a small amount of lisp code. The only real dependency is having the languages you are using in the path. If python or perl doesn't run from a command prompt, this will not work very well.

Requirements
If you are using linux, OSX, or Solaris you will edit your ~/.emacs file. If you are running Windows, you will be editing your %HOME%/_emacs

For this howto, emacs-22 is being used under Ubuntu Linux 9.10. I doubt the OS or version of emacs will make a difference.

Launch Interpreter using F5
Simply open your .emacs or _emacs file and add the following
;; ============================================================================
;; Runs Scripting Languages Program using compile key bindings
;;
;; if you have another lanugage, simply create a ____-run defun and add an if
;; statement to script-run()
;; ============================================================================

(defun python-run ()
"Use compile to run python programs"
(interactive)
(message "Running python for buffer %s." (buffer-name))
(compile (concat "python " (buffer-name)))
)
(defun perl-run ()
"Use compile to run perl programs"
(interactive)
(message "Running perl for buffer %s." (buffer-name))
(compile (concat "perl -w " (buffer-name)))
)
(defun script-run ()
"Use to run a script"
(interactive)
(if (equal (file-name-extension(buffer-name)) "py")
(python-run))
(if (equal (file-name-extension(buffer-name)) "pl")
(perl-run))
)
(global-set-key [f5] 'script-run)

If the file is python (py) then python will be run in a separate buffer. If the file is perl (pl) then perl will be run in a separate buffer. To trigger this, all that is needed is to press F5 while in a buffer that is either a python or perl file. That is it.

Final Thoughts
This approach can easily be expanded for other languages like ruby and such. I hope this helps simplify programming using emacs. Find my full .emacs file here.

Personally, I am glad I dusted off emacs. Humm... what about VI.