Help with versioning tools and systems

[hammerhead]

Looking for instruction, advice, and/or (ideally free) resources for learning how to efficiently and reliably implement source repositories and version and maintain backups of build histories using Microchip X IDE, and XC series of compilers.

Background:

The company I work for is a small operation.  Right now I am the entire development team.  We've been developing products primarily with Microchip PIC for the past 15 years that I've been here.  In the past we focused primarily on PIC16 and some PIC18 projects and for the most part they were not very complicated, there weren't many variations, and there was never more than myself plus one other person involved, so versioning was not an issue. We'd usually zip up a backup of the source, give it a name that was somehow significant, then start making whatever changes to the software we needed.

However, now we've moved our flagship products to PIC32 because our designs are becoming more complex and we need the extra horsepower.  With more complex designs comes more ways to break things.  Especially since we're using Microchip Harmony framework, which through forced updates of the configurator has broken my software more than once by overwriting modifications and bug-fixes I had made.

I've looked into some, but 20yrs ago when I went to school, versioning and source-code repositories were not anything we covered and I feel a little like a fish out of water.  I've scratched the surface on some.  I've installed mercurial into MPLAB X IDE, but I'm positive I'm not using it correctly or efficiently.  I'd also like opinions or information on systems with cloud repositories, but it would have to be secure and private because most of what we do is confidential and proprietary.

We're going to be hiring more developers soon and I'd like to have some sort of formal system in place to avoid confusion and with an eye to implementing stricter quality control procedures for the entire process.
Any help or pointers would be greatly appreciated.

[h0nk]

Hello hammerhead,

more developers means more and exact documentation and a workflow
to implement changes.
All the work must fit together.

My last customer used Atlassian tools.
They have all what You need:
- Atlassian Bitbucket as Git-server,
- Atlassian Confluence as Documentation system (Wiki-style),
- Atlassian Jira as Bug-Tracker and
- Atlassian SourceTree as Git-client.
- Plug-Ins which connect the systems.

They have evaluation licenses available, so You can test it before.
The products are most licensed by user count, so for a couple of worker
they should not hurt.

All there software can run in a secure in-house mode without connection
to the internet.

But: You will need a big-iron as a server and a database for their products.
And a backup of all in case something fails.

And You will need a person/trainer to learn all the details to work with repositories.


Best Luck, and

Best Regards

[hammerhead]

Thanks for the input, h0nk.  I just did a quick check at the Atlassian site.  Looks very promising.  Will definitely check it out in more detail in my spare time.  Bitbucket Cloud and SourceTree are both free for up to 5 users, so I think I may experiment with that a little and see how it works out, while I scour Google for guides and how-to's.  Much appreciated!

[Sideshow Bob]

I do not know if this book is diretcly up your alley. But I posted it so you may leaf through it and maybe pick up something. The book is more directed against what is named Agile Project Management using Using Team Foundation Server from Microsoft. But version control is a important part woven into the full project mangement concept.
http://www.sonsivri.to/forum/index.php?topic=62619.msg179216#msg179216
This is a open link regarding version control
 http://ericsink.com/vcbe/

[pablo2048]

Hi, I'm using Subversion for many years. You can install it on your own machine and test... And here are few links:
http://subversion.apache.org/ - main SVN page
https://tortoisesvn.net/ - IMHO best SVN client for Windows
http://www.microchip.com/forums/m622031-p2.aspx - SVN with MPLAB X (old thread)

[Signal]

Looking for instruction, advice, and/or (ideally free) resources for learning how to efficiently and reliably implement source repositories and version and maintain backups of build histories using Microchip X IDE, and XC series of compilers.
<...> Especially since we're using Microchip Harmony framework, which through forced updates of the configurator has broken my software more than once by overwriting modifications and bug-fixes I had made. <...>

Any useful resources (from primary sources) can be found by mature professional himself by keywords. Some keywords are below.

I see three parts of question.
1) selection of versioning system
2) organization of system environment: server, maintenance, ...
3) repository layout specific to Harmony framework


SELECTION OF VERSIONING SYSTEM

There are two camps: centralized and distributed VCS. Both of them require a beginner to become familiar with new concepts, terms, use-case scenarios, etc. Part of required new knowledge is common for both approaches. Bigger part is somehow different. So knowing only one is not enough to use another without additional study.
I use both but have much more experience with centralized version control systems. Bear it in mind taking my advices. And I do not consider myself as expert in VCS, just experienced user. I used Source Safe, TFS, Subversion, Git. Svn and Git are currently in use.

I use Git for episodic unrelated projects where I am the sole developer. Advantage - I just "git here" on project folder tree and do not bother about repository location, access, etc.
The task that I do not want to solve regarding DVCS is how to arrange hierarchy of repositories and how to maintain it by team. I believe that with DVCS users must maintain arrangement themselves with almost no help from DVCS. This is the part of lack in my experience. In other words I use DVCS in very very limited way.
There are some other unresolved questions that I leave about DVCS, since ...

... For the major part of my work I use Subversion. The truth is - a consolidated work of team is centralized naturally at least by final product. So I appreciate that most part of this centralization is supported by VCS.


VERSION CONTROL SYSTEM ENVIRONMENT

There are several ways to use Subversion that differ by access to repository.
The most easy way to start is to use just client without server executables using file:// access. Badly suited for team work. Do not waste your time trying it. Believe - it works, but not enough.

Instead decide will you use your working PC as a server for your team or better setup separate server. There are some choices: to use third party installers that will do most of configuration or to configure server manually. "installers" usually setup Apache server. For manual configuration I suggest simple svnserve server.

Beginner friendly "installers" are VisualSVN, Collabnet Subversion Edge (and others). First time I used CollabNet built executables (svnserve), then Edge. After occasional fail of Edge on subsequent update (due to java incompatibility issue) I step from CollabNet to VisualSVN. It looked slightly easier for novices.

Currently I use intermediate and flexible solution. I run Debian minimal server on VirtualBox virtual machine running (currently) on my working PC.

Virtual PC allows me to easily relocate server between different PCs (that happened couple of times in case of upgrade). Server has its name and dynamic IP (obtained from our router). DNS on the router is configured to resolve this name for all LAN users. Also full server backup is trivial in case of virual PC.

Running server on the same working PC reduces maintenance efforts for small team. Also backup is easier and faster if backup media is connected to the same PC.

Use of Linux was more foolpfoof for me to setup Trac and its proper integration with Subversion forth and back (comparing my efforts doing the same on Windows).
I use Subversion and Trac from stable Debian packages.


BACKUP

From working PC I run servername_backup.bat script that uses plink.exe to run backup script on server (as custom "backup" user).
This script on server do following:
a) run "svnadmin hotcopy" for all SVN repositories and "trac-admin hotcopy" for Trac.
b) archives hot copied repositories with other key setting of server (init.d/svnserve, init.d/tracd, passwd, group, ${0##*/},...) to lrzip archive "server_date_time.tar.lrz".
Then servername_backup.bat uses pscp.exe to copy lrz file from server to locally connected media - on local hard disk and on one of removable Compact Flash cards in rotation.


REPOSITORY LAYOUT FOR HARMONY BASED PROJECTS

I am trying to be king and polite talking about difficulties in handling of Microchip Harmony projects. There are plenty of them.
Key specifics related to version control are:
1) This is mainly raw product. You should consider corrections made by yourself to this framework.
2) Parts of framework rely on relative paths to locate each other.
3) Harmony framework is supposed to be extensible. For example if you like to add new BSP configuration it is supposed to be located in framework's official folder for BSPs
4) Harmony is not a mature product. Interfaces even of comparatively "production ready" modules sometimes changes with new version.
5) Harmony framework is supplied in plain zip format without information of relation between files in current version and between versions. Maybe this is the case where Git is more able than Subversion. Any opinions are welcome!
6) New versions of MPLAB Harmony Configurator may require new MPLABX and vise versa, and it is a possible issue that it could work only with new Harmony revisions. So opening old project may require to install old MPLABX in addition to old framework.
7) There are no official recommendations from Microchip about repository layout for Harmony based projects.

My current solution is the following. I will appreciate any comments about it.
Variant A:
/repository
   /branches
   /microchip
      /harmony   /*here are exact copies of distributives*/
         /v1_02
            /apps
            /bin
            ...
         ...
         /v1_07_01  
            /apps
            /bin   /*candidate not to store in repository to lessen its size*/
            /bsp
            /build
            /config
            /framework
            /third_party
            /utilities
   /tags
   /trunk
      /harmony   /*made your corrections to Harmony here*/
         /apps     /*all demo projects are here*/
         /bin
         /bsp
         /build
         /config
         /framework
         /third_party
         /utilities
         /work    /*I prefer to locate my projects separately from official apps*/
            /my_project_A
            /my_project_B
               /firmware
                  /my_project_B.X
                  /src
                     /system_config
               /software_for_pc_that_may_share_the_same_sources_with_firmware

Then your working folder is supposed to be check out of /trunk/harmony

In case of update of Harmony I do following actions:
1) Checkout previous version /microchip/harmony/v1_x
2) Make modifications to working copy of v1_x that it becomes identical with new version. This is awkward because changing of files locations and its presence is not handled automatically. Maybe Git is more suited for this task.
3) Branch modifications to new /microchip/harmony/v1_y
   then I see two ways:
I:
4) Merge modifications of just made commit (/microchip/harmony/v1_y) to /trunk/harmony
5) Consequently migrate ALL /work/my_projects to be fine with new harmony

II: (That is the correct scalable variant for team.)
4) copy trunk/harmony to trunk/harmony_Y
5) work with your current project (trunk/harmony_Y/work/my_project_A) leaving others intact.
Variant B:
   /trunk
      /harmony     /* version_X */
          ...
         /work
            /my_project_A  
            /my_project_B   /*other team members can continue theirs work here*/
      /harmony_version_Y
          ...
         /work
            /my_project_A   /*current project under migration process*/
            /my_project_B

The following variant could be an alternative that i did not test:
Variant C:
   /trunk
      /harmony_version_X
      /harmony_version_Y
          ...
      /work
         /my_project_A   /*current project under migration process*/
         /my_project_B   /*other team members can continue theirs work here*/
In this case your working copy for each project is check out of /trunk that will contain multiple Harmony versions. Another drawback - all relative paths in project configurations must be updated and I know only manual way.


I see another extreme - each project contain its own copy or Harmony.
I am going to try this variant next time when I find that a modification to layout is cheaper than handling old one.
Variant D:
   /trunk
      /my_project_A
         /firmware
         /harmony  /* copy or appropriate /microchip/harmony/v1_xx */
      /my_project_B
         /firmware
         /harmony  /* copy or appropriate /microchip/harmony/v1_yy */
            /framework
            ...
         /software
Here local changes and corrections in Harmony framework required by one project will not affect others unintentionally. Otherwise just merge modification of Harmony between projects.

[hammerhead]

Great responses, everyone.  There's some real helpful info here.  Thanks!

[Gallymimu]

I use VisualSVN for our backend, and tortoiseSVN for the frontend.  MPLABX has an SVN client built in, but it's more often broken than working.

Here is our basic instructions for SVN with MPLABX.  It's slightly incomplete without some other documents but it should give you an idea and allow you to avoid all the dynamically generated JUNK in an MPLABX project from cluttering your repo

Code:

When using MPLAB X adhere to the following to keep SVN clean and minimize extraneous data being
stored in the repository.

The project should start with a top level project folder such as "project"

Within the project MPLAB X will create a build, dist, and nbproject folder.

It is advised to create a src folder which contains all sourcecode.

Only a few files should be stored in the repository such as all source files (the src directory)

Makefile and .dep.inc from the project folder.
nothing under the build or dist folders should be included (these are all generated files)

under nbproject only include:
configurations.xml
project.properties
project.xml
all the other files in nbproject can be autogenerated.

---------------
use of svn properties allows for the source code to auto update with useful svn information.

This requires two actions.
1). insert the subversion keyword XML template.txt info at the top of all source files
2). import the SVN properties.svnprops to the file using svn properties dialog

[Signal]

use of svn properties allows for the source code to auto update with useful svn information.

Can you share how exactly do you use SVN properties information inside sources?

I am curious because find limited worth of it for me.
1) An information about source file revision in header comment block (such as revision number, date, developer) is useful only in case of publication/distribution of sources outside versioning system. And even in that case I see it as somehow obsolete tradition (but often required by code standards in companies). For internal use the versioning system is used extensively - every build is allowed only from versioned working copy and production build only from "green" (without modification) and/or tagged copy. In that case commit information is not needed inside every file.
2) Information updated automatically at commit is not enough.

Obviously information about revision of sources and build time is useful for reference. At least it should be available as human readable from label on product, maybe indirectly through serial number or external version number. Some projects gain from integrating build time information at least for reporting.

Finally I realized that I need commit-, build- and, sometimes, flash/production- information stored in a product.
I use homebrew build/program system based on make utility. From there I use SubWCRev.exe to make actual version.inc from template:

Code:

..\version.inc ::
	@echo version.inc
	@SubWCRev.exe ".." "..\version.inc.template" "..\version.inc" -mq >NUL

version.inc.template file example:

Code:

	#ifndef VERSION_INC
          #define VERSION_INC
          #define REVISION     $WCREV$
          #define REV_MODIFIED $WCMODS?1:0$
          #define REV_MIXED    $WCMIXED?1:0$
          #define BUILD_DATE   "$WCNOW$"
          #define BUILD_YEAR   $WCNOW=%y$
          #define BUILD_MONTH  $WCNOW=%m$
          #define BUILD_DAY    $WCNOW=%d$
          #define BUILD_HOUR   $WCNOW=%H$
          #define BUILD_MINUTE $WCNOW=%M$
	  #define BUILD_PATH   "$WCURL$"
	  #define SRC_TAGGED   $WCISTAGGED?1:0$
        #endif

Last two keywords could be useful for production builds for extra control.

version.asm PIC18 assembly file example:

Code:

	#include "version.inc"
ID_LOCATION code 0x200000
	data (REVISION>>8) + (0xFF00&(REVISION<<8))
	db   (REV_MIXED*0xF0)+(REV_MODIFIED*0xA), BUILD_YEAR, BUILD_MONTH, BUILD_DAY
	db   BUILD_HOUR, BUILD_MINUTE
	end

That way an information about build is stored in hex file and is human readable somehow.

Currently I do not do serialization for my PIC projects. I plan to use automatically generated SQTP file passed to IPECMD to add production-time information to device without need of rebuild. Michael Wieckowski's article could be a good starting point in a way of implementing it - http://michaelwieckowski.com/?p=449 (ADDING PRODUCTION SERIAL NUMBERS IN MICROCHIP’S 8-BIT PIC).