Monday, November 4, 2013

Version info and enforced commits

One of prerequisites to successfully operate multiple software servers is to have a good production code inventory. Read an excellent example of an impact of a failed software inventory if you have any doubts about it. Below is some guidance for building software with inventory data compiled in to the code. It provides running systems with data needed for factual inventory. It also has a benefit of committed code enforcement.

In a rush to patch a hot issue developers may deploy a dirty codebase build. Dirty in a sense that the build happens while there are changes to the code which are not committed to the repository, even if local. If that dirty build goes to production while source code gets more changes before committing, one will never have certainty why that production build behaves a certain way should a problem arise.

A way to combat it is to put version and repository status information into the build executable. The example below is a generic guideline for Microsoft Visual Studio 2012 C++ (VC++ in this post) solution building from a Git repository. It also shows a possible integration with GitHub service, if needed. This post gives you a workflow outline, which can be applied to most environments I know of.

The general steps from build to execution are:

  1. The build process has a pre-build script, which gathers version and repository status information. In our case, VC++ uses the Windows PowerShell version_info.ps1 script.

  2. The pre-build script generates a source code file, which is expected by the rest of the codebase. The generated file has everything needed to allow or deny running the version. In our case it is a C++ header file version.h.

  3. A code contains function which checks if the build is legitimate to run, and logs and stops the process as needed. That example code is in the versionLogAndVet function of the version.cpp example file.

  4. At run time, the versionLogAndVet function allows to run only permitted combinations of repository status / build configuration. It also logs version information.

In this specific environment there are three build configurations. DEBUG and RELEASE configurations are standard, while the LOGGED configuration turns additional logging without debug code overhead. It is important, is that the code in version.cpp should only be invoked from server start-up routines, so that tests still can run on uncommitted code in production build configurations.

Here is the source code and notes:

It makes sense in most VC++ solutions to configure these checks on a per-project basis. Specifically configure it for projects which generate executable files. In the line below you likely need to customize two things - a path to the version_info.ps1 script and the word "server" - which will become the namespace for storing constant values describing the repository state and information.

powershell.exe -ExecutionPolicy RemoteSigned -File "$(SolutionDir)Util\Build\version_info.ps1" server "$(ProjectDir)\" "$(SolutionDir)\"

A word of precaution - the interpreter expands special character sequences on that command line. It took me a lot of time to realize, that the $(...) variables filled by Visual Studio have backslashes as path separators and also end with a backslash. Which means, that PowerShell takes the last backslash as an escape character for the following '"' and breaks the command line. For that reason there is an extra backslash before '"' which turns the last backslash of the path into itself after expansion. The unsolved problem, though, is if your paths have directories starting with escapable characters. In that case, directories may start with a bad character and break things. Microsoft did not provide any help with this issue (see here and the footnote).

As we use VC++, it does not allow to specify the PowerShell script directly - we need to specify the powershell interpreter and desired security policy. Also, this step should be embedded in an XML project properties file so that it can be assigned to multiple projects and build configurations:

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <ImportGroup Label="PropertySheets" />
  <PropertyGroup Label="UserMacros" />
  <PropertyGroup />
  <ItemDefinitionGroup>
    <PreBuildEvent>
      <Command>powershell.exe -ExecutionPolicy RemoteSigned -File "$(SolutionDir)Util\Build\version_info.ps1" server "$(ProjectDir)\" "$(SolutionDir)\"</Command>
    </PreBuildEvent>
  </ItemDefinitionGroup>
  <ItemGroup />
</Project>

Here is how the step will look in the project properties (note, that the value is not bold, meaning it is inherited from the .props file.

The pre-build step calls the version_info.ps1 Microsoft PowerShell script. The version_info.ps1 script runs a few git commands which query current repository information and generate a header file to be used later in the build. This is the script which you want to edit to provide more or different repository information to your software:

Param (
 [String]$Namespace,
 [String]$Project,
 [String]$GitRoot,
 [String]$HeaderFile="version.h",
 [String]$VerPrefix="https://github.com/<USER>/<REPO>/commit/"
)

Push-Location -LiteralPath $GitRoot

$VerFileHead = "`#pragma once`n`#include <string>`n`nnamespace $Namespace {`n"
$VerFileTail = "}`n"

$VerBy   = (git log -n 1 --format=format:"  const std::string VerAuthor=`\`"%an `<%ae`>`\`";%n") | Out-String
$VerUrl  = (git log -n 1 --format=format:"  const std::string VerUrl=`\`"$VerPrefix%H`\`";%n") | Out-String
$VerDate = (git log -n 1 --format=format:"  const std::string VerDate=`\`"%ai`\`";%n") | Out-String
$VerSubj = (git log -n 1 --format=format:"  const std::string VerSubj=`\`"%f`\`";%n") | Out-String

$VerChgs = ((git ls-files --exclude-standard -d -m -o -k) | Measure-Object -Line).Lines

if ($VerChgs -gt 0) {
  $VerDirty = "  const bool VerDirty=true;`n"
} else {
  $VerDirty = "  const bool VerDirty=false;`n"
}

"Written $Project\" + (
  New-Item -Force -Path "$Project" -Name "$HeaderFile" -ItemType "file" -Value "$VerFileHead$VerUrl$VerDate$VerSubj$VerBy$VerDirty$VerFileTail"
).Name + " as:"
""
Get-Content "$Project\$HeaderFile"
""

Pop-Location

A few notes regarding this script. It's placement is not very important, but it is important to include it in the repository. A git executable should be on the PATH. The way this example is organized it makes possible for each project to independently choose to create or not to create the header version file. That version header file mentioned on the fifth line can be named differently - either via an extra parameter when calling PowerShell or as the default value on that fifth line. It is important to include the header file name (version.h in this case) in .gitignore file so that it is ignored by git. Finally there is no automation I came up with to generate an html link to the repository on GitHub. You will need to manually edit the <USER> and <REPO> placeholders to point to your repository. I am not going over the script in greater detail for a simple reason - if you can not make sense out of it, you should not be using it - even if it is not destructive.

As an example, the version_info.ps1 script will generate a version.h header file in a project root directory, which may look something like this:

#pragma once
#include <string>

namespace server {
  const std::string VerUrl="https://github.com/<USER>/<REPO>/commit/<SHA_WILL_BE_HERE>";
  const std::string VerDate="2013-10-22 15:00:00 -0700";
  const std::string VerSubj="properly-tested-commit-enforcement";
  const std::string VerAuthor="The Developer <[email protected]>";
  const bool VerDirty=true;
}

Since each project generates version information by running it's designated pre-build step and the PowerShell script set up to write data into the project's directory I found no issues with parallel builds. I tried to force it by running the pre-build script on about twenty simple projects set to compile in parallel - and did not find any indication of a problem. That said, parallel execution is often wacky and you should check it as a culprit if run into weird/inconsistent behaviors.

Finally, let's pull this into the code base and have it log version information and enforce production builds running only out of clean repositories. Here is an example of what I am doing. I trust it you will know how to adjust the namespace and variables you use to customize the above steps:

#include "version.h"

void Server::versionLogAndVet() {

  Log::Info("Version Date: ",       server::VerDate);
  Log::Info("Version URL: ",        server::VerUrl);
  Log::Info("Version Info: ",       server::VerSubj);
  Log::Info("Version Author: ",     server::VerAuthor);
  Log::Info("Version Repo Clean: ", (server::VerDirty? "NO, it is DIRTY" : "yes" ));

#if defined(_DEBUG)
  Log::Info("Build configuration: DEBUG");

#elif defined(LOGGED)
  Log::Info("Build configuration: LOGGED");

#else
  Log::Info("Build configuration: RELEASE");

#endif

#if ! defined(_DEBUG)

  if (server::VerDirty)
  {
    Log::Error("Must NOT run production code build from a dirty repository. Server process STOPPED.");

    std::cerr << std::endl;
    std::cerr << "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!" << std::endl;
    std::cerr << "Must NOT run production code build from a dirty repository." << std::endl;
    std::cerr << "STOPPED.    Press  Enter key  to exit  or close the window." << std::endl;
    std::cerr << "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!" << std::endl;
    std::cerr << std::endl;

    exit(1);
  }

#endif

}

In that code all build configurations but those defining _DEBUG macro considered production builds and are not allowed to run the server process built from a dirty repo. Note, that this is a safety guard - this is NOT a security feature.

You can download all code exaples from this post as an archived file from the gist page.


Microsoft is apparently not considering the undefined parameter expansion behavior problem as a bug. They have closed the corresponding ticket with the "by design" reasoning, meaning the undefined behavior is OK with them and they will not fix it. Which means, that if any of the developers trying to use this solution run into the escaping issue, they will need to move or potentially restructure the affected repository.

read more ...

Monday, August 19, 2013

On justice in Bhutan

While in Bhutan, we naturally happened to compare crime in Bhutan and Western societies when talking to locals, and then switched to how communities deal with criminals. Apparently in Bhutan courts encourage local, "communal" conflict resolution for non-violent issues. This is more than a judicial attitude nuance.

As far as I understand, any society has some people demonstrating deviant behaviors. I suspect it is the context which society creates around deviant behaviors, which determines scope of their cumulative impact.

In Western societies, especially in the US, there is little personal touch in crime prevention. The punishment is often disproportionate. And community re-integration is hard and rare.

In Bhutan, with it's 700K population and about 100K in the capital, most people live in small communities (by Western measures). Crime prevention is naturally personal. People know who may be up to no good, observe what's going on, and catch many troubles before they happen. Of course I am not quoting from a statistical survey but from a single witness, a.k.a. anecdotal evidence. Yet it fits well into the rest of the picture we saw in Bhutan.

If, like I was told, judicial system is very dependent on a community, it creates all the difference in judging. No matter how emphatic is a judge, he or she judges a non-related person. And while it is the whole point of blind justice, it has a clear downside of formal law application. The problem is worsened by commercial interests lobbying to fill prisons. When villagers judge, they are very much considering somebody who is not foreign to them, and with whom they lived and will live for a while. They are also likely neighbors of the offender's family. It is a very different conflict resolution process with a natural re-integration done by the community.

I am not trying to romanticize Bhutan's court of community. It's potential dangers should be recognized. It is dependent on a particular Buddhist culture. It is not uniform - outcomes are unpredictable case-by-case. It will not scale to larger, weakly-connected communities. Heck, I know very little about it. Nevertheless I can not help but observe some of it's positive properties in the specific environment. And note it as a good example of a government being mindful of the nation's culture instead of adopting an "all-out" Western approach.

(Also, a somewhat related observation published in my other Google+ post on Bhutan)

read more ...

Monday, March 4, 2013

Patent-less information technology / what is your job

It seems that patent systems frustrate more and more people in information technology these days. From what I hear so far mostly patent holders, enforcers, their more aggressive practices, and abuse of intent of law are blamed for the trend.

I think there is another side of the story. Here is the foundation of my reasoning:

An invention patent is an exchange in it's core. A society promises an inventor the right to exclude others from using the invention for a period of time in exchange for the public disclosure of the invention.

For a patent holder this exchange is valuable as long as they are able to exploit the exclusivity period. The cheaper is the enforcement and the longer is the exclusivity period - the better business opportunity it presents.

For the society the exchange is valuable for two reasons. First, it guarantees availability of a non-trivial invention to a general public after a fixed period of time. Secondly, if done properly, it creates a double network effect - encouraging and enabling further inventions.

There are two concerns: (1) how did parties (en-masse) took care of their perceived obligations under the exchange and (2) whether the original premises are still as valuable and relevant as before.

The first concern is much talked about. The combination of patent being a sellable economic right, society's failure to guarantee exclusivity at a reasonable cost, and patenting trivial inventions - all contributed to a madhouse full of patent trolls, gorging lawyers, and companies clawing each other's throats. With disenfranchised individual inventors in the midst of it.

The second concern does not get enough attention - yet I believe it is directly responsible for the recent uproar against the patent system.

Up until recently (in historical time scale) inventors were rare and far between. There simply were not that many inventions around and businesses had the survival need to acuire inventions that were worth execution. It was also society's interest to avoid businesses hoarding inventions. Inventions themselves had value past the patent exclusivity period - meaning that it was more likely that the patent would expire without being re-invented, than someone else to come up with the same invention before the expiration. Think about it - if another inventor is very likely to solve same problem sooner than the exclusivity period is over, why is it a society's interest to protect the original inventor? The game of the exclusivity timeframe is a bet - where a society's bet is that it is more efficient to protect one inventor and give public access to the invention after a period of time, than to have the problem solved ad hoc individually by those who need it solved.

Nowadays a percentage of creative workers in the workforce is of no comparison to a hundred years ago. In information technology companies actually do foster conveyor-style innovation and blur the borderline between innovation and invention. When paired with proper prioritization, innovation happens with a fast enough pace to be considered invention after a short period of time. Invention is no longer driven by the lure of exclusively seizing an opportunity. It is driven by the basic modern business need to «innovate or die». Invention is no longer a differentiator. It is on par with the rest of business survival prerequisites: skills, priorities, motivation, and execution. Inventions no longer drive businesses. Businesses drive inventions.

What does this mean for patents?

For one thing it means that a society's incentives to grant exclusivity are much reduced. There is no upside the society has in exchange for the right to exclude. An ability to execute is a more scarce resource now, than innovation. Most benefit for the society is to incentivise not those who know how. But those who can prioritize, motivate, and deliver. And economy is doing it anyway. A business which gets priorities, motivation, and delivery right will find people to innovate and invent for it just fine - at the very least in information technology. May be in other industries as well, but I did not observe them enough in my life.

There is also no dire need for public disclosure of an innovation, should the innovator decide to keep it secret. It is very likely that someone will reinvent it faster, than the patent exclusivity period would have expired.

Looks like there is no benefit for the society to grant patents. Worse, patents get on a way and create frictions. Should information technology patents be just...... abolished?

No need to scream - “But who will innovate?!!!”

Employees and freelancers will. And academics at universities will - many tax-sponsored. Innovation is a part of a job for hire from now on.

No magic.

No exclusivity.

And no Bosch-hell-like patent underworld.


Creative Commons LicenseThis "Patent-less IT / what is your job" by Vlad Didenko is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

read more ...

Monday, February 25, 2013

Data Science as a science

For some time I felt unsatisfied every time going to the Wikipedia article on Data Science, a canonical Radar article and some other sources about Data Science. It does not feel, that there are any actual definitions of the term "Data Science". We see more descriptions of what Data Science includes, requires, or produces. Most of those seem to have a healthy dose of marketing or job security build inside. Kind of like the "Big Data" feeling — which may have some meaning to in a narrow circle of academics, yet completely lost it when abused by marketing teams.

If you look at the Wikipedia's Data Science Illustration, you will find that at least three leafs "feeding" Data Science concept are not defined themselves on the Wikipedia as of January 22d, 2013 - half a year after the file was uploaded. Specifically, "Domain Expertise", "Hacker Mindset", and "Advanced Computing" are intuitively familiar concepts, but not defined by themselves. Strictly speaking they are not fit to be a basis of another definition, being complex concepts themselves.

I think the reason for that is simple. The definition is overly complex. We should make it simple to have a change to solve problems. My suggestion is to apply the Occam's Razor principle. Cut everything but essentials off and see if that is enough.

A simple definition

Data Science is a data science.

In other words, consider defining the term Data Science through the combination of notions of data and science.

More practically, we may put it more verbose like this:

Data Science is an accumulated knowledge about transforming data streams using scientific method for the purpose of ongoing value extraction from alike data streams.

Data sets in this context need to have reoccurring commonalities to be valuable for consideration. With enough commonalities they effectively can be considered discrete forms of data streams.

An (incomplete) break down of Scientific Method

A common (among others) break down of scientific method is mentioned in the Overview of Scientific Method on Wikipedia as follows:

  • Formulate a question
  • Hypothesis
  • Prediction
  • Test (Experiment)
  • Analysis

(I was writing this when the excellent +Daniel Tunkelang post came along)

Later in the Wikipedia article and other resources (including Daniel Tunkelang) add extra steps to the process, most importantly:

  • Replication
  • Application

Application being a practical use of the obtained knowledge to create value.

Applying Occam's Razor tests

We now need to walk through the common definition (rather, description) of Data Science and see, if this short definition of "Data Science" as "data science" implies what is expressed in the wider common description from Wikipedia. I have selected what I feel most representative claims of Data Science from the Wikipedia article. Clearly, this is my biased selection, and I do not address repeating concerns.

Data science seeks to use all available and relevant data to effectively tell a story that can be easily understood by non-practitioners.

A result of scientific method is what is considered a "proven fact", meaning that a consumer of that fact can use it without possessing a special skills needed for the proof itself. Not very clear, what is understood as "telling a story" in the article, but I count that as check.

Data scientists solve complex data problems through employing deep expertise in some scientific discipline.

Complex data problems - check, deep expertise in some scientific expertise - check (considering that Data Science is a scientific discipline by definition).

It is generally expected that data scientists are able to work with various elements of mathematics, statistics and computer science… Data science must be practiced as a team, where across the membership of the team there is expertise and proficiency across all the disciplines

All mentioned areas of knowledge are necessary in a curriculum of one claiming themselves a Data Scientist in this definition - check. Must the scientific research in this context be a team effort - yes (see below on hardware knowledge and data size) - check.

Good data scientists are able to apply their skills to achieve a broad spectrum of end results [followed by examples of that spectrum]

Given data taxonomy or lingo (see below) at a right level of abstraction results of data science research can apply uniformly to data sets from diverse fields of knowledge - check.

What is the next step?

To live up to be a science, Data Science need to be able to pose questions in a way which allows hypothesis to be tested on multiple data streams. And as it stands right now, we cannot do it as a general case. It is very rarely we can predict, that "a data stream with qualities X,Y, and Z satisfies hypothesis H with α confidence." When we can make such a statement, it is great. However, there are no commonly accepted ways to describe a data stream. Without being able to describe the subject of our experiments precisely in its nature and scope we will not enable others to reasonably replicate or analyze any experiment we conduct.

With the need to describe our subject of study — data streams — it seems to me that the most dire need of Data Science is for a modern Carl Linnaeus to come to scene and create some sort of taxonomy of data and data streams. Although foundations of such taxonomy may already be present as branches of semiotics (syntax, semantics, and pragmatics), there is no unifying data taxonomy effort I am aware of, which would enable posting a meaningful hypothesis. Not to say that I am right on this one :-) .

It does not have to be a taxonomy. It may happen to be a common lingo, specific to each sub-field, like it happened in mathematics. The point is that to meaningfully follow scientific method when exploring data streams we need the ability to describe the nature and scope of data elements in streams being tested for a hypothesis.

Does size matter?

With all the disrespect to marketing uses of "Big Data" label, it is still important to understand if the data size matters for Data Science.

Like in every other science it does. Sample size may be too small to proof anything or too big and waste valuable resource. Think of this - you do not need a bowl of soup to decide if it is overly salted. If it is reasonably mixed, a table spoon will be enough. Same with data samples in any science - they need to be above threshold of statistical significance for us to be confident in the results.


Consider the confidence illustration in the article on statistical significance, also shown above. If signal to noise ratio will be determined by data quality and the quality of our data taxonomy or lingo, then the rest of confidence comes from less-then linear correlation with sample size. Bigger samples will improve the confidence, but processing more data once a comfortable confidence level achieved is just a waste of compute resources.

Given poor signal to noise ratio in some internet-generated streams, as well as potentially a very selective hypothesis (the one which makes a statement about a small subset of data), there is and will be a need to process massive amounts of data for the sake of a proof, especially on the intake end. That pulls in hardware, data layout and architecture expertise as a prerequisite to projects under those conditions.

Finally, consider the application step of the scientific process loop. As results of Data Science process are applied in practice, the value of the process increases with each chunk of data processed. By the nature of the way Data Science produces value it encourages more data being processed. Even if a hypothesis did not take much data to get to a proof, its' application, the engineering chunk of the process, may end up dealing with cumulatively large data sets.

A contentious issue of tools

It should matter if a prediction is tested in Hadoop or MongoDB only in one sense - that results are replicable using technology of any capable vendor. Likewise, when chemists test a spectral analysis prediction, it is not ultimately important which brand of spectrometer is used in the experiment, but it is important, that the prediction is confirmed outside of a single tool vendor's ecosystem.

Multiple vendor consideration may put extra constrains on how to specify data stream parameters in hypothesis.

Is it going to happen?

Science is traditionally carried out in academia and proprietary research labs. However, corporate research labs most often focus on engineering innovations, not theoretical science progress - so the likes of Linked-in, Facebook, or Google are unlikely to pick up this fairly abstract topic.

Some colleges offer what they consider Data Science training. Judging by course descriptions, though, those lean towards practical number-crunching skills, and not the application of scientific method to data. It is yet to be seen if any of them tackle generic abilities of Data Science - starting with precise definitions of data streams.

I will cite my lack of understanding of how academia works and withhold a prediction if a widespread scientific approach in Data Science (in common sense) is a possibility. Given current commercial focus of universities my bets are low.

And that is a pity, because we do not know a more effective methodology than scientific method to achieve a reliable knowledge.


This work is licensed to general public under the Creative Commons Attribution 3.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by/3.0/.

read more ...

Thursday, January 31, 2013

Compensation components and other terminology

The way I think of financial interactions between an enterprise and an employee is relatively simple. It has six components, which come together for a total package, if we put other benefits, like pensions, insurance, tuition, and similar aside for now. But before defining them, let me define three time horizons for the purpose of this discussion:

  • short-term - time comparable with the enterprise's financial planning period. From a quarter to couple years.
  • mid-term - time comparable with an average employment duration at the enterprise. From two to a dozen years.
  • long-term - Anything beyond mid-term. Often tied to expected company longevity or major shareholders' interest span.

The resulting thought I want to leave behind is that different compensation tools have specific purposes. I will also try to show how dangerous are attempts to achieve multiple effects with a single instrument, seemingly tuned for wider application. Finally it should show the importance of clear communication of compensation terms and intents behind them.

Salary

A guaranteed income for a defined set of functions with clear performance and quality expectations. The purpose of the salary is to foster a sense of stability and balance the urgency of "need it done tomorrow" with "will have to deal with my outcomes later". Undersized salary creates a negative attitude, hostile interactions between individuals and enterprise. What I have seen is that oversized salaries drove anxiety up, encouraged political posturing and other bad things. Yearly salary changes wider than 5% without changes in underlying expectations destroy the sense of stability and put people in the "underpaid" or "overpaid" buckets, with detrimental effects mentioned above. This is the only compensation component I know of which may and should send the stability message.

Commission

A portion of revenue promised to an individual or a group based on their specific and ongoing performance regardless of the enterprise bottom line. My experience in commission-driven sales teams brings the observation that commission drives very short-term-minded behaviors, usually tied to the reporting period, with planning horizon no further than 2-4 such periods. An interesting thing to observe in such teams is how different their definitions of time horizons. Something farther than four sales compensation cycles is beyond their view of long-term and does not worth more than a lounge chit-chat mention.

Bonus

A non-guaranteed, non-recurring achievement recognition. A tool to drive or encourage a short-term behavior. To be effective, bonus either can be a surprise "thank you" payment, or tied to an extremely thoroughly defined set of objectives. Things I observed as problematic when using bonuses: (a) creating an expectation of a bonus; (b) having vague bonus criteria; (c) for whatever reason refusing to pay bonus if criteria met; (d) announcing a bonus pool / making one person's bonus in reverse relation to another person or a group.

Specifically on each point:

  1. When bonus becomes routine, there is a conflict of expectations. An enterprise still believes it can freely change the bonus amount to drive short-term behaviors. An affiliate then sees bonus as a part of the salary and stability message. When enterprise actually fluctuates the bonus, then employees experience negative effects similar to salary fluctuation.
  2. A poorly defined bonus criteria drives manipulative behaviors, especially for hard to achieve tasks. People do "just enough" to be able to argue for the criteria met, or come up with reasons why effort should be compensated as well as the result would have. Often vaguely defined criteria is a sign of a lazy or incompetent management, one which feels they will not be held accountable, or one which is not motivated to even care about company's mid-term success.
  3. Companies and departments operate in dynamic contexts. Available funds may grow or shrink. It is totally wrong to expose that uncertainty to people at the bonus level. As a tool to drive a specific objective it deteriorates very fast if promises are broken. Employees start to see it as a carrot on a stick tied to their head. They will not run faster next time management wants them to. Because of that I think it is a very good idea to put promised bonus money aside in accounting procedures, almost in an escrow account.
  4. It so obviously creates inner tension in an enterprise, which takes forms of inner politics, manipulation, and sabotage, that it is mind-boggling how companies refer to it as "constructive inner competition". I have only seen it hurt companies in mid-term and beyond.

Profit sharing

A portion of an enterprise or a division profit promised to an individual on an ongoing basis. It is good to use as a mid-term priorities driver. To be effective for the mid-term horizon it needs to be relatively stable and well defined. It addresses people who are expected to see and impact a bigger picture both horizontally (multiple lines of business) as well as vertically (both top and bottom lines, or income and expenses).

Given that: profit sharing can not exceed a certain portion of a balance sheet; need to be relatively stable to have the desired mid-term impact; and should accommodate profit-sharing flexibility of future employees - it is important to avoid over-allocation of it, or to build in time-guards and absolute caps. Profit sharing is one type of compensation, which at least in one case I heard of it linearly declining in percentage over a course of 10 years down to a third of the original formula, with a review possibility.

In terms of what can be done wrong: revenue sharing does not belong to lower-impact levels in the organization. Without material ability to impact mid-term balance sheet most people I saw considered it as a part of salary, quite often as an insignificant corporate gimmick.

A frequently (yearly or similar) changing formula of profit sharing (which includes shifting percentages between departments), creates two problems.

  1. First, it's impact becomes short-term. The benefit itself plays an effective role of a bonus - and people start to treat it as such.
  2. Second, profit sharing is by it's nature is a piece of a pie in a very damaging way. So it becomes a bonus where one person's income is in reverse proportion to another person income. The company with such an arrangement will quickly find on it's hands the bonus problem (d) - and the culture will be destroyed in no time.

I have seen companies trying to combine bonus and profit sharing programs. In the specific case I am thinking about, quarterly bonus was tied to some very vague objectives and depended on a shared profit formula. This is close to an ultimate wrong - achieving none of the benefits, yet bringing all the damages.

Equity (affiliates)

I can think of two objectives when giving up equity to people working at the enterprise.

One is to motivate a long-term positive impacts from people, who are in a position for such impacts. It often takes form of option grants to leverage the motivation impact of the equity share. The pitfalls have to do with sizing and pricing the grants, as well as assessing cultural impact on the enterprise when the information becomes public - if the public's perception matters.

Another objective is to motivate early contributors in a situation when there is not enough money to immediately compensate them otherwise. What they really get is a recognition of their time and effort investment (or other resources) in a form of a portion of the company future value. By the nature of startup effort it is a high risk venture for those involved already. Adding uncertainty to the mix greatly reduces the perceived value of the reward. Reducing the uncertainty increases the perceived value.

Two major components of the uncertainty are being able to control a liquidity event and a pricing formula. Even though specific time or price are not known upfront, the clear definition of who and when can "press the button" is a must in all situations I have seen. For all I have heard for startups, equity vests after the first year - on average about when business viability becomes clear. Pricing formula is often not addressed for this early equity allocation for companies which drive towards a capital event - with a common expectation of a cash out to pay for the invested time and effort. I have seen startups which do not plan a capital event in a foreseeable time to spell out reasonable repurchase conditions, may be in a multiplier of rolling revenue or similar.

The biggest problem in both scenarios (start up and long-term motivation) I saw was when the status of equity at the time of separation was not clear - and there is no public market for that equity. When company leaves itself an option to buy or not to buy the equity out, then people stay longer than they should without much interest in their work. They hope that something will get clear "soon" - and rumor mill always finds something to feed the anxiety in that regard. All in all it is more beneficial to have a well defined separation process - and a valuation as soon as possible. That helps to move out of the way those who non-productive faster before the atmosphere becomes toxic.

The second issue creeps up is when shares are diluted faster then they appreciate absolute value. Equity owners obviously do not like to see the asset value to go down. Especially when they see it happening not because of market forces, but because they think management needs to please other people by issuing extra shares.

Equity (third parties)

As this write up talks about relationships between an enterprise and an employee, then the third parties with equity are most likely are past employees or heirs of past employees. Depending on equity transfer rules equity may also be gifted, split at a time of divorce, etc. For publicly traded company this likely is not an issue. However, for private equity company this may become an issue. So the units transferability should not be a part of the equity benefit, except for survivorship, inheritance, and where required by law.

I do not see any benefit or legitimate use for a company to encourage equity transfer away from current or past employees. I do see benefits of better focus when such equity is routinely bought out and kept to an unavoidable minimum.

The effects of mixing things

As I have mentioned at the very beginning, I saw much damage done when tools with different expected impact horizons - and purposes - stirred together. There is a reason the options are presented above in the order they are. A common management pattern is to use a concept made to benefit a farther horizon, but to modify it, so that it effectively plays role of a closer-horizon tool. Above I mentioned bonuses becoming part of a salary, profit sharing treated as a bonus. In many trading companies people talk about bonuses - when really those are commission arrangements. Even equity may be arranged in an operating agreement in a way to mimic profit sharing features.

A valid question is: why does it matter? Supposedly we can address the bad scenarios mentioned above, watch out for new problems and not to worry about intricacies of compensation terminology.

It needs to be addressed because quality of used terms affects the quality of communication. In a good case it causes people spend extra time to get on the same page with each new person involved. In a bad case it causes lasting misunderstanding and wrong business and personal choices. In a worst case it enables someone to use the ambiguity tor manipulation, create an arbitrage on different understandings of used concepts. There is no positive outcome to ignore the terminology.

I see three reasons for discrepancy between the intent and description to happen - besides an low-functioning plot to confuse employees.

The one reason I am on board with is potential tax benefits. If it is good for one or both of the sides, everyone clearly understands intentions and how they are arranged in compensation package, and everything is legal - nothing to worry about, the discrepancy is worth the nuisance.

Another observed reason for such discrepancy is when the compensation structure is created with very clear intentions and proper verbiage initial. However, it later shifts to accommodate new scale, tax, or legal realities. Sometimes in this process the original intentions are lost and new concepts serve current perceptions of the intentions, or new intentions altogether. This is not a good scenario, as it goes hand-in-hand with poor communication to employees. Really, if the management can not maintain continuity of intentions, what would it communicate to employees?

Finally, it is possible that the management acting with best intentions, however does not clearly map those intentions into compensation simply because of the lack of expertise or perceived priority. This is theoretically easier to fix than the previous case. It will take time and effort which are unavoidable. If not done carefully it may also leave a cultural scar behind. This is the scenario, which is easier to avoid than to fix.


This work is licensed under the Creative Commons Attribution 3.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by/3.0/.

read more ...