Author: Wellington Goncalves

Perforce Visual Client (P4V) – Baseless merges before version 2012.1

I needed to merge two branches without a common ancestor using Perforce Visual Client (P4V) 2015.2 but for my surprise the “-i” flag for p4 integrate has been removed. This flag is used to do indirect integrations. Indirect integration is when two branches are not directly related to each other like in the following example:

Indirect integration

Merging Y to Z is trivial because the two branches are directly related. However, going from X to Z is an indirect integration.

p4 integrate for release 2004.2 and later always considers (no need for “-i”) indirect integrations through intermediate branches when determining that files are related and what changes need to be integrated and I think that is the reason why the flag has been removed from P4V.

The problem happens when you need to do a baseless merge when the two branches are not related at all. In this case, you can’t use P4V and you will need to run the command with the “-i” flag.

Luckily, this has been fixed in version 2012.1 of Perforce and p4 integrate will now automatically schedule baseless merges to be performed with an empty base file. The “-i” flag is still accepted but has been deprecated.

I thought it was worth to write about it as there might be other people using an old version of Perforce.

For more information: http://answers.perforce.com/articles/KB/3394

Python – Generators

Do you know the difference between the following lines in Python?

square_numbers_1 = [x*x for x in [1,2,3,4,5,6]]
square_numbers_2 = (x*x for x in [1,2,3,4,5,6])

The first line will create a list of squares for each element of [1,2,3,4,5,6] in memory which means you can access it by using square brackets:

# This will return 1
print square_numbers_1[0]

The second line will return a Generator object which means you cannot use the square brackets.
You can access the values using a for loop or the next() method.

# This will print all items:
# 1
# 4
# 9
# 16
# 25
# 36
for number in square_numbers_2:
    print number 

# This will return one item each time.
# The following line will return 1.
print next(square_numbers_2)

You need to call ‘next’ whenever you need an item and you will receive an exception (StopIteration) when you call it more than the number of items. More than six times for square_numbers_2.

You can also create a Generator by calling yield as in the following example:

def square_numbers(numbers):
    for n in numbers:
        yield (n*n)

square_numbers_generator = square_numbers([1,2,3,4,5,6])

for number in square_numbers_generator:
    print number

Why is Generators useful?

  • Reduce the amount of memory used as it will only allocate memory when you retrieve an item
  • Run faster because it stops when it gets to the first yield in the above example

Python – httplib

You might want to use the requests module instead which is easier to use.
If you need to use httplib, this article is for you.

Here is a basic example on how to use httplib to access TeamCity REST API using just GET and POST:

Global variables:

TEAMCITY_SERVER="<teamcity-server>/httpAuth/app/rest/"
TEAMCITY_USER="user"
TEAMCITY_PASSWD="password"

Method for the HTTP request with basic HTTP authentication.
It returns a tuple with the following fields: status, reason and body:

def teamcity_api_request(url, data=None):
    """
    Send a TeamCity request using REST API
    """
    try:
        verb = 'POST' if data else 'GET'
        print "Sending HTTP " + verb + ": " + TEAMCITY_SERVER + url
        conn = httplib.HTTPConnection(TEAMCITY_SERVER)
        user_and_pass = b64encode(TEAMCITY_USER + ":" + TEAMCITY_PASSWD).decode("ascii")
        headers = {'Authorization':'Basic %s' %  user_and_pass, "Content-Type": "application/xml", "Accept": "application/json"}
        if data:
            print "HTTP Message Body: " + data
            conn.request(verb, url, data, headers=headers)
        else:
            conn.request(verb, url, headers=headers)
        res = conn.getresponse()
    except:
        print "Unexpected error:"
        print sys.exc_info()[0]
        print sys.exc_info()[1]
        print sys.exc_info()[2]
        print "Error while retrieving TeamCity information!"
        sys.exit(1)

    res_status = res.status
    res_reason = res.reason
    res_body = res.read()

    # Dumping out request
    print "--------------- HTTP response --------------------"
    print "Status: " + str(res_status)
    print "Reason: " + res_reason
    print "Body: "   + res_body
    print "--------------------------------------------------"

    # Stop execution if HTTP code is different than 200
    if res_status != 200:
        print "HTTP code is different than 200. Aborting..."
        sys.exit(1)

    request = collections.namedtuple('Request', ['status', 'reason', 'body'])
    # Convert body to json if request was successful
    if res_status == 200:
        res_body = json.loads(res_body)
    req = request(status=res_status, reason=res_reason, body=res_body)
    conn.close()
    return req

How to use it:

# GET - get a list of builds
request = teamcity_api_request("/builds")
print request.status

# POST - cancel a build
request = teamcity_api_request("/buildQueue/id:<build ID>", "<buildCancelRequest comment='<comment>' readdIntoQueue='true' />")
print request.status

Linux – How to cut a log file based on timestamp

You can use the command ‘sed’ to specify an interval to cut a log file. ‘sed’ performs editing operations on information coming from standard input or a file. It edits line-by-line and in a non-interactive way.

Example to cut a log file based on timestamp:

sed '1,/2017\/01\/11 09\:00/ d;/2017\/01\/11 10\:00/,$ d' <logfile>

This command will just show lines between ‘2017/01/11 09:00’ and ‘2017/01/11 10:00’.
In reality, the command is removing all lines before ‘2017/01/11 09:00’ and after ‘2017/01/11 10:00’ (including lines with ‘2017/01/11 10:00’).

You need to modify the command to match your log file and your needs.

Bash – How to return a value in a function

Here are some ways to return a value in a function in a Bash script:

1. Strings

function foo()
{
    echo "true"
}

retval=$(foo)

if [ "$retval" == "true" ];
then
     echo "true"
fi

2. Exit status

function foo()
{
    return 0
}

foo
retval=$?

if [ $retval == 0 ];
then
     echo "okay"
fi

3. Shared variable

retval=-1

function foo()
{
    retval=0
}

foo

if [ "$retval" == 0 ]
then
     echo "okay"
fi

Adapted from: http://stackoverflow.com/questions/8742783/returning-value-from-called-function-in-shell-script

TeamCity agent doesn’t show the application window when a build is run

If you are running TeamCity agent as a service in Windows you probably noticed that you cannot see any application window that your build agent is running.

You don’t see any window because, for security reasons, the latest versions of Windows runs services in Session0 and not Session1. Sometimes you need to see it, specially if you running tests. If you need to run TeamCity agent on Session1 and not Session0 and be able to see what is going on without having to access Session0 you shouldn’t be running your TeamCity agent as a service.

In the installation wizard, you should untick the option to install the agent as a service:

teamcity-service

After installing it, you need to configure the build agent to be run automatically when the machine starts. You need to add a shortcut to “<installation path>\bin\agent.bat start” in “C:\ProgramData\Microsoft\Windows\Start Menu\Programs\Startup” for all users.

Reference: https://confluence.jetbrains.com/display/TCD9/Setting+up+and+Running+Additional+Build+Agents#SettingupandRunningAdditionalBuildAgents-agent_start_windows

DevOps for offline products

It is much easier to talk about DevOps in companies in which their products are online but in companies in which most applications, if not all, run in the customer side and we don’t have access to them is very challenging.

One of the most important values of DevOps is to have a “Ops Ready” application. A “Ops Ready” application means that Operation team should be able to detect problems in it easily.

A “Ops Ready” application has a good way to be monitored after deployment so you can anticipate problems and avoid hearing from customers. A very good way is to have some kind of heartbeat in all system components which sends information about the system and the machine it is running on. In the end, you should figure a problem out instead of being told about it.

Although a very good idea, it is very hard (or impossible) to have this kind of heartbeat system when the application normally  runs offline. It is harder but we still need to have some way to track the problems down.

The solution for this problem depends on the type of application you are working on but you should create some kind of offline monitoring mechanism and have this data stored somewhere and maybe sent to you once a connection is available. Supporting an application if this kind of mechanism is much easier.

You should take privacy into account but this is a subject for a different post.