The magic of CLI

You want to turn something like this:
[bash]
chartsales_messages_es.properties
chartsales_messages.properties
closedpos_messages_es.properties
closedpos_messages.properties
closedproducts_messages_es.properties
closedproducts_messages.properties
customersdiary_messages_es.properties
customersdiary_messages.properties
customers_messages_es.properties
customers_messages.properties
inventoryb_messages_es.properties
inventoryb_messages.properties
inventorydiff_messages_es.properties
inventorydiff_messages.properties
inventory_messages_es.properties
inventory_messages.properties
people_messages_es.properties
people_messages.properties
productlabels_messages_es.properties
productlabels_messages.properties
productsales_messages_es.properties
productsales_messages.properties
productscatalog_messages_es.properties
productscatalog_messages.properties
products_messages_es.properties
products_messages.properties
taxes_messages_es.properties
taxes_messages.properties
usersales_messages_es.properties
usersales_messages.properties
[/bash]

into this:

[bash]
chartsales_messages.properties
chartsales_messages.properties
closedpos_messages.properties
closedpos_messages.properties
closedproducts_messages.properties
closedproducts_messages.properties
customersdiary_messages.properties
customersdiary_messages.properties
customers_messages.properties
customers_messages.properties
inventoryb_messages.properties
inventoryb_messages.properties
inventorydiff_messages.properties
inventorydiff_messages.properties
inventory_messages.properties
inventory_messages.properties
people_messages.properties
people_messages.properties
productlabels_messages.properties
productlabels_messages.properties
productsales_messages.properties
productsales_messages.properties
productscatalog_messages.properties
productscatalog_messages.properties
products_messages.properties
products_messages.properties
taxes_messages.properties
taxes_messages.properties
usersales_messages.properties
usersales_messages.properties
[/bash]

I can’t imagine an easy way to do it with Windows but with bash and any Unix like OS, you get it done in less than a minute.

[bash]
for line in $(ls *.properties)
do
mv $line echo $line | sed "s/_es.properties/.properties/"
done;
[/bash]

Dialer: tracking calls with Asterisk/AMI

Long story short: I was asked to write a real time dialer callable from a web page that takes the number from the parameters received, make the call, and play a pre-recorded message when the called picks up. This post is not about the dialer itself, it’s about tracking the call originated by it, detect the “answer” event, detect the “hangup” event and save a CDR.

The approach I chose involves AMI to generate and track the calls. It offers a really nice interface based on a text protocol and the most important thing, the API can be mapped in almost any programming language like C# (Asterisk.NET).

Some of the protocol internals

  1. The connection is made via TCP/IP sockets using user/password authentication.
  2. Once logged in, you receive every event on the system, even if that event doesn’t belong to you. This could be a real mess because it introduces a lot of noise into the dialer and the software must support some kind of filtering mechanism.
  3. Every event comes with certain data, including a field named Uniqueid which relates the events to a transaction.
  4. The order the events were received not necessarily match the order they were generated.

The native way to generate a call is through the OriginateAction but the problem is that OriginateAction doesn’t reply with the uniqueID of the transaction, it just tells you whether the call was successfully queued or not. Then, you start receiving the events with the uniqueIDs but you have no way to know if that call belongs to you, to another dialer instance, to a human making a call, etc.

Searching the web I found what I thought was the solution: use asynchronous origination.  The usual OriginateAction is a blocking call that returns only if success or failure. With async origination the answer to the request is delivered through the OriginateResponse event and with the parameters you receive, the so wanted uniqueID. I built the dialer on top of this “fix” just to later discover that if the call fails, the uniqueID is empty. Once again, you have no way to know which call failed and for what reason.

I was so pissed off because something as basic as this wasn’t correctly available, or at least, there is no documentation to expose an standard solution. I started considering parallel approaches to abandon AMI but reading the input parameters for OriginateAction a dirty workaround came to my mind.

OriginateAction takes as input a field named CallerID, this field is always available and always filled with information. I used this field to uniquely identify the call whether it succeeded or not. In my particular case, the calls goes to a GSM network so overwriting the callerID has no effect and the call follows its normal destination.

Bellow is part of the dialer class. If you can read C#, it would explain itself.

[csharp]

public class Dialer
{
ManagerConnection _managerConnection;
Timer _timer;
public delegate void CallFinalizedCallback(Dialer sender);
string _uniqueID = string.Empty;
bool _wasOriginated = false;
object _locker = new object();

List<HangupEvent> _hangupList = new List<HangupEvent>();

CallFinalizedCallback _callFinalized;
public CallFinalizedCallback CallFinalized
{
get {
return this._callFinalized;
}
set {
_callFinalized = value;
}
}

CallInfo _callInfo;
public CallInfo CallInfo
{
get
{
return this._callInfo;
}
}

public Dialer()
{
_managerConnection = new ManagerConnection(Configuration.ManagerHost,
Configuration.ManagerPort,
Configuration.ManagerUserName,
Configuration.ManagerPassword);

//_timer = new Timer(Handle_timeoutCallback, Configuration.CallNotificationTimeout);

AddEventHandlers();
}

void Handle_managerConnectionOriginateResponse (object sender, OriginateResponseEvent e)

{
if (!e.CallerIdNum.Equals(_callInfo.CallId.ToString()))
return;

Logger.Print(string.Format(“Call to {0}-{1} was originated”, _callInfo.CallId, e.Exten));

bool succeeded = e.Response.ToLower().Equals(“success”);

if (succeeded)
UpdateDialoutQueue(CallStatus.Established);
else
UpdateDialoutQueue(CallStatus.FinishedWithError);

lock(_locker)
{
_wasOriginated = true;
_uniqueID = _callInfo.CallId.ToString();
}

foreach(HangupEvent hEvent in _hangupList)
{
if (hEvent.CallerIdNum.Equals(e.CallerIdNum)) // the call failed for some reason
{
Handle_managerConnectionHangup(this, hEvent);
break;
}
}
_hangupList.Clear();

}

public Dialer(CallInfo callInfo, CallFinalizedCallback callback) : this()
{
_callInfo = callInfo;

if (callback == null)
throw new ArgumentException(“CallFinalizedCallback cannot be null”);

_callFinalized = callback;
}

void AddEventHandlers()
{
_managerConnection.Hangup += Handle_managerConnectionHangup;
_managerConnection.OriginateResponse += Handle_managerConnectionOriginateResponse;
}

public void DisconnectFromAMI()
{
_managerConnection.Hangup -= Handle_managerConnectionHangup;
_managerConnection.OriginateResponse -= Handle_managerConnectionOriginateResponse;
}

void Handle_timeoutCallback()
{
try
{
Logger.Print(string.Format(“Call {0}-{1} forced to finalize after timeout”, _callInfo.CallId, _callInfo.Extension));

UpdateDialoutQueue(CallStatus.Timedout);
//_timer.CancelAlarm();

if (_callFinalized != null)
_callFinalized(this);

DisconnectFromAMI();
}
catch(Exception ex)
{
Logger.Error(string.Format(“Handle_timeoutCallback: {0}”, ex.Message));
DisconnectFromAMI();
}
}

void Handle_managerConnectionHangup(object sender, HangupEvent e)
{
try
{
lock(_locker)
if (!_wasOriginated || !_uniqueID.Equals(e.CallerIdNum))
{
_hangupList.Add(e); // register this event in case it was fired before the OriginateAction event
return;
}

UpdateDialoutQueue(CallStatus.Finished, e.Cause, e.CauseTxt);
//_timer.CancelAlarm();

Logger.Print(string.Format(“Call {0}-{1} was hung up”, _callInfo.CallId, e.Channel));

if (_callFinalized != null)
_callFinalized(this);

DisconnectFromAMI();
}
catch(Exception ex)
{
Logger.Error(string.Format(“Handle_managerConnectionHangup: {0}”, ex.Message));
DisconnectFromAMI();
}
}

public void Call()
{
Call(_callInfo);
}

public void Call(CallInfo callInfo)
{
try
{
OriginateAction originate = new OriginateAction();

originate.Channel = callInfo.Channel;
originate.Priority = callInfo.Priority;
originate.Context = callInfo.Context;
originate.Exten = callInfo.Extension;
originate.CallerId = callInfo.CallId.ToString(); // <——–
originate.Timeout = 500000;
originate.Async = true;

_managerConnection.Login();

MarkCallAsProcessed();

//_timer.SetAlarm();

UpdateDialoutQueue(CallStatus.Trying);
ManagerResponse r = _managerConnection.SendAction(originate, originate.Timeout);
}
catch(Exception ex)
{
Logger.Error(string.Format(“Call: {0}”, ex.Message));

//_timer.CancelAlarm();

if (ex is Asterisk.NET.Manager.TimeoutException)
{
UpdateDialoutQueue(CallStatus.Timedout);
DisconnectFromAMI();
}
else
UpdateDialoutQueue(CallStatus.FinishedWithError);
}
}
}

[/csharp]

 

Date/Time operations

Since I am not very fanatic of software homogeneity, when I write perl scripts and I need some function that performs certain task I use commands from console whenever it’s available even knowing that could be an equivalent Perl module available out there.

When your main goal is to get things done in the least amount of time, you cannot afford the time wasted in searching a library/module when the tools are just as far as calling a external program. This doesn’t mean that your code has to look horrible, it means that there’s no need to search for a (long) piece of code to attach it to yours in order to make your program homogeneous. After all, if you know how to write clean code, you will do it even if you mix 2323423423 languages inside your perl script.

Date/Time arithmetic using Perl.

I am aware of the existence of DateTime.pm but I have no experience using it. This was not the case with date command which finally led me to skip the learning curve and pass directly to the coding job.

The problem.

From a CDR entry containing end-datetime and duration, calculate the start-time, and transform the output using the YYMMDD/HHMMSS format.

[perl]

use strict;
use warnings;

sub main()
{
my $startTime = “02/01/2011 12:02:00”;
my $secs = 3650;

print “$startTime – $secs secs (YYMMDD HHMMSS) = @{ [ CalculateDate(“$startTime”, $secs) ] }”;
print “$secs secs in HHMMSS format is = @{ [ FormatTime($secs) ] } n”;
}

sub CalculateDate($$)
{
my ($date, $secs) = @_;

$date = FormatDateTime(date --date "$date $secs seconds ago");

return $date;
}

sub FormatDateTime($)
{
return date --date "$_[0]" +"%y%m%d %H%M%S";
}

sub FormatTime($)
{
my ($seconds, $hours, $mins, $secs) = shift;

$hours = padLeft(int($seconds / 3600));

$mins = padLeft($hours == 0 ? int($seconds / 60) : int(($seconds – ($hours * 3600)) / 60));

$secs = padLeft($hours == 0 && $mins == 0 ? $seconds : $seconds % 60);

return “$hours$mins$secs”;
}

sub padLeft($)
{
my $value = shift;
return (length “$value” == 1) ? “0$value” : $value;
}

main();
[/perl]

And the output is

[bash]
$ perl convert.pl
02/01/2011 12:02:00 – 3650 secs (YYMMDD HHMMSS) = 110201 110110
3650 secs in HHMMSS format is = 010050
[/bash]

Now, what I did not found searching the web was an easy method to convert seconds to a hour:minute:second format. This forced me to write my own algorithm which, hopefully, is bug free 😀
[perl]
sub FormatTime($)
{
my ($seconds, $hours, $mins, $secs) = shift;

$hours = padLeft(int($seconds / 3600));
$mins = padLeft($hours == 0 ? int($seconds / 60) : int(($seconds – ($hours * 3600)) / 60));
$secs = padLeft($hours == 0 && $mins == 0 ? $seconds : $seconds % 60);
return “$hours$mins$secs”;
}
[/perl]

What would a portability freak say?

It would probably say that my solution ignores the portability features of Perl. Yes, it is true but I don’t really care since my target is, and will always be, Unix based machines.

 

Merging PDF files with Linux

Almost everything I write here is related to problems I face daily and this time I came with something new and not very common :D.

A client of mine asked me to find a solution which allow him to combine several PDFs files into a single one without the “complicated” steps of Windows software. I think there is no free solution to do that and the one I know, Adobe Reader, comes with this feature only with paid licence.

The physical scenario involve a Linux file server reachable by a network of Windows machines acting as SMB/CIFS clients and the procedure to merge the files looks like the following:

– The file server shares, with read/write permissions, a directory (folder) called zip2pdf.
– If a user wants to combine a series of files he just need to zip the files and copy the compressed file into the shared directory.
– After a certain time, proportional to the size of the documents to be merged, a PDF file will “appear” in the shared directory containing the PDFs combined.

How I did this?

– I used a filesystem watcher which notifies me every change on a target directory, in my case, file creation.
– I unzipped the file, sorted the output and merged the PDFs with ghostscript.
– I put it all in a bash script and ran it in background.

[bash]
#!/bin/bash
inotifywait –monitor –event CREATE –format “%w%f” $1 | while read file
do
MATCH=echo $file | grep -P ".zip$"
if [ “$MATCH”!=”” ] && [ -e $file ];
then
sleep 3 # wait an arbitrary value to “ensure” the file was closed.
# There are another ways to check it using lsof for instance,
# but it would complicate the script a lot

STAMP=date +"%Y-%m-%d_%N"
DIR=”uncompressed_$STAMP/”

mkdir -p $DIR

unzip $file -d $DIR

FILES=find $DIR -type f -iname *.PDF | sort
#echo “LIST: $FILES”
gs -dNOPAUSE -sDEVICE=pdfwrite -sOUTPUTFILE=$STAMP.pdf -dBATCH $FILES

mv $STAMP.pdf $1
rm -rf $DIR
else
echo NOOOOOOOOOOO
fi
done
[/bash]

[bash]
$ ./z2p.sh /home/carlos/Desktop/zip2pdf/
[/bash]

It works and it was pretty simple 😀