Monday, April 7, 2014

Shell Script Programming Challenge

I'm just trying give myself a test on problem solving skill using Bash programming.

Given a scenario where the file, say file_A, locate in directory dir_A is required to rename into this format pattern <file>_yyyymmdd.txt. I’m require to write a bash script to accomplish this task. In addition to this, dir_A sometimes may contain multiple files, the script must be able to handle such situation.

Here is my work:
...
...

for XFILE in `ls ${DIR_A} | grep -v "\."`
do
   mv ${XFILE} ${XFILE}_`date +%Y%m%d`.txt
done
Surprise me! It just take 4 lines of code.

Wednesday, April 2, 2014

Failed to transfer file with SFTP in batch mode

There is a problem when I transferring files over SFTP in batch mode. Say I have the file transfer script that do this:
sshpass –e sftp –b filelist.txt theuser@123.123.12.123:/home/theuser << EOF
bye
EOF
Assuming there is no problem found in filelist.txt, if the file transfer script is execute, the following error would be seen:

Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
Couldn't read packet: Connection reset by peer


Running the sftp command with –v option shows that it seems not able to find the key pair for authentication. Later I found out that missing oBatchMode option may failed the process, thus this should be the work around I found in forum.
sshpass –e sftp –oBatchMode=no –b filelist.txt theuser@123.123.12.123:/home/theuser << EOF
bye
EOF

Tuesday, April 1, 2014

Passing authentication with SSHPASS

There is a requirement to create a cron script that will transfer files through SFTP command. But one problem I’m facing here is the script will prompt for password in order to proceed its job. This is not a correct behavior since a cron must be able to perform the task without human interaction.

Since the server is not yet ready for me to play around with, and I don’t know how to setup public/private key pair, the only way I can opt for is SSHPASS command. I think this should be the cheapest solution I could opt for. I do understand that SSHPASS is insecure and not encourage to use, but this should be fine under development stage.

Assuming I’m going to transfer a file to a destiny location at /home/theuser with username theuser and IP 123.123.12.123. With SFTP, this is what I’ll do in the script:
sftp theuser@123.123.12.123:/home/theuser << EOF
put thefile.txt
bye
EOF
Whenever this script is run, I’ll get a prompt for password then only the process will continue. Now if I do it with SSHPASS command, this is how it look like:
sshpass –p ‘password’ sftp theuser@123.123.12.123:/home/theuser << EOF
put thefile.txt
bye
EOF
There is also an alternate solution as shown below, by keeping the password in environment variable but which is strongly not a good way to go.
export SSHPASS=password
sshpass –e sftp theuser@123.123.12.123:/home/theuser << EOF
put thefile.txt
bye
EOF

Compiling C code with Informix 4GL

When I was assigned a task that involve integration work between MQ and 4GL, this was really exciting me because this will involve C programming language since C is my favorite language. But one thing to note is that this is not the regular C I’ve been doing in the pass. It is highly dependent on the version of Informix version I’m currently running.

As I ask around, the Informix environment we are running is called Rapid Development System, RDS for short. In this enviroment, there is a program that used to make the C code, named cfglgo.v2. It is not a compiler, rather it is a P-code (pseudocode) runner. My senior told me that there are also c4gl or i4gl, but these are not usable in ours environment. Following is the example on how this runner is call:

cfglgo.v2 -I/opt/mqm/inc fgiusr.c myrunner.c /opt/mqm/lib64/libmqic.so -o myrunner

Following output should be seen after the execution on the above statement:
Reading specs from /usr/local/lib/gcc/sparc-sun-solaris2.10/3.4.6/specs
Configured with: ../configure --with-as=/usr/ccs/bin/as --with-ld=/usr/ccs/bin/ld --enable-shared --enable-languages=c,c++,f77
Thread model: posix
gcc version 3.4.6
 /usr/local/libexec/gcc/sparc-sun-solaris2.10/3.4.6/cc1 -quiet -v -I/usr/informix/incl/tools -I/usr/informix/incl/esql -I/opt/mqm/inc -D__arch64__ -D__sparcv9 fgiusr.c -mptr64 -mstack-bias -mno-v8plus -mcpu=v9 -quiet -dumpbase fgiusr.c -m64 -auxbase fgiusr -version -o /var/tmp//ccNzNEmS.s
ignoring nonexistent directory "NONE/include"
ignoring nonexistent directory "/usr/local/lib/gcc/sparc-sun-solaris2.10/3.4.6/../../../../sparc-sun-solaris2.10/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/informix/incl/tools
 /usr/informix/incl/esql
 /opt/mqm/inc
 /usr/local/include
 /usr/local/lib/gcc/sparc-sun-solaris2.10/3.4.6/include
 /usr/include
End of search list.
GNU C version 3.4.6 (sparc-sun-solaris2.10)
        compiled by GNU C version 3.3.2.
GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
 /usr/ccs/bin/as -V -Qy -s -xarch=v9 -o /var/tmp//cc4f9CWC.o /var/tmp//ccNzNEmS.s
/usr/ccs/bin/as: Sun Compiler Common 10 Patch 09/04/2007
 /usr/local/libexec/gcc/sparc-sun-solaris2.10/3.4.6/cc1 -quiet -v -I/usr/informix/incl/tools -I/usr/informix/incl/esql -I/opt/mqm/inc -D__arch64__ -D__sparcv9 myprogram.c -mptr64 -mstack-bias -mno-v8plus -mcpu=v9 -quiet -dumpbase myprogram.c -m64 -auxbase mqfunc -version -o /var/tmp//ccNzNEmS.s
ignoring nonexistent directory "NONE/include"
ignoring nonexistent directory "/usr/local/lib/gcc/sparc-sun-solaris2.10/3.4.6/../../../../sparc-sun-solaris2.10/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/informix/incl/tools
 /usr/informix/incl/esql
 /opt/mqm/inc
 /usr/local/include
 /usr/local/lib/gcc/sparc-sun-solaris2.10/3.4.6/include
 /usr/include
End of search list.
GNU C version 3.4.6 (sparc-sun-solaris2.10)
        compiled by GNU C version 3.3.2.
GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
 /usr/ccs/bin/as -V -Qy -s -xarch=v9 -o /var/tmp//ccv77fcB.o /var/tmp//ccNzNEmS.s
/usr/ccs/bin/as: Sun Compiler Common 10 Patch 09/04/2007
 /usr/local/libexec/gcc/sparc-sun-solaris2.10/3.4.6/collect2 -V -Y P,/usr/lib/sparcv9 -Qy -o fglmq /usr/local/lib/gcc/sparc-sun-solaris2.10/3.4.6/sparcv9/crt1.o /usr/local/lib/gcc/sparc-sun-solaris2.10/3.4.6/sparcv9/crti.o /usr/ccs/lib/sparcv9/values-Xa.o /usr/local/lib/gcc/sparc-sun-solaris2.10/3.4.6/sparcv9/crtbegin.o -L/usr/informix/lib -L/usr/informix/lib/esql -L/usr/informix/lib/lib/tools -L/usr/local/lib/gcc/sparc-sun-solaris2.10/3.4.6/sparcv9 -L/usr/local/lib/gcc/sparc-sun-solaris2.10/3.4.6 -L/usr/ccs/bin/sparcv9 -L/usr/ccs/bin -L/usr/ccs/lib/sparcv9 -L/usr/ccs/lib -L/usr/local/lib/gcc/sparc-sun-solaris2.10/3.4.6/../../../sparcv9 -L/usr/local/lib/gcc/sparc-sun-solaris2.10/3.4.6/../../.. -L/lib/sparcv9 -L/usr/lib/sparcv9 /var/tmp//cc4f9CWC.o /var/tmp//ccv77fcB.o /opt/mqm/lib64/libmqic.so /usr/informix/lib/tools/libfmain.a /usr/informix/lib/tools/libfglgo.a /usr/informix/lib/tools/lib4gl.a /usr/informix/lib/tools/lib4io.a /usr/informix/lib/tools/libnmenu.a /usr/informix/lib/tools/lib4io.a /usr/informix/lib/tools/librdsterm.a /usr/informix/lib/tools/libfmain.a /usr/informix/lib/tools/libfe.a /usr/informix/lib/tools/libfmain.a -ltermlib -liffgisql -lifasf -lifgen -lifos -lifgls -lnsl -lsocket -ldl -lm /usr/informix/lib/esql/checkapi.o -lifglx -laio -lm -ldl -lelf -ltermlib -lgcc -lgcc_eh -lc -lgcc -lgcc_eh -lc /usr/local/lib/gcc/sparc-sun-solaris2.10/3.4.6/sparcv9/crtend.o /usr/local/lib/gcc/sparc-sun-solaris2.10/3.4.6/sparcv9/crtn.o
ld: Software Generation Utilities - Solaris Link Editors: 5.10-1.1518

The final output of this will generate a custom runner called myrunner. This runner will contain the required C implementation that is needed to execute a 4GL program. On 4GL site, the program must compile using fglpc in order to link with the runner. Assuming the the_4gl_program.4gl has successfully compile into the_4gl_program.4go, this would be the step to launch 4GL program:

myrunner the_4gl_program

Saturday, March 15, 2014

mqlicense unable to locate Java Program

In the recent WebSphere MQ version (v7.0.1.3 as of this writing) installation on Ubuntu 13.04, I have some difficulty issue in accepting the WebSphere license. The output shows that the JAVA program went missing. Below is the message being prompt for the missing JAVA program.
...

ERROR: No useable java programs at ./lap/jre/jre/bin/java 

ERROR:  Installation will not succeed unless the license
        agreement can be accepted.

        If the error was caused by a display problem,
        read the license agreement file  (LA_xx, where
        xx represents your language ) in the 'licenses'
        directory, and then run the following command:

            mqlicense.sh -accept

        Only use this command if you accept the license
        agreement.

        For other errors, contact your IBM support centre.
I find it interesting why the mqlicense shell script is looking for the JAVA program in ./lap/jre/jre/bin. My curiosity is leading me to dig into mqlicense.sh to find out what is going on to the script, and I found this piece:
...

# Set JRE location
  JRE=${JRE:-$(find $PROGPATH/lap -type d -name bin)/java }
  if [ ! -x ${JRE?} ]; then
    echo "ERROR: No useable java programs at ${JRE}"
    errormsg
    exit 1
  fi

...
I was so surprise that WebSphere isn't install the JAVA program in default location (/bin or /usr/bin). Anyhow I just change the JRE variable to suit my needs. Now the screen will turn out a license agreement page for me to accept. Once accepted, following message will be seen:
...
...

Displaying license agreement on :0

Agreement accepted:  Proceed with install.

Thursday, March 13, 2014

Preventing users from committing specific files to SVN

There was an incident where someone has misuses SVN (Subversion) as a file server which causes the server running out of disk space. I have no choice, in order control such an irresponsible attitude staff; I’m force to implement a security check on each file whenever a commit request has received.

The script is implemented in pre-commit hooks script. This file locate inside the hooks directory of each repository, initially it has a file named pre-commit.tmpl, I’m require to remove the tmpl extension and grant this file with execute access right.

Append the following content into pre-commit file.
#Put all the restricted formats in variable FILTER
FILTER=".(xls|xlsx|exe|pptx|PPTX|vsd|VSD|bak|BAK|class|CLASS|zip|ZIP|doc|DOC|docx|DOCX)$"

# Capture extension token of each new files to be commit
FILES=$(${SVNLOOK} changed -t ${TXN} ${REPOS} | ${AWK} '{ print $NF }') > /dev/null

for FILE in $FILES 
do
  EXTENSION=`echo "$FILE" | cut -d'.' -f2-`

  if [[ "$FILTER" == *"$EXTENSION"* ]]; then
    echo "Your commit has been blocked because you are trying to commit a restricted file." 1>&2
    echo "Please contact SVN Admin. -- Thank you" 1>&2
    exit 1
  fi
done
Now whenever users commit a file or files having the extension mention in the FILTER, the commit will not be success.

Sunday, March 9, 2014

apt-get gone crazy due to my mistake

I was installing WebSphere MQ for my own research. Due to some technical mistake, I have cause some trouble on apt-get command not able to function. Below is the error I got:
...
...

chown: invalid user: ‘mqm:mqm’
chown: invalid user: ‘mqm:mqm’
chown: invalid user: ‘mqm:mqm’
dpkg: error processing mqseriesserver (--configure):
 subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
 mqseriesserver
Not knowing why this could happened to me? Due to my mistake I guess? Then I spot apt-get autoremove would lend me a help. What this command actually do?
autoremove is used to remove packages that were automatically installed to satisfy dependencies for other packages and are now no longer needed.
I did try on that but no luck. While I'm almost hopeless, I see some light, it is pointing to apt-get remove --purge. What this command actually do?
--purge
Use purge instead of remove for anything that would be removed. An asterisk ("*") will be displayed next to packages which are scheduled to be purged. remove --purge is equivalent to the purge command. Configuration Item: APT::Get::Purge.
For my case, I'm going to purge any of the mqseriesserver thing, thus by launching following command:
sudo apt-get remove --purge mqseriesserver
I can see the rubbish is being purge from apt-get command:
...

Removing mqseriesserver ...
Purging configuration files for mqseriesserver ...
Processing triggers for libc-bin ...
ldconfig deferred processing now taking place
Now my apt-get is back to normal. Phew~