Finding & Resolving Mysql Connection Leaks

As the title imply; we need to know why Mysql runs out of connections even if it may be configured to handle much more connections. First, most leaks come from the code, check for unclosed connections before anything. If you are using JPA/Hibernate - which is my case - you use a data source either managed by the container as a JNDI resource or a simple component used by the application.

In my application I had a lot of time debugging why mysql runs out of connections and finally I did the following to know the cause:

1- Limit the running features of your app: for example; I ran only the scheduled jobs of the system to maintain a clear log and also monitor the DBMS health slowly. Let your app run for a while till the problem happen or a suitable time has passed.
2- Mysql Admin: see the health of the DBMS from the "mysql admin" tool, if you find the connection usage rising up then we have a connection leak problem for sure.
3- show processlist: in the mysql command line tool, type "show processlist;". Notice the "Command" column of the results in front of you if you find a lot of connections marked as "Sleep" then we are on the right track for the disaster as sooner or later the connection pool will run out of connections.
4- wait_timeout: The "wait_timeout" variable tells Mysql when to consider an unused connection available for termination. for me, that was the problem, a default setting for the DB server not suitable for production or heavily loaded environments. Open the system variables tab and find the "wait_timeout" variable and check its value - default is 28880 seconds -, change this to a much lower value say 15 for example.

What caused the sleeping connections?
A data source may not close the connection after communicating with the DBMS for future use purposes, so, it puts the connection in a sleeping mode and it is the responsibility of the DBMS to terminate it as needed. What we have done in the last step is making sure the DBMS does its part in the play.

Book Review: JasperReports 3.6.0 Development Cookbook

Two years ago I wrote a series of quick start articles here that introduce java reports development through one of the most popular open source library JasperReports. Back then the main motivation for writing such articles was the lack of good learning materials whether blogposts, books or tutorials. I remember struggling with the documentation and digging the web looking for the best practices and avoiding the unknown limitations, it formed a drawback for using the well-known open source library for technical architects who need an assisting quick reference for decision making either to choose JasperReports or another provider according to their instant needs and also it have been a real pain for frontline developers to consider more time and effort navigating the entire web looking for a good learning reference and a professional best practices guide.

Fortunately, that is a past. Recently Bilal Siddiqui has published a new book on the topic JasperReports 3.6.0 Development Cookbook which I’ve been reviewing through the last week and found it could be useful to share my impressions.

The book consists of 9 chapters; could be divided into two parts, the first one contains three chapters covers the basics of report designing using iReport v 3.6.0 through a step-by-step guiding and a dedicated chapter for enhancing the look and feel. The second part addresses the best practices for report developing e.g. using multiple data sources, Crosstabs and graphs; and finally a chapter for wrappers and consuming external swing applications as data source.

Generally, the book is useful for the beginners either designers or developers, it provides the kick start and minimize the learning time and effort to the minimum by guiding you throughout the report designing, development, testing and integrating phases and addressing the most common problems and their solutions. However, the weak points of the book are the same reasons of its strength that it scratches the surface for a better beginner look but lacks of more advanced topics, e.g. row JRXML manipulation for more flexibility, performance best practice specially working with huge datasets and caching sub-reports, also integrating reports in web application could be useful.

SQLServer : How to know the last modified table in your DB

Here is simple T-SQL tip but great to be known. It was part of my task today :). It was making auditing tables for tables created after specific date.

Select * from sys.objects where type_desc= 'USER_TABLE' ORDER BY modify_date desc

type_desc : is object type in SQL and here we wrote 'USER_TABLE' as we need tables created by user.

Do people in non-English-speaking countries code in English?

I was reading a question with the same name on Stack-Exchange Beta site for Non-programmer-related questions (aka Programmers) when I came across this answer posted by a (probably Ukrainian) developer named Igor.

Accompanied by an earlier attempt by a colleague at work to write an enum in Arabic, I tried to write this piece of code:

Strangely enough it compiled and even ran showing the expected output (10)!! Note, the order above of the 2nd & 3rd line is mingled due to RTL issues in Gist. Visual Studio will show it in the right order (or at least, a more readable order for me, copy it & past it in your IDE to know how it looks).

Off course, I’m pretty convinced that this isn’t the best practice you should follow, (check my answer to the same question). Yet, I find this really hilarious! As a matter of fact, I took the claim that it’s bad practice too much for granted to the extent that I’ve never even tired to write such a strange piece of code :D

Does Google Instant really ruin SEO?

I was just reading an article I found on twitter about how Google Instant (the new Google search enhancement to be rolled out to all users soon) will ruin Search Engine Optimization (SEO) which I find kinda wrong.

First: SEO is a way to aid (content providers to provide ways to) internet users to find their way to their content. ie, it’s a mean to an end, not an end! So if Google (or Bing, etc) added new features to enhance the user experience (UX) and personalize search results that might break SEO, SEO should find a way to adapt. Killing innovation is not an option :)

SEO is not about deceiving people to direct them to your content, whether it’s relevant or not!!

Second: & that’s the main point here: As far as I can see, Google Instant is not ruining SEO as Steve claims. Search results are still gonna be shown as the good old days, the only main difference is that you won’t have to click ENTER after typing your search words. You can get instant feedback about where your search results meet your expectations or not. So if content-providers did their SEO job right, their content will show up according to the relevance of their content to the typed keywords.

So, what do you think? correct me if you can see it in any other way..

Update:

I find this tweet by @Bashmohandes more valid though:

Google Instant is cool, but I worry about advertisers, as some of their impression budgets are used while the user didn't show enough intent

FCIH Initiative launch

Finally we are glad to announce the launch of

FCIH Initiative Program

On Next Sunday 25/7/2010 at FCI-Helwan

Join our great event; Join a group of carefully selected sessions highly needed in your practical life.

Event Schedule:

Day 1 [Sunday: 25-7-2010]: Let's get started
10:00	10:30	Orientation
10:30	12:30	Object Oriented Blocks
12:30	01:00	Break
01:00	02:30	Good Code
02:30	03:00	Break
03:00	04:00	English for Business
Day 2 [Monday: 26-7-2010]: Develop your path
10:00	11:30	Web Development A – Z
11:30	12:00	Break
12:00	02:00	Software Quality Engineering
02:00	02:30	Break
02:30	04:00	CV Writing and interviewing skills
Day 3 [Wednesday: 28-7-2010]: IT, World of Endless Choices
10:00	11:00	GIS: Geographic information system
11:00	12:00	CMS: Content Management System
12:00	12:30	Break
02:00	02:30	BI: Business Intelligence
02:30	03:30	RIA: Rich Internet Application
Day 4 [Thursday: 29-7-2010]: Into the business world
10:00	01:00	Agile Software Development
01:00	01:30	Break
01:30	3:00	Understanding Business Analysis
03:00	04:30	Global / Local IT Market
04:30	05:00	Closing

Don't miss the chance…..and don't forget our date "25/7/2010" at FCI-Helwan.

Waiting you there :)

Yours,

FCIH Initiative Team

Adding arabic letters jeem, peh, and veh to keyboard layout in Linux

The letters

چ

پ

ڤ

Do not come on the standard layout of arabic keyboard on linux. It is easy to add them.

You will need to edit /usr/share/X11/xkb/symbols/ara and log-out and login again.

The need to edit only the lines corresponding to they keys you'll attach those letters to. I chose the key corresponding to

ج

ب

ف

(naturally).

The 3 modified lines are:

key { [ Arabic_feh, 0x100fef9 , Arabic_veh ] };

key { [ Arabic_jeem, less , Arabic_tcheh ] };

key { [ Arabic_beh, bracketleft, Arabic_peh ] };

Notice that the new characters are in the third position. The first position is the letter typed when you press the key. The second position is the letter if you were pressing shift, and the third one is the letter if you were pressing the 3rd level selector key. To know which key you have (or to change it to a more convenient one) to go System>Preferences>Keyboard, Mappings tab, click on "Options" then click "Key to choose 3rd leve" then choose whichever key you prefer.

Enjoy your new characters :)

FCIH Initiative.. Postponement announcement

Hello all,

Because of some out-of-hand circumstances, we are sorry to announce that the big event is going to be postponed for few days. We are going to update you with the new dates along with the complete schedule within few days isA.

Sorry for any inconvenience,
FCIH Initiative team.

Creating a simple Open Search Provider/Plug-in

Yesterday, I needed to check/search my delicious bookmarks more than once. I then realized that delicious have its a search plug-in as a part of a tool bar (along with other bookmarking tools, few search tools, though), which -I guess- loads all your bookmarks to browser & make them available for search.

As I’m not so much into too much plug-ins, specially if they load lots of data (in contradiction to Pointy-haired Boss) , not to mention that I won’t need bookmarks when I’m offline, anyway, I decided I should create my search provider.

Creating the provider:

Here’s an example:

<?xml version="1.0" encoding="UTF-8"?>
<OpenSearchDescription xmlns="http://a9.com/-/spec/opensearch/1.1/">
   <ShortName>My Delicious</ShortName>
   <Description>Search Shady’s Delicious Bookmarks</Description>
   <Tags>Delicious Bookmarks Search Favorites</Tags>
   <Image width="16" height="16" type="image/x-icon">http://delicious.com/favicon.ico</Image>
   <Url type="text/html"
        template="http://delicious.com/search?p={searchTerms}&amp;context=userposts|m.shady|&amp;lc=1"/>
</OpenSearchDescription>

Away from the fact that I badly need a WYSIWYG code hi-lighter, let’s check that XML. First the ShortName which appears in your search box, something like this:

Description and Tags are obvious, & I think optional too.

Image is the small icon that appear beside when you select your provider (check the pic above again). You can either supply it as a URL to an image file (usually .ico or .png) or you can replace it with the base 64 string of the image (though I wasted quite sometime to make that work), like:

<Image width="16" height="16">data:image/x-icon;base64,AAA…AAA==</Image>

Finally, URL note that I’m using {SearchTerms} to pass the string typed in the search box. You can send other parameters to your search page. You might want add another URL for suggestions, here I’m only providing the search URL.

Using the provider:

To add the provider to your browser (as far as I’m concerned it works fine for Firefox & IE) you have one of two options:

Adding it as a suggested search provider: This can be done by adding a single line in your page <Head> tag, note that the attribute title in the below tag refers to the name appearing in the list as [Add “title”], check the image below the tag.

<link rel="search"
           type="application/opensearchdescription+xml"
           href="http://localhost/SearchProviders/delicious.xml"
           title="Delicious" />

Using JavaScript to add the provider: The other available option is to add the provider using scripting, here’s an eg of how you can do that:

<a href="#" onclick="window.external.AddSearchProvider('http://localhost/SearchProviders/delicious.xml')">Add delicious Search to your browser</a>

Clicking a link like the one above will pop up a windows asking if you want to add the search provider.

References: http://www.opensearch.org

Changing regional settings for System accounts (including ASPNET) in Windows 7

One of the small issues I faced while switching my development machine to windows 7 was a bug/exception in an application I was working on in parsing date format.

After switching my regional settings to the “expected” format, I still got the same exception. After playing a little with the regional settings, I realized that I was switching only my account date format, not the whole machine's date format. To push those settings to other system accounts (like ASPNET, etc), I did the following steps:

After setting your your account regional settings, switch to “Administrative” tab:

From “Welcome screen and new user accounts”, click “Copy setting”:

Make sure “Welcome screen and system accounts”  check box is checked, press ok, and Voila!

FCIH Initiative -- Students to Business

FCIH Initiative …..
Don't wait the chance, Create It

Thinking about your future? Worried about your career?

Thinking that you are far away from real market life and limited by your academic life?

Want to be professional but don't know where to start?

Interested to know how people work and deliver projects?

Need to gain experience in market? Need to know about Egyptian IT market?

FCIH Initiative is now here for you

Our goal is you, preparing for you, planning for you

Build your own market and Make companies sigh for you

This summer, we'll be together to make your dreams come true

To bridge your academic study to your career

Plan for your career right now

Meet your older colleagues and learn from them

Join us and enjoy:

Trainings – Research – Business Experience – Technical Knowledge – English Club - and more….

FCIH initiative program will start on 17-7, so don't miss the chance.

We're planning to provide:

· Technical sessions

· Training program in 3 tracks (java, .net, and PHP)

· Sessions talking about market, and adding to your personal skills.

· Program for those who are interested in research.

· Soft skills program.

· English club.

Now, don't worry about the future...Together we'll build a better tomorrow

Contacts:

Mail: FCIH.Initiative@fcih.net

Website: http://www.helwan.edu.eg:90/fcih/details.asp?id=161

FaceBook Event: http://www.facebook.com/#!/event.php?eid=128813070490801&ref=mf

Customizing your eclipse download

In this site (http://ondemand.yoxos.com/geteclipse/start) you choose which plugins you want to download you eclipse version with; as opposed to downloading a predefined standard template and customizing it later after you install it. Enjoy :)

Downloading metacafe videos

The following script will download metacafe videos for you:

metacafe-dl.sh:

url=`wget @1 -o /dev/null -O - | grep 'mediaURL' | sed 's/%22/\n/g' | grep -A 6 mediaURL | sed 's/%3A/:/g' | sed 's|%2F|/|g' | sed 's/%5C//g' | sed 's/%25/%/\

g' | tr '\n' '#' | awk '-F#' '{print @3 "?__gda__=" @7}'`

wget --referer=http://s.mcstatic.com/Flash/vp/mc/Portal_3.2.1.1.swf @url -O `echo @url | sed 's%/%_%g'`.mp4

Just call the file like this

./metacafe-dl http://url-of-the-meta-cafe-video

(due to technical error replace @ with a dollar sign instead).

And you will have your file there. Have a nice time ;)

VS 2008 to TFS 2010 (on windows 7, or may be x64 in general)

Probably you already know that Visual studio (Team explorer) 2008 can connect to TFS 2010 if you installed the right Service Pack (there’s one for VS 2005 too) and types the right server name.

 

Yet, a week (or so) ago I switched to windows 7 (64 bit) which lead to a couple of issues with my development environment (feel free to ask me or share your experience with me on that topic, I’m still struggling at that point). One of these issues was that the Visual Studio 2008 SP 1 Forward Compatibility Update for TFS 2010 wouldn’t work for me.

 

After a long search, & several installs, uninstalls, reinstalls, etc.. I came to the following solution that worked for me & (later) for a couple of my team mates at work. Here’s what you might want to do to fix that:

• If you have Office 2007, uninstall it (I’m exactly sure if Office 2007 is one of the causes but it was one of the paths I tried)
• Reinstall Team explorer 2008 (even if you already have it installed)
• Reinstall VS 2008 SP1 (even if you already have it installed)
• Reinstall VSTS 2008 Forward compatibility update for TFS 2010 (even if you already have it installed)

Try to use the same order above, and make sure you reinstall the above stuff even if you already have them.

How to consume REST based-services?

Q. How to consume REST based-services?
Assume we have REST based-service contains method Add which takes two operands op1 and op2 and return the summation.

A. Create any project type (Console, Windows or even Web-based application)
Add using System.Net; using System.IO;
Write some code to consum the service
string parameters;
string response;
// Create the request obj
WebRequest request = WebRequest.Create("serviceURL/Add");
// Set values for the request back
request.Method = "POST"; //REST based-services using Post method
request.ContentType = "application/json"; //tells request the content typs is JSON
parameters = "{\"op1\":2,\"op2\":\"1\"}";
request.ContentLength = parameters.Length;
// Write the request
StreamWriter requestWriter = new StreamWriter(request.GetRequestStream(), System.Text.Encoding.ASCII);
requestWriter.Write(parameters);
requestWriter.Close();
// Do the request to get the response
StreamReader responseReader = new StreamReader(request.GetResponse().GetResponseStream());
response = responseReader.ReadToEnd();
responseReader.Close();
//response value should be 3 if you implement Add method correctly :))

Python Script to download TED Talks translations :))

Well, i admit it, i am a TED lover. I love the passion, and the "new" factor TED brings to me :)

I was watching this presentation for Simon Sink and someone I know downloaded the video but wasn't able to view the translation locally, so I thought I might be able to download the translations for him and convert them to SRT format to be able to display them on any Video Player. and Here is how I did it :)

If you visited the site before, you will find it has a flash control where you can view the video and choose the translation if needed.

First, To be able to investigate how the flash control gets the translation, you have to open a network sniffer which will enable you to view all the packets going to and from your network card. A good packet sniffer is Wireshark, and if you don't know it check those links to know more about how to use it :)
Packet Sniffing using Wireshark Tutorial (Video)
Fifteen Minute Wireshark Tutorial - Wheeler Software


Second, you need to add a filter for "HTTP" requests only, and navigate through any video (like the one above) to view how the flash control communicates with the server :)

When you choose a translation, you will find a request to a URL like this: www.ted.com/talks/subtitles/id/848/lang/eng , where 848 is the Talk ID and eng is the language choosed "English".

So there two questions now, given a URL to the TED Talk

What format are translation subtitles returned?
How do i get the Talk ID?

To answer the first question, just click the link above, and you will find the translation is returned in JSON (Javascript Object Notation) format. So this is good news. We just need to find a good library to handle JSON, and be able to convert it to SRT format to be able to use it on almost any Video Player like VLCMedia Player.

And the answer to the second question is to do a simple "View Source" and search for this number, you will find it in several places through the page's source code, so simple parsing the page HTML should do the job.

Here is the full Script to do the job given the URL on ted.com and the language code which is eng for english and ara for arabic. Will try to provide the rest of the language codes later.

This is a simple command to test the script:

python TEDSubtitles.py "http://www.ted.com/talks/simon_sinek_how_great_leaders_inspire_action.html" "eng"

And here is the full source code, will try to upload it somewhere soon.

Hope it's worth spreading :))

Updates:
You can get the script here .
You need at least Python 2.6 for the json module to be available.

Update on 22/09/2010:

I was informed in the comments by Mohammad that he created a program http://sourceforge.net/projects/tedgrabber/ to download TED Talks and subtitles. He also created another one http://sourceforge.net/projects/timecovergrabbe/ , which allows you to grab Time Magazine covers.This is another example that illustrates the concept and ofcourse it can be applied everywhere. Good luck to Mohammed.

Although it's hosted on sourceforge, the source code is not available. It would be better if it was shared.

Update on 24/04/2011:

A Google Appengine application was created for the same functionality on http://tedsubtitles.appspot.com

You can view the source below, but for better colorized viewing check this link here

import os
import sys
import json
import urllib2

# Format Time from TED Subtitles format to SRT time Format
def formatTime ( time ) :
milliseconds = 0
seconds = ((time / 1000) % 60)
minutes = ((time / 1000) / 60)
hours = (((time / 1000) / 60) / 60)
formatedTime = str ( hours ) + ':' + str (minutes) + ':' + str ( seconds ) + ',' + str ( milliseconds )
return formatedTime

# Convert TED Subtitles to SRT Subtitles
def convertTEDSubtitlesToSRTSubtitles ( jsonString , introDuration ) :
jsonObject = json.loads( jsonString )

srtContent = ''
captionIndex = 1

for caption in jsonObject['captions'] :
startTime = str ( formatTime ( introDuration + caption['startTime'] ) )
endTime = str ( formatTime ( introDuration + caption['startTime'] + caption['duration'] ) )

srtContent += ( str ( captionIndex ) + os.linesep )
srtContent += ( startTime + ' --> ' + endTime + os.linesep )
srtContent += ( caption['content'] + os.linesep )
srtContent += os.linesep

captionIndex = captionIndex + 1
return srtContent

def getTEDSubtitlesByTalkID ( talkId , language ) :
tedSubtitleUrl = 'http://www.ted.com/talks/subtitles/id/' + str(talkId) + '/lang/' + language
req = urllib2.Request(tedSubtitleUrl)
response = urllib2.urlopen(req)
result = response.read()
return ( result )

tedTalkUrl = sys.argv[1]
language = sys.argv[2]

req = urllib2.Request(tedTalkUrl)
response = urllib2.urlopen(req)
result = response.read()

## Get Talk ID value
splits = result.split ( ';ti=' )
talkId = splits[1].split ( '&' )[0]
print talkId

## Get Talk Intro Duration value
splits = result.split ( ';introDuration=' )
talkIntroDuration = splits[1].split ( '&' )[0]
talkIntroDuration = int ( talkIntroDuration )
print talkIntroDuration

jsonString = getTEDSubtitlesByTalkID ( talkId , language )

srtContent = convertTEDSubtitlesToSRTSubtitles ( jsonString , talkIntroDuration )

# Generate SRT file name
splits = tedTalkUrl.split ( '/' )
srtFilename = splits[len ( splits )-1].split ('.')[0]

srtFile = open ( './' + srtFilename + '.srt' , 'w' )
srtFile.write ( srtContent.encode ( "utf-8" ) )
srtFile.close ()

ArabEyes Arabic Technical Dictionary

The ArabEyes project is managing a wiki-based technical dictionary for Arabizing English technical terms.

Not all the terms are translated, and some are still under discussion by the wiki members. Some terms have many proposals and is under a voting process.

The dictionary is used as a reference for translating open source software, like Ubuntu and OpenOffice.

If you want to leave your signature or recommend a better translation for some terms, you are more than welcome to participate on the URL :

http://wiki.arabeyes.org/القاموس_التقني .

GridView with Highlighting and dragging rows

Two days ago, I needed a grid like GridView to be project’s inbox but must have some UI effects like highlighting rows and sorting rows according to column’s name or by drag and drop rows by mouse pointer.
Of course the first thing came in my mind is using JQuery with the gridview as it runs by a little lines of code and give the client a big influence and satisfaction.

I am already started with trying some plug-ins for Drag and Drop but I found a big issue with GridView itself as it’s render the header and footer as ordinary row (TR Tag) NOT (thead tag) and (tfoot tag) so any plug-in can’t differentiate between any row and the header (the section of columns name) or footer (the section of paging ).

So I google for solution and get a part of it by Suprotim Agarwal then I modified it to fit my needs and some changes in the Table Drag and Drop JQuery . So take care to use the attached (modified) Dragging plug-in as I modified it and use the start up script.

The Grid before fix (as you can see, you can drag any row even the header or footer):

After fixing:
Download the complete Example.