Easy way how to transfer really big files

2008/04/29

What does really big mean ? In my case few hundred megabytes, up to 1 GB. Clearly, too much for email attachments – most providers caps them at around 5-15 MB.

Since we started doing screencasts, I need quite often to transfer  work in progress – screen recordings, rendered MOV files between me and my co-host of the screencasts. I tried to upload the file to the company servers using VPN, but the speed of the secure uploads is nowhere close to being practical.

What works much better is free service called TransferBigFiles.com. There is no registration required, you just enter the recipient email and start upload. After uploading, the system sends you an email with link and the files stays up for few days, then is deleted. More than enough for the other party to download. With self-destruction, it is also less worries for you to  remember clean-up.

The site also offers client for Windows, which I did not try (for obvious reasons) and shows ads selling software. These ads made me (strangely enough) more comfortable, as the intention of the creators is clear: attract traffic by free service and monetize on software sales / ads.

The other possible explanation is of course collecting of email adresses for not-so-noble purposes. I have no knowledge of this being the fact – since about a week and half of using the service, there was no spike in amount of spam. No spike meaning nothing beyond usual 300+ offers of 10 dollar Rolex, body enlargements, natural weight loss programs and help requests to transfer money from Nigeria and similars …

All my emails are consolidated on Gmail, which does terrific job of weeding out 99.99% of these – so I do not really care so much if one more spammer gets my email. But if you are really worried, you can always get the 10 minute email, use it for receiving notifications and then email the link manually. The down side is that you must keep the 10 minute email alive until upload finishes – this may take few refreshes.


Quick online diagramming tool

2008/04/28

I am using OmniGraffle for most of my diagrams. OmniGraffle is an OS-X platform replacement for Visio – only easier to use and better looking (in my opinion). On rare occasions I do not work on my notebook, it is quite useful to have free alternative that allows to create a diagram that does not look too awful, without going through the hoops of downloading and installing software. The bubl.us is exactly that tool.

Using bubl.us you can very quickly create visual representation of simple structure with basic relations (parent/child) and links. You can define the text and links between elements and save the result as XML, HTML or JPG/PNG image. The result can look e.g. like this:

No a bad result, considering that its creation took less than 60 seconds.

If you need more control – like different shapes and stencils, look at Gliffy, which is much closer to actual desktop diagramming tool. It is free tool with paid enhanced version, available here.


A false “hang” of Finder

2008/04/24

Yesterday, I have noticed interesting thing related to Finder and “beach ball of the death”. Suddenly, for no obvious reason, the top menu became unavailable and cursor changed to beachball every time I moved it there. The Bluetooth mouse stopped responding. Finder stopped responding. All symptoms as if Finder would hanging …

The cause of the issue was crash of unrelated application (iShowU, if you are curious), which brought up the crash report window. For some reason – maybe because it happened while changing spaces – the window was not visible. After closing the window, everything came back to life and even mouse reconnected (after disabling and re-enabling Bluetooth).

So next time it looks like an application did hang, make first sure that no windows expecting input are not open and hidden. Do not rely upon jumping dock icon – with bad luck and Spaces enabled, it is easy to switch the notification off.


Agile development team member mantra

2008/04/21

Introducing code review and pair programming has often an impact on relationships in the team – especially when a critique needs to be delivered. It takes great degree of experience on both sides to communicate defects in the code for both the reviewer as well as for the code author to get something useful out of it.

I found this at http://www.finetix.com/blogs/finetixblogs.html which can be helpful (page is 404-ing now, but I luckily saved it in Google Docs way back) :

Everyone, including me, repeat after me:

  • I am not my code. A criticism of my code doesn’t have to be a personal attack on me.
  • I am not the tools that I use.
    A criticism of my tools shouldn’t be construed as a personal attack on me.
  • I am not the programming language that I use

I will try to remember it (and read it to all participants) next time we get into the passionate discussion about whether inheritance or delegation is the correct approach to implement feature X 🙂


Weird kind of spam that passes through Akismet

2008/04/20

Spam has always been a big problem for all blogs. Fortunately, worpress.com provides wonderfull Akismet spam filter which is works amazingly well. I do have switched on moderation for all comments, to make sure no spam gets through, but without Akismet it would be impossible to manage it because of the spam volume. Since I started this blog (August 2006), it caught over 17’000 (yes, seventeen thousands) spam posts. In other words, about 30 spam comments a day or almost 60 spam comments per published post.

In past few days I have notices new type of spam comments that pass through Akismet. What is really confusing, is that there is no obvious message in the text – no usual wonderdrugs, ten dollar Rolex or enlargement of body parts offers. Some of the links even point to quite legitimate, non commercial sites. I did not spend any time on it to find out a pattern yet. Some kind of campaign ? Few sites were related to stop animal cruelty – a very noble and worthy goal to support, but certainly not with methods like this …

Anybody else having similar experience ?

Here is what it looks like (I am posting it as image to avoid promoting the spammer):


Heroes Happened … here

2008/04/17

Last week I attended one day event organized by Microsoft named Heroes Happens Here. It is series of introduction targeting new products: Windows Server 2008 (not out yet) SQL Server 2008 (not out yet) and Visual Studio 2008 (out and well).

I do not do much Microsoft development these days – my world is currently rather unevenly divided between Java (about 70 %), Web-ish technologies (XHTML, CSS, JavaScript, some Ruby) and OS-X (Objective C) which I am trying to squeeze in and get finally started. But Microsoft is such important platform that one must pay attention what is going on.

The event was well prepared and well executed. After all, Microsoft is doing this for quite some time and experience shows. For the first time, there were more people attending that the space in Cineplex allowed – so there were overflow rooms with huge projection screen for the slides/demos and smaller screen showing the video-feed of the presenter. We ended up in overflow room. One very important comment to whoever was in charge of camera control: please do NOT zoom in on presenter and then try to follow him as he walks. Seeing how the space floats in front of your eyes is quite nauseating experience. Even if the speaker is good looking guy (as I overheard from some female attendants), most of the audience does not really care about detailed snapshot of his face with moving background, but about slides and his words. Think about the people in the other room – if they are fine with seeing the presenter from large distance, so are we. No need to play movie director and add “dynamism” to the stream.

The above is probably only objection to the format and execution of the event – all the rest was fine. The speakers were great (thanks, Christian) and the script was surprisingly well acted on. You guys have more talent than just technology evangelism.

From the products introduced I was mostly interested in VS 2008 and SQL Server 2008. I really liked the enhancements done in 2008. Nothing revolutionary, but nice features that make coding experience and developer productivity so much better. I wish the VS 2005 had this features :-). However – and there is always however – I have seen nothing that Eclipse/IntelliJIDEA/Netbeans would not offer for quite some time. In many areas the VS is still playing catch-up (like refactoring), in others will probably never quite catch up (extensibility – being a closed source product not an open source platform). But now, it is closer than ever. Where VS will probably beat Eclipse is speed of execution and look && feel (being a native App). I am looking forward to find out. Great work, Microsoft. It is important that there is more than one platform/toolset to keep the innovation going.

I am also quite excited about enhancements in SQL server 2008, specially in BI area and reporting. I will wait for the release to find out though – no cycles right now. I just hope the features will not disappear the same way as the WinFS did from Vista.

The Windows Server 2008 is least important from my personal perspective. With more and more services moving into the cloud and progress in virtualization (“software appliances”), there are IMHO better alternatives to host your VM’s on – as well as better OS-es to be used as guest OS. I would probably prefer to use some derivate of Unix for both (OS-X or Linux) or true VM Host (VMWare) and Unix combination. Buying Windows license to run appliance starts to be prohibitively expensive, if you want to build many of them.

One final remark to the general impression from this and past Microsoft tech events: I realize that tone of the announcement and presentation must be upbeat and optimistic, but there is no reason make it sound as if Microsoft just invented it and if that was the greatest innovation since the TCP/IP protocol. Even more if it is basically a bug fix feature: “ .. and now, the graphic designer of Web page actually works and does not rearrange and screw up your markup every time you switch into visual mode and back as in VS 20xx“.

I was very happy to observe that this “spirit” was tuned down a lot. What I have seen on this year HHH is more sober, mature Microsoft that has really competitive product and does not need to sink down to the FUD as often as before – they even mentioned “works with Firefox” ;-), but still did not dare to say the G word.

On the other hand – what kind of “innovation” is to say that you do not really need to run GUI on Web server ? Did the last 30 years not happen or what ? Just because until PowerShell came out, the Windows “shell” was such a lame and hardly usable tool for doing anything non-trivial (compared to e.g. BASH), it does not make making GUI mandatory a good idea. And allowing GUI-less server should be more appropriately presented as “OK, we were wrong but we fixed it” rather than “Look what cool feature we have just added”.

As Henry Spencer once said:
“Those who do not understand Unix are condemned to reinvent it. Poorly”


The dark screen club

2008/04/16

It happened few days ago for the first time: after opening the lid on my MacBook Pro, the machine woke up, but the screen stayed dark. Whatever I did, I just could see cursor and nothing underneath. I had to resort to unthinkable – reboot 😦

Some research showed this is known problem – which is from certain point of view good, because many people found hopefully many workarounds. The more information can be found here, here and here. Very helpfull was also this page.

There are speculations about some software that may be related to the problem. As matter of fact, I have installed after some time of not using it one of the suspects: iStat Pro dashboard widget.

What I have done to minimize chances of reoccurence:

– removed the iStat Pro and Polar Clock screen saver (which was also new install)
– switched off the setting ‘Automatically reduce the brightness of the display before sleep’
– started to switch off the bluetooth mouse before closing the lid (so that BT is off when sleeping)

I have also switched off the “safe sleep” mode which is not of much use as my notebook never stays unopened for longer that 12 hours :-).

So far, everything works OK. Let’s keep the fingers’ crossed …


Quote of the day

2008/04/09

Program managers want an infinite number of features in zero time, testers and service operations staff want zero features over infinite time, and developers just want to be left alone to code cool stuff.

From I. M. Wright’s “Hard Code”
by Eric Brechner


Trying out Groovy with Oracle database

2008/04/07

There is a saying that necessity is the mother of invention. Such necessity happened last week and forced me to try out the Groovy language.

The trigger was need for creating good data set for testing changes in a full text search. I had to locate few hundred of obsolete technical documents to be used as test data, dump the metadata as well as the BLOB data to disc, create INSERT sql scripts and loader that would from Ant pre-load the database with the test set and insert the BLOB’s. It is fairly simple task, the issue were boundary conditions:

Database in question is Oracle. I have no OCI8 installed on my MBP and refuse to install Oracle on OS-X – it was little fun getting it up and running on Linux. Usually I run Oracle in Windows or Linux VM and with lack of OCI8, only option how to communicate with it is Level 4 JDBC driver.

Because of missing OCI8, Ruby was out of question (the gems for Oracle access needs OCI). So was PL/SQL, for the reason of administator priviledges. I am not exactly a PL/SQL programmer, but know enough to write stored proc that loads or unloads BLOB, assumed that UTL_FILE and DBMS_LOB packages are available and accessible. Which was not the case.

Before falling to default and writing it all in Java (which would work, but take certainly longer than I was willing to invest), I decided try out this Groovy thing :-). My exposure to the language was minimal – I never programmed in it (not counting 2 hours spent browsing Ruby in Action), but I hoped that similarity with Java and almighty Google will help me out …

Connecting to database in Groovy is simple and elegant:


import groovy.sql.Sql

db = Sql.newInstance(
            'jdbc:oracle:thin:@myoraclehost:1521:XE',
            'USER', 'PWD', 'oracle.jdbc.driver.OracleDriver')

      def sql = """
            select d1.document_id, d1.version, d1.filename from document d1
            where d1.document_id in ...

           .... REST OF SQL DELETED ....
        """
        list = db.rows(sql)

The result is hash-like structure that can be used to drive both generating the DELETE and INSERT SQL statements into text file as well as retrieving the BLOB’s and saving them to HDD:


    def writeBlobToFile(db, docid, version, filename, reallyWrite=false) {
        def sqlString = "SELECT d.DOCUMENT_BODY FROM document d where d.document_id = '$docid' and version = $version"
        def row = db.firstRow(sqlString)
        def blob = (oracle.sql.BLOB)row[0]
        def byte_stream = blob.getBinaryStream()
        if( byte_stream == null ) {  println "Error for ${docid} : ${version}"  }

        int total = blob.length();
       
    // Write to a file
       if (reallyWrite) {
            byte[] byte_array = new byte[total]
            int bytes_read = byte_stream.read(byte_array)
           def fullname =     "/Users/miro/tmp/BLOBS/$filename"
           def fos= new FileOutputStream(fullname)
           fos.write(byte_array);
           fos.close()       
       }      
        println "Document $docid:$version, file: $filename, size $total"
        return total
    }

Loading the documents and metadata from disk to database is done by Ant script

<path id="groovy.classpath">
  <fileset dir="${lib.dir}">
    <include name="groovy-all-1.5.4.jar"/>
  </fileset>
</path>

<taskdef name="groovy"  classname="org.codehaus.groovy.ant.Groovy">
    <classpath>
        <path refid="groovy.classpath" />
        <path refid="database.classpath" />
        
    </classpath>
</taskdef>

<target name="reload-document-fixtures" description="Prepare the database for testing">
    <sql
        driver="oracle.jdbc.driver.OracleDriver"
        url="${database.url}"
        userid="${database.user}"
        password="${database.password}"
        print="yes"
        output="database_load-${DSTAMP}.txt"
        autocommit="true"
        onerror="continue"
        >
        <classpath refid="database.classpath"/>
        <transaction src="src/sql/Fixture/DELETE_DOCUMENTS.sql" />
        <transaction src="src/sql/Fixture/INSERT_DOCUMENTS.sql" />

    </sql>
    
    <groovy src="src/groovy/com/company/app/utils/LoadDocuments.groovy">
        <arg value="src/sql/Fixture/BLOBS"/>
        <arg value="src/sql/Fixture/METADATA.txt"/>
    </groovy>
</target>    

The Groovy task must be defined before used (with proper classpath). The groovy
task shows passing arguments and executing script. The most important part of
the script (loading binary file and inserting the BLOB) is here:


// each line in METADATA.TXT looks like this:
reg =  /document_id: \'(\w+)\', version: (\d+), filename: \'(.+)\', size: (\d+)/

db = Sql.newInstance(
        'jdbc:oracle:thin:@myoraclehost:1521:XE',
        'USER', 'PWD', 'oracle.jdbc.driver.OracleDriver')
counter = 0
new File(args[1]).eachLine { line ->
    line.eachMatch(reg) { match ->
        try {
            str = "${args[0]}/${match[3]}"
            println "Opening file: ${str}"
            FileInputStream fis = new FileInputStream(str)
            int size = fis.available()
            byte[] data = new byte[size]
            fis.read(data)
        
            // get the BLOB
            row = db.firstRow("select document_body from document where document_id = ? and version = ? for update",
                    [match[1], match[2]])
            my_blob = (oracle.sql.BLOB)row[0]
            if( my_blob == null ) println "my_blob is null!"
            outstream = my_blob.getBinaryOutputStream();
            outstream.write(data);
            outstream.close()
            fis.close()    
            counter = counter+1
        } catch (Exception e) {
            print "Exception: ${e}\n"
    }
}
println "Processed files: ${counter}"

The loader is controlled by the file METADATA.TXT that contains information linking document metadata in database (inserted by SQL statement) with file on disk. This indirect way allows easily “inject” document body with required search phrases, reload database and run tests.

The whole experience was quite pleasant and considering how little I knew about Groovy, it took very little time to create something useful. I wish I could have done some thing in Ruby (which I still like better) – but I was amazed how powerful the combination of Groovy + Ant can be.

The most valuable feature is no need to add anything really new or exotic to Java environment – all you need is Jar file and Eclipse plugin, no new/different library packaging scheme – Jars are still jars, not gems.

The only minor hickup and weird thing I found was passing arguments between Ant and Groovy script. It looks like that the groovy script should be a script and not use classes + static main method (which would be natural instinct for Java developer). Thanks to Christopher Judd for the hint.


Couple of new podcasts

2008/04/02

With the walking season almost upon us (so far only walks by streets, for the Ottawa River Pathway one still needs a snowshoes), it is time to replenish the stock of listening material. Here are some of the about 10 new podcasts that I tried out recently and decided to keep:

1) Pragmatic Programmers podcast. That was a pleasant find (thanks Milos). The topics covered are very wide – as are topics covered by the books. Average lengths is about half hour and the podcasts appear about 1-2 per month. I really liked the interviews with Michael Nygard and Johanna Rothman. The disadvantage of these podcasts is fairly miserable audio quality – the discussions are done by phone or something is wrong with the recording. Maybe I am spoiled by great professional audio of most of the TWIT network podcasts – but after Leo and the gang, this audio is really hard on ears.

2) Fortunately, the second podcast – WebDevRadio.com does not suffer from bad audio quality. Topics covered is as name says and as added bonus, web site provides links to episodes.

3) Late Night Cocoa is podcast focusing on Mac development with Objective C and Cocoa. There is also screencast available.

4) The series of podcasts on InformIT: so far I tried out OnOpenSource and found it quite interesting. Technically, it is a video podcast. But in reality, you will not loose much if you just listen to it, because the video adds little to information value: two guys sitting and talking. While it is interesting to see how the people from Open Source project look like, it is not really necessary to watch the full interview – it is what they say that counts. Main disadvantage of this podcast on iPod Touch is that you have to let the screen switched on which eats battery much faster. Not ideal while walking.

If you happen to be able to understand Czech, the group of guys in Prague is publishing Java centric podcast named Java Portal. Audio quality varies, content is surprisingly good. Prague is large spot on world map of software development centers that contributed a lot to projects like Netbeans or IntelliJIDEA.

For me I found that it is very refreshing to listen to podcasts in other languages than English. Without satellite television available carrying TV stations from the old country I have lost contact with everyday living Czech language. If you feel the same, give Java Portal a try. I am now in a search for good tech podcast in German. Any recommendations ?