Automate the download of VMWorld 2013 online session content

If you are like me and have been looking for an automated way to download the session content from VMWorld 2013 San Francisco well you probably found yourself clicking through the links hoping there was a better way.

Well I hope you have found this blog post because I have done the work for you without breaking the rules.

The perl script at the end of this post uses WWW::Mechanize to connect to VMware’s servers and download the video content and resource links from the online sessions.

All you should need is to pass your VMWorld 2013 username and password to the script via command line and let it run. It will go through the site and identify all the mp3 and pdf content. It will then output a list of commands to download the content URLs including the mp4 videos.

To run it and download the content do something like this:

# script <username> <password> > output.sh
# sh ./output.sh

This will download the URL’s in sequence using curl. If you want to get a little more adventurous look at one of my other blog posts on using GNU parallel to run commands in parallel.

The code was developed on Linux and is provided as-is. Hope it saves you some time.

NOTE: Many of the sessions do not contain pdf or mp3 content. Not sure if this is by design,a bug or the content is yet to be posted. Anyway, the content is around 28GB as of writing.

#!/usr/bin/env perl
#
# Downloads MP4, MP3 and PDF session content from VMWorld 2013
#
# Accepts your VMWorld 2013 username and password
#
# Usage:
#
# download_VMWorld2013  

use WWW::Mechanize;
my $wp = WWW::Mechanize->new();

my $username = shift;
my $password = shift;

if (!$username || !$password)
{
        print "Must supply username and password on command line\n";
        exit 1;
}

$wp->get('https://www.vmworld.com/login.jspa');

$wp->submit_form(
        form_name => 'loginform01',
        fields      => {
                username    => $username,
                password    => $password
        }
);

my(@downloadurls);

$wp->get('http://www.vmworld.com/community/sessions/2013/');

for my $i (@{$wp->find_all_links(url_regex => qr#docs/DOC-[0-9]+#)})
{
        my $url = "http://www.vmworld.com" . $i->[0];
        my $name = $i->[1];
        $name =~ s/[^[:ascii:]]+//g;
        $name =~ s#\s+#_#g;
        $name =~ s#/#_#g;
        $name =~ s#[:?]##g;
        $wp->get($url);
        $wp->content =~ m#<tr><th><p>Session ID:</p></th><td><p>([^<]+)</p></td></tr><tr>#;
        my $sessionid = lc $1;
        my $res = $wp->find_link(url_regex => qr/mylearn\?classID=[0-9]+/);
        next if (!defined $res);
        $res->url =~ m#classID=(\d+)#;
        my $classid = $1;
        $wp->get("http://www.vmworld.com" . $res->url);
        $url = "http://sessions.vmworld.com/mgrCourse/launchCourse.cfm?mL_method=player";
        $wp->get($url);
        for my $l (@{($wp->find_all_links(url_regex => qr#/courseware/[0-9]+/.+[0-9]+\..+#))})
        {
                my $url = $l->[0];
                my $filename = $url;
                $filename =~ m#([^/]+)$#;
                $filename = join("_", $name, $1);
                push(@downloadurls, [$url, $filename]);
        }
        my $url = "http://sessions.vmworld.com/lcms/mL_course/courseware/$classid/$sessionid.mp4";
        my $filename = join('_', $name, "$sessionid.mp4");
        push(@downloadurls, [$url, $filename]);
}

# Output URL's to stdout
#

map { print "curl -s -S -o \"$_->[1]\" \"$_->[0]\"\n" if (!-f $_->[1]); } @downloadurls;

Comments

  1. Sorry I couldn’t find the perl script can you please post it again

Leave a Comment