Over two years ago EMC became very interested in this space.

Size isn’t the most important point about Big Data. There is an enornmous amount of data and growth rates are high.

We already have more than we can analyse. Most of it isn’t in a form we can readily analyse – unstructured.

In retail and financial services there’s a huge amount of unstructured data (purchasing habits etc) which would be useful to analyse.

The velocity of the data is another issue – coming into the data centre continually, not periodically analysed as we have in the past.

How can services benefit from Big Data? Let’s look at healthcare. Access to patient records, precision on individual patient genome data, combine that with very large patient studies. We’ll see continual movement on this in next decade. Will be able to see how a patient might respond to treatment based on data.

In retail banking, if we’d been able to make more of big data we might have been able to avoid the last big crash.

How do we classify and segment Big Data?

• Rich content stores (media, content creation, GIS, satellite delivery)
• Generated from worflow (manufacturing, simulation, electronic design) – typically in mechanical spaces
• New intellectual property – particularly pharmaceutical data, custom drug development
• Companies, public sector and utilities mining data for business advantage – in the public sector, what’s exciting are very large datasets about populations, and how that can improve services in public sector. Can achieve great savings in public services by applying data science to the datasets we have available.
• Consumer data mining – high volume and potentially high volume. This has been lead by the big internet companies. They’ve already developed a lot of the technologies. People are saying I don’t look like Google, but I can use some of their technology to improve my services.

Big Data is file and unstructured data. Only a tiny fraction of data is structured – this is what business is based on.

Why is Big Data appearing now?

Anytime a new trend, worth asking is there a particular reason for this coming now?

• Cheaper to manage, store and analyse (disks cheaper, scalable, technologies for analysis)
• Increasing access to data via the cloud
• Innovative software and analysis tools

There’s a lack of specialised data scientists at the moment – that’s holding us back a bit, but the tools available now are helping.

• Proliferation of sensors and mobiles devices – lots of devices are continually providing data back to cloud

Gartner’s 3 V’s of Big Data

Volume, Velocity and Variety.

Google has moved to analyse data (searches, downloads etc) in real time – that’s the direction we’ll all go in.

The “internet of things” – a lot more devices, applications etc are all sending, receiving and storing data.

More use cases than simply marketing.

To speak about the velocity of data – shows GoGlobe image In 60 seconds.

This is data that’s being used – not just sitting in repositories. Google for example looking at mails in real time – who are you, who are your friends, what are you interested in?

Companies want to store big data. Google originally thought of as search engine. Entire corporate value of Google is based on the information they keep.

Facebook and Twitter store lots of structured data. Most important thing is the social graph – and that information is the most valuable information anyone can have (from an advertiser’s point of view).

Amazon – your every purchase, forever.

Carriers not being left out – mobile phone services know where you’re located (don’t need GPS for that), using that they can advertise to you.

It takes time to build all this. The race is on.

Big Data leads to the optimised organisation. If you want an organisation that moves according to the scientific results of the data, it needs pretty big organisational change.

Many companies have a head start and have already starting investing in people and change.

Takes a long time to build data warehouses, analytics tools.

Rob used to work at Amazon. Hugely efficient backend – purchase information used to adjust the supply chain.

They recognise trends amongst customers much more quickly than anyone else.

Not just Amazon – other retailers are looking to develop these capabilities.

Good data is hard to get – so how do you make decisions? A lot of decisions are actually not the right ones.

Lots of companies do the same things for too long and don’t change fast enough to adjust to the market.

They fail to detect shits in consumer demand.

Analysing consumer data – their own or other available sources – could prevent this.

The internet has made customers more segmented and causes customer choice to change.

Moving to a data-driven model – need to make a science out of data. Moving away from “gut feel”.

Big Data based decisions will lead to:

• more transparent and usable information
• more accurate information
• better services

What’s holding us back?

It’s not ICT or the quantity of data – it’s not that large yet (not necessarily in petabytes).

It’s also not the value – plenty of evidence of ROI. The model pays for itself.

It’s organisation change – adapting to the new model – and talent acquisition.

This is where many companies – including EMC – are working, to consult on making this organisation change. A services model around big data rather than a technology one.

Only you know what data you have in your organisation – the question is how you decide to use it.

Big Data has a different fundamental storage model to the traditional methods of storage. Mainly related to scalability, availability, performance, management and analysis.

Most clustered computing and virtualisation models – often commoditised services – should be able to scale out storage architectures.

Greenplum and Isilon – enterprise Hadoop. EMC Greenplum purpose built for Big Data. Massively Parallel Processing – scalable. EMC are releasing tools to support this too.

Rob finishes off with a diagram – looking at the Big Data stack, from storage to analytics to application and where his company’s offerings fit in.

He hopes today will be different

What do we mean by Big Data? We’ve taken as the starting point Ed Dumbill’s piece from O’Reilly Radar. Andy thinks it’s one of the best definitions.

Bid Data refers not only to the data itself, but how you analyse that data, anlaytics and so on. There tends to be a focus on ‘data’ in education and government. There’s potential confusion with the Open Data agenda, particularly in government space.

Other way of thinking about Big Data are the tools that are being used – Hadoop, MongoDB – but it’s important to not focus solely on technology but big picture.

Emerging role of big data scientist – an interesting trend.

Why is Eduserv interested? We’ve been working on cloud services, particularly our Education Cloud and we’re interested in talking to people who want to use that cloud service for their big data services and applications. We also think it’s a great topic and always try to focus our symposia on topics that are of interest to the communities we serve.

There were nine parallel tracks over 83 sessions over the two days, with 15 deep-dive training seminars. Two keynote talks from Quentin Clarke (Corporate Vice-President Microsoft), David Flynn (CEO Fusion-IO) and Conor Cunningham (Principal Software Architect for SQL Server) gave attendees an overview of the new features in SQL Server 2012.

The three main issues addressed by SQL Server 2012 are data explosion, consumerization of IT, and Public, Private Cloud. I’ve highlighted some of the technical features discussed below:

Core

• A Mission Critical new feature called AlwaysOn that allows integrated high availability and disaster recovery. It enables idle hardware to be utilised with some data features being offloaded to the active secondary servers.
• Simplified deployment and management of high availability.
• Support for direct attached storage.

Business Intelligence

• Enhancements for data visualization with self-service analytics tools like Power View and SharePoint collaboration.
• Data consistency achieved with a new tool called Data Quality Services and enhancements to Master Data Services.
• Column Store Index ‘xVelocity’, an in-memory technology, significantly improves performance for data warehousing and business intelligence.

Development

• SQL Server Data Tools (SSDT) includes a new development environment for database developers to work on the SQL Server platform for both on and off premise development from SQL Server 2005 upwards.
• Application testing can utilize Distributed Replay to help with tuning and access the impact of future upgrades.

Cloud

• On Premise and Cloud architectures brought together with bi-directional data sync. This allows the sharing of data using replicas in the cloud which span multiple environments.
• Big data capabilities based on Apache Hadoop are incorporated to process large volumes of unstructured data.

With so many interesting sessions scheduled, it was difficult to choose what to attend. I went to various talks on Big Data and Cloud services within SQL Server which were most enjoyable, sharing a vast array of technical information with the attendees.

Victoria Holt is a Senior Database Administrator at Eduserv

The Big Data and Data Warehousing session by Thomas Kejser looked at the paradigm shift and where Hadoop fits with scalability – you have to love the cute elephant! The architectural patterns and practices SQL Azure session by Ewan Fairweather highlighted various items to consider when dealing with shared database infrastructure. The SQL Server 2012 TAP lessons learned showed an interesting case study on how a migration of a mission critical application to SQL Server 2012 was carried out. I gained some useful insight on things to consider when upgrading to SQL Server 2012.

My absolute favourite session of this conference was the DW Design with the Product Team with Conor Cunningham and Thomas Kejser.  This talk addressed architectural patterns for designs in data warehouses, comparing the mathematical and customer requirements. I will be watching videos of many of the sessions I wasn’t able to attend once they become available on the SQLBits website.

All in all the conference was amazing and it was privilege to help out at the event.

You can follow Victoria on Twitter @victoria_holt

Here’s the blurb for our event:

Data volumes have been growing exponentially for a long while – so what’s new now? Is Big Data ^[1] just the latest hype from vendors chasing big contracts? Or does it indeed present wholly new challenges and critical new opportunities, and if so what are they?

The 2012 Symposium will investigate Big Data, uncovering what makes it different from what has gone before and considering the strategic issues it brings with it: both how to use it effectively and how to manage it.  It will look at what Big Data will mean across research, learning, and operations in HE, and at its implications in government, health, and the commercial sector, where large-scale data is driving the development of a whole new set of tools and techniques.

Through presentations and debate delegates will develop their understanding of both the likely demands and the potential benefits of data volumes that are growing disruptively fast in their organisation.

[1] Big Data is “data that exceeds the processing capacity of conventional database systems. The data is too big, moves too fast, or doesn’t fit the strictures of your database architectures. To gain value from this data, you must choose an alternative way to process it.”  What is big data?  Edd Dumbill, O’Reilly Radar, Jan 2012

As usual, the event is free to attend and will be followed by a drinks reception.

You’ll note that we refer to Edd Dumbill’s What is big data? article in order to define what we mean by big data and I recommend reading this by way of an introduction for the day. The Wikipedia page for Big data provides a good level of background and some links for further reading. Finally, O’Reilly’s follow-up publication, Planning for Big Data - A CIO’s Handbook to the Changing Data Landscape is also worth a look (and is free to download as an e-book).

You’ll also note that the defining characteristics of big data include not just ‘size’ (though that is certainly an important dimension) but also ‘rate of creation and/or change’, and ‘structural coherence’. These are typically known as the three Vs – “volume (amount of data), velocity (speed of data in/out), and variety (range of data types, sources)“. In looking around for speakers, my impression is that there is a strong emphasis on the first of these in people’s general understanding about what big data means (which is not surprising given the name) and that in the government sector in particular there is potential confusion between ‘big data’ and ‘open data’ and/or ‘linked data’ which I think it would be helpful to unpick a little – big data might be both ‘open’ and ‘linked’ but isn’t necessarily so.

So, what do we hope to get out of the day? As usual, it’s primarily a ‘bringing people up to speed’ type of event. The focus will be on our charitable beneficiaries, i.e. organisations working in the area of ‘public good’ – education, government, health and the charity sector – though I suspect that the audience will be mainly from the first of these. The intention is for people to leave with a better understand of why big data might be important to them and what impact it might have in both strategic and practical terms on the kinds of activities they undertake.

We have a range of speakers, providing perspectives from inside and outside of those sectors, both hands-on and more theoretical – this is one of the things we always try and do at our sympoisia. Our sessions include keynotes by Anthony D. Joseph (Chancellor’s Associate Professor in Computer Science at University of California, Berkeley) and Rob Anderson (CTO EMEA, Isilon Storage Division at EMC) as well as talks by Professor Anthony J Brookes (Department of Genetics at the University of Leicester), Dr. Guy Coates (Informatics Systems Group at The Wellcome Trust Sanger Institute) and Max Wind-Cowie (Head of the Progressive Conservatism Project, Demos – author of The Data Dividend).

By the way… we still have a couple of speaking slots available and are particularly interested in getting a couple of short talks from people with practical experience of working with big data, either using Hadoop or something else. If you are interested in speaking for 15 minutes or so (or if you know of someone who might be) please get in touch. Thanks. Another area that I was hoping to find a speaker to talk about, but haven’t been able to so far, is someone who is looking at the potential impact of big data on learning analytics, either at the level of a single institution or, more likely, at a national level. Again, if this is something you are aware of, please get in touch. Crowd-sourced speakers FTW!

All in all, I’m confident that this will be an interesting and informative day and a good follow-up to last year’s symposium on the cloud – I look forward to seeing you there.

Andy Powell is Research Programme Director at Eduserv. You can follow him on Twitter @andypowe11

This post first appeared on eFoundations.

I was once asked to talk about why identity initiatives fail. In preparation for my chat, I interviewed users and designers of various schemes that had succeeded and failed. The conclusions of those discussions were obvious but I am surprised how passionate people still get about identity initiatives that exhibit similar tendencies.

The almost universally expressed view was that it wasn’t about identity at all: an identity credential isn’t an end product or service but something that you require or acquire to get or do something.  If what you get or do with an identity credential just isn’t attractive for whatever reason, or is available in a much easier way, then it is very unlikely that the identity product or service will be a “starter for 10”.

The second strand was equally obvious: if the identity credential – however well designed or privacy protecting – is more expensive or onerous to get/use than other ways of doing things, then forget it.  How many wallets have tried to replace simply typing in your credit card number over an SSL connection over the last decade? I have lost count.  Keep it simple for consumers and relying parties.

There were some simple questions that came up again and again in assessing the value provided by schemes – listed here by order of frequency:

Consumers

• What are the services I can get using the identity credential? Are they compelling?
• Is the credential expensive or difficult to get or use?
• Can I get the service without the credential?

Relying Parties

• What’s the business service being implemented? Does it really provide value over how I do business today? Can I do business with more people, more easily?
• What will the experience be like for my customers? Is it likely to put them off?
• How complicated is it to install?
• What is the risk if I rely on the token – will I get stung for failures, fraud etc. or is there greater protection than I have today?
• What makes it better than the status quo – do more business more cheaply? (SSL, Username, Password)

However wonderfully designed your ID service, it is the services that it enables that drive its success or failure – and only if it makes those services more easy to obtain than otherwise or obtainable on better terms.

 Jonathan Sowler is Government Client Director at Eduserv. Follow Jonathan on Twitter @jonsowler

Todd Parker – jQuery Mobile Keynote

Todd Parker talked mostly about one of the hottest parts of jQuery from last year: jQuery Mobile, a framework built to put mobile devices first.

Todd talked merrily about building the framework on principles of progressive enhancement and responsive design. I think most people walked away from this talk excited about the stuff we’re able to do now and I’m looking forward to a project where we can have a play with it.

Dion Almaer and Ben Galbraith – Web vs. Apps

A fascinating talk about the future of native mobile applications. As modern browsers get better and better developers can do pretty much anything they like, so will native applications eventually die out? Dion and Ben felt that they wouldn’t for quite a while but we’re starting to see hybrids of native applications and web applications (for example the Facebook application) where developers have more control over forcing users to upgrade and writing core components just once.

Haymo Meran

For a company that works with CMSs a lot it’s fascinating for us to see things we could leverage to make our users’ lives easier. This was a talk that did just that. Haymo talked about Aloha Editor, a WYSIWYG editor that has the potential to extend the TinyMCE. It works quite a lot like Sitecore page editor but looks even more advanced.

Paul Irish

Paul works on Google Chrome development. He was  a great speaker and his talk featured loads of geeky goodness, covering a whole range of recent technologies we should all be playing with, such as CoffeeScript, LESS and HAML – it’s way too much to cover here but you can get grab his slides here.

I had to run off for babysitting duties before the evening do kicked off but even missing some free drinks does not take away from what an interesting, inspiring and well-organised event this was. Many thanks to the organisers White October and I look forward to next year’s event.

Before joining Eduserv in 2010, Steve developed internet banking applications for the financial sector. He enjoys Eduserv’s relaxed work culture, interesting projects and the wide variety of technologies that the web development team uses. His recent projects include the JISC, NHS Direct and Victim Support websites. Outside work Steve can usually be found cycling, pottering down to the coast or just happily sitting around enjoying tea and biscuits.

It is important to consider the total cost of ownership and the cost with open source comes with the time and expertise required to implement and maintain it.

What happens when the person or team that installed the solution moves on?

To consider this properly, we need to look at the skills that are needed in your IT team to install or maintain open source options and whether these particular skills are widespread in your IT team because of the other things they do; i.e. is support built into your team or do you need to maintain a specialist? If you’re a large University then it’s probably already in the team; if you’re a small college then it probably isn’t.

Next we need to consider how often the relevant staff move on and need replacing. In a large team where several have the required skills you can take your time, but in a small team or where there’s a single specialist, chances are you’ll need to recruit a replacement in a hurry and if you want someone good, fast, you’ll have to pay more for them.

On the surface that seems simple enough to monetise: recruitment costs divided by reasonable churn. The number’s higher than you probably expect – ask your personnel manager. That simplicity depends on two assumptions. One, that the documentation of the installation and especially any customisation that’s been done is not only complete, but makes sense to the new person. This is by no means guaranteed even through one generation, but throw in a couple and you can be looking at a complete reinstall just so the IT guys can understand it. Two, it doesn’t break down while you’re between the people who understand it.

OpenAthens LA has been designed from the outset to be as hands-off as possible – there are no XML files to edit, you don’t need to know Linux, Java or Apache and you don’t even need to build any servers since it’s available as a pair of pre-built virtual machines. All this means that almost any member of even the smallest team will have the required skills, and there’s a full time service desk on the end of the phone if they need help.

A simple choice

If you have a large IT team with lots of skills and awareness of open source options, and if you have a low staff turnover, then they could be very good value for you.

If your IT team is small, or busy, or has a high staff turnover, then OpenAthens LA is definitely where to start. Find out more about OpenAthens LA.

We spend a lot of time thinking about the underlying technology that supports identity and access management but it’s important not to forget the user experience. We need to give administrators the tools they need to make informed decisions about organisation subscriptions based on usage.

In fact, screenshots from Raptor immediately made me think of our last release of OpenAthens LA, which includes similar GUIs for displaying trend information captured over time.

OpenAthens provides administrators with a wide variety of statistics including rich graphical interfaces to data collected from users using either internal or external resources.

I just got back from a very successful REFEDS event at the APAN Conference in Chiang Mai. APAN is the equivalent of TERENA, but serving the Asia Pacific community.

Joining me at the event was Roland Van Rijswijk from SURFnet, to talk about the Tiqr app. Tiqr is a very neat tool that allows you to use a QR code to provide what we call two-factor authentication. Two-factor authentication helps to improve the security of the log-in process by asking you to provide two different things – typically something you know (such as a password) with something you have (such as a token). Most banks are now moving towards this by issuing readers, token generators or keypads to customers. The nice thing about the Tiqr approach is that it is not another thing to carry around, you use your mobile phone as the tool that collects and passes across the token – in this case the QR code.

Watching great developments like this inevitably brings about the question of assurance within the context of a federation for me. As usual within the federated world there are immediately two angles to consider: 1) technical trust – the sort of practical approach Tiqr is looking at and 2) policy trust.

Many Service Providers are interested in asking for greater levels of assurance, but many do not know what they want to ask for – do they want stronger authentication (perhaps a desire to tackle password sharing?) or do they want greater assurance of the identity vetting processes carried out by Identity Providers? One of the nice things about Tiqr is it provides more assurance with a manageable cost overhead – whereas identity vetting and audit can quickly cost an organisation a lot of money.

The UK federation is currently looking at both pulling together some advice for organisations that want to look at assurance, and higher trust within a federated context. Mindful of the cost implications for institutions, this is likely to take the form of some simple attribute profiles that can be flexibly applied within your organisation rather than the more formal assurance schemes – but we are very interested to hear from people with use cases for assurance.  Please contact the UK federation helpdesk if you’d like to share your requirements or discuss assurance requirements further.

For more details about the REFEDS BOF at APAN, please visit the REFEDS website.

But healthcare organisations and communities are often unable to join such federations because membership is largely confined to academic institutions. This has led many of them to contact Eduserv wondering if they are missing out or being left behind.

As I mentioned in my last post, we created the OpenAthens Federation to help our customers in the healthcare sector benefit from access management federations. Before it was launched we drew up membership criteria that would benefit each and every member.  When we invited healthcare customers to join the federation we had reactions like, “Yes, yes, yes! But… not all our customers are in the healthcare sector, we want them to be able join as well”. Fortunately, the membership criteria are broad enough to enable a wide range of organisations to participate.

Why join an access management federation?

So, what’s in it for your organisation?  In short, access to a wider range of online resources, products and services.  The OpenAthens Healthcare Federation gives service providers an option to make their Shibboleth-protected content available to your OpenAthens users, which simplifies their access options.  These improved integration capabilities also:

• increase security
• reduce administrative overheads
• improve statutory and regulatory compliance, and
• enable greater personalisation of information for end-users.

What’s in it for service providers?  Well, until now if your organization wanted to subscribe to a particular resource, but a service provider was not offering OpenAthens as an access option, then the service provider had to decide whether there was sufficient justification to implement OpenAthens software.

That may still be a good option for the service provider that doesn’t have Shibboleth, as OpenAthens can give them that capability.  However, if they already have Shibboleth-protected resources, then it’s likely that they are already available through multiple access management federations.  We are now enabling those service providers to simply add the OpenAthens Healthcare Federation [metadata] to their existing Shibboleth federations. This has a much smaller technical impact than a full OpenAthens implementation and so is a much more cost-effective option for them.

So the question is; what is changed from the informal Athens Federation to the OpenAthens Healthcare Federation?

In short – nothing. But beyond that, if your organisation starts subscribing to Shibboleth-protected content, then you may choose to add new links in MyAthens to those products so that your users can see them. These are simple, one-off actions that can be completed with just a few clicks of a mouse and are all documented in our Help section – the OpenAthens Service Desk can also advise you.

There are of course legal obligations to customers joining the OpenAthens Healthcare Federation, but these are unchanged from our existing Terms & Conditions for use of OpenAthens products and services and there are no new legal implications of membership.

Please read the OpenAthens Healthcare Federation wiki space for more information about benefits, eligibility and member responsibilities. If you have any questions or would like to know more about what it means for you, then please do not hesitate to contact Phil Leahy or me directly.