Christina's LIS Rant
Wednesday, January 07, 2009
A Structural Exploration of the Science Blogosphere: Director's Cut
Due to popular demand (well 3 requests :) ), this is a commentary and additional information for my conference paper and presentation:
Pikas, C. K. (2008). Detecting Communities in Science Blogs. Paper presented at eScience '08. IEEE Fourth International Conference on eScience, 2008. Indianapolis. 95-102. doi:10.1109/eScience.2008.30 (available in IEEE Xplore to institutional subscribers or e-mail me if you don't have access that way).
The presentation is embedded in another blog post, and is available online at SlideShare. The video of me talking about it is (will be?) available on the conference site, but I haven't gotten it to load.
Context:
I'm interested in scholarly communication in science, engineering, and math. Specifically, informal scholarly communication and how information and communication technologies, in particular social computing technologies, can/do/might impact informal scholarly communication in science/math/engineering. I'm also interested in knowledge production and public communication of science, two sub-areas of STS (this acronym has several translations - the most common probably science and technology studies).
As a blogger, and a 2-time (soon to be 3) attendee of what was the NC Science Blogging Conference and a reader of science blogs, I became curious about how and why scientists use blogs and if their use is: a) similar to how non-scientists use blogs b) for informal scholarly communication (to other scientists about their work) c) for public communication of science d) for personal information management e) maybe for team collaboration(?)... The first way I looked at this was by doing a study with content analysis and interviews of chemists and physicists (this has not been published yet, but maybe someday, these things aren't as perishable as writings in other fields, I hope). The second study swings all the way to a structural analysis of the science blogosphere - and that's what was reported here.
In social network analysis (SNA), you look at the link structure, not the attributes of the actors or nodes. The idea is that links show evidence of potential information flows or influence. You can pick out prestigious or central actors, and groups which are more tightly connected to each other than to the rest of the network.
The first major problem was locating science blogs - and even drawing any sort of boundary as to what a science blog was or wasn't. Given that I'm interested in how these things contribute to science, I drew the line thusly:
Blogs maintained by scientists that deal with any aspect of being a scientist
Blogs about scientific topics by non-scientists
Omitted were:
Primarily political speech
Ones maintained by corporations
Non-English language
(you could definitely draw the line somewhere else, but this is what I did!)
Also given that I'm a great searcher but almost not a coder at all, I did this by search, snowball, and any hook or crook to get as big a set as possible. I went to each of these, and copied off the URLS from the blogrolls (to answer a question from a Scibling - if you had a rotating list that showed up in javascript on the page source, I probably got it; if you have a second page with a list of 300 blogs (cough - Bora - cough); I probably got it, likewise if generated by like GoogleReader or something)... so this was incredibly tedious, and probably missed a few, but probably pretty accurate. So that was the first network.
The second network - and I originally had a much grander scheme - took the "most interesting" (most central by common measures) blogs from the first network, and then used Perl scripts (core script developed by Jen Golbeck, and then I customized to work for non-wordpress blogs, and blogs where people changed their templates a lot - you all really could have made this easier, lol) to pull all of the commenter links off of the last 10 posts (this was done in like April).
Blogs have links between them a) in the content b) in the blogroll c) in signed comments... other studies have used basically any link on the page, but the fact is that it's not really saying much to link within a post (a little link love, but not a real endorsement). Blogrolls are some sort of endorsement, typically, and signing a comment means *something*.
So then I ran all the typical SNA things across it to look at central actors and to find cohesive subgroups. As far as centrality - no real surprises. As far as cohesive subgroups - a bit more tricky. Basically one large component - and not terribly clumpy, with the exception of the astro bloggers - they're pretty tight. Most of the community detection techniques use a binary split - or start with binary splits - none of these were at all effective in dividing up the hairball. Spin glass, OTOH, worked beautifully to return 7 clusters. So then I went back and looked at the blog and figured out the commonality for each of the clusters (yes, I could have used some NLP to extract terms and automatically label the clusters, but there were 7 so...).
The single component isn't too surprising because we know from diffusion of innovations for ICTs that we would expect people to pick this up from other people and then probably link back. The power law degree distribution is also very typical when you're talking the activities of people (whether Lotka, Zipf, Pareto, Bradford.... whatever law). The clusters were related to subject areas - very broad subject areas. One question in my mind was how much people would be outside of their home discipline in their reading/commenting... based on this network, certainly outside of their particular speciality, but still in the neighborhood with the exception of a few "a-list" science bloggers who everyone reads.
What was interesting - and most definitely worthy of further investigation - is this cluster of blogs written mostly by women, discussing the scientific life, etc. The degree distribution was much closer to uniform within the cluster, and there were many comment links between all of the nodes. This, to me, indicates other uses for the blogs and perhaps a real community (or Blanchard's virtual settlement).
Also, picked out the troll very easily using the commenter network - so this method could be used to automate troll identification. (in the first study I talked about this guy with a physicist and the physicist basically only reins the troll in when he's so out of bounds as to be gross... so ID-ing a troll doesn't necessarily meaning banning).
I'm quickly running out of steam in this blog post - but this might end up being a pilot for my dissertation, so I'm definitely more than happy to talk about it either in the comments here, or on slideshare, or on friendfeed... or twitter or... just look for cpikas :)
Pikas, C. K. (2008). Detecting Communities in Science Blogs. Paper presented at eScience '08. IEEE Fourth International Conference on eScience, 2008. Indianapolis. 95-102. doi:10.1109/eScience.2008.30 (available in IEEE Xplore to institutional subscribers or e-mail me if you don't have access that way).
The presentation is embedded in another blog post, and is available online at SlideShare. The video of me talking about it is (will be?) available on the conference site, but I haven't gotten it to load.
Context:
I'm interested in scholarly communication in science, engineering, and math. Specifically, informal scholarly communication and how information and communication technologies, in particular social computing technologies, can/do/might impact informal scholarly communication in science/math/engineering. I'm also interested in knowledge production and public communication of science, two sub-areas of STS (this acronym has several translations - the most common probably science and technology studies).
As a blogger, and a 2-time (soon to be 3) attendee of what was the NC Science Blogging Conference and a reader of science blogs, I became curious about how and why scientists use blogs and if their use is: a) similar to how non-scientists use blogs b) for informal scholarly communication (to other scientists about their work) c) for public communication of science d) for personal information management e) maybe for team collaboration(?)... The first way I looked at this was by doing a study with content analysis and interviews of chemists and physicists (this has not been published yet, but maybe someday, these things aren't as perishable as writings in other fields, I hope). The second study swings all the way to a structural analysis of the science blogosphere - and that's what was reported here.
In social network analysis (SNA), you look at the link structure, not the attributes of the actors or nodes. The idea is that links show evidence of potential information flows or influence. You can pick out prestigious or central actors, and groups which are more tightly connected to each other than to the rest of the network.
The first major problem was locating science blogs - and even drawing any sort of boundary as to what a science blog was or wasn't. Given that I'm interested in how these things contribute to science, I drew the line thusly:
Blogs maintained by scientists that deal with any aspect of being a scientist
Blogs about scientific topics by non-scientists
Omitted were:
Primarily political speech
Ones maintained by corporations
Non-English language
(you could definitely draw the line somewhere else, but this is what I did!)
Also given that I'm a great searcher but almost not a coder at all, I did this by search, snowball, and any hook or crook to get as big a set as possible. I went to each of these, and copied off the URLS from the blogrolls (to answer a question from a Scibling - if you had a rotating list that showed up in javascript on the page source, I probably got it; if you have a second page with a list of 300 blogs (cough - Bora - cough); I probably got it, likewise if generated by like GoogleReader or something)... so this was incredibly tedious, and probably missed a few, but probably pretty accurate. So that was the first network.
The second network - and I originally had a much grander scheme - took the "most interesting" (most central by common measures) blogs from the first network, and then used Perl scripts (core script developed by Jen Golbeck, and then I customized to work for non-wordpress blogs, and blogs where people changed their templates a lot - you all really could have made this easier, lol) to pull all of the commenter links off of the last 10 posts (this was done in like April).
Blogs have links between them a) in the content b) in the blogroll c) in signed comments... other studies have used basically any link on the page, but the fact is that it's not really saying much to link within a post (a little link love, but not a real endorsement). Blogrolls are some sort of endorsement, typically, and signing a comment means *something*.
So then I ran all the typical SNA things across it to look at central actors and to find cohesive subgroups. As far as centrality - no real surprises. As far as cohesive subgroups - a bit more tricky. Basically one large component - and not terribly clumpy, with the exception of the astro bloggers - they're pretty tight. Most of the community detection techniques use a binary split - or start with binary splits - none of these were at all effective in dividing up the hairball. Spin glass, OTOH, worked beautifully to return 7 clusters. So then I went back and looked at the blog and figured out the commonality for each of the clusters (yes, I could have used some NLP to extract terms and automatically label the clusters, but there were 7 so...).
The single component isn't too surprising because we know from diffusion of innovations for ICTs that we would expect people to pick this up from other people and then probably link back. The power law degree distribution is also very typical when you're talking the activities of people (whether Lotka, Zipf, Pareto, Bradford.... whatever law). The clusters were related to subject areas - very broad subject areas. One question in my mind was how much people would be outside of their home discipline in their reading/commenting... based on this network, certainly outside of their particular speciality, but still in the neighborhood with the exception of a few "a-list" science bloggers who everyone reads.
What was interesting - and most definitely worthy of further investigation - is this cluster of blogs written mostly by women, discussing the scientific life, etc. The degree distribution was much closer to uniform within the cluster, and there were many comment links between all of the nodes. This, to me, indicates other uses for the blogs and perhaps a real community (or Blanchard's virtual settlement).
Also, picked out the troll very easily using the commenter network - so this method could be used to automate troll identification. (in the first study I talked about this guy with a physicist and the physicist basically only reins the troll in when he's so out of bounds as to be gross... so ID-ing a troll doesn't necessarily meaning banning).
I'm quickly running out of steam in this blog post - but this might end up being a pilot for my dissertation, so I'm definitely more than happy to talk about it either in the comments here, or on slideshare, or on friendfeed... or twitter or... just look for cpikas :)
Labels: IEEEeScience08
¶ 10:45 AM| (5) comments |cites (technorati) |
Friday, December 12, 2008
IEEE eScience: WOOL: A Workflow Programming Language
Speaker Geoff Hulette
workflows
(oh thank goodness, some definitions!)
scientists want to focus on domain problems but use computer models so must deal with programming
workflows
- separation of coordination from computation
- data flow naturally expresses parallelism
activities- primitive computations
- any number of inputs and outputs
- connections connect activity inputs and outputs
workflow is a collection of activities and connections
science – triana, vaverna, vistrails, kepler
business – bpel4ws, wsfl
mac – quartz composer
abstract – agwl
another language –
- simple workflow for neuroinformatics and image processing
- other systems don’t scale down easily wrt complexity
abstract – what not how
- runtime neutral
- same workflow representation ona laptop, cluster, or cloud
- text based with a simple syntax ( to use text tools like subversion)
- rich type system
- hierarchical and composable workflows
workflows
(oh thank goodness, some definitions!)
scientists want to focus on domain problems but use computer models so must deal with programming
workflows
- separation of coordination from computation
- data flow naturally expresses parallelism
activities- primitive computations
- any number of inputs and outputs
- connections connect activity inputs and outputs
workflow is a collection of activities and connections
science – triana, vaverna, vistrails, kepler
business – bpel4ws, wsfl
mac – quartz composer
abstract – agwl
another language –
- simple workflow for neuroinformatics and image processing
- other systems don’t scale down easily wrt complexity
abstract – what not how
- runtime neutral
- same workflow representation ona laptop, cluster, or cloud
- text based with a simple syntax ( to use text tools like subversion)
- rich type system
- hierarchical and composable workflows
Labels: IEEEeScience08
¶ 2:40 PM| (0) comments |cites (technorati) |
IEEE eScience: Final Keynote
Edward Seidel, Director, Office of Cyberinfrastructure, NSF (since the summer, I guess)
he’s a physicist and a computer scientist, formerly of LSU
how he got here – started in HEP then moved to general relativity-
- a series of problems that require supercomputers, computer scientists, high speed nets, grids, visualization
black hold perturbation theory – very hard, try supercomputers
black hole collisions
neutron star collisions
themes –
costs a lot, requires collaboration, idea to reuse tools..
coastal modeling – an example for cyberinfrastructure
another example LHC – quantity of data
data driven era of science
NSF vision (see Atkins report 2003 – Cyberinfrastructure Vision for the 21st century)
1 virtual organizations for distributed communities (large-scale teams to solve complex problems)
2. hpc
3. data visualization/interaction
4. learning and workforce
Data Net
- developing communities and tools to solve complex problems –example climate change : overlaying chemistry, environmental, etc.,
Virtual organizations for distributed communities
learning and workforce development
- need computational science (not cs but broader?) programs
Teragrid
- track 2 (Texas, Tenn, Pittsburgh… another under review)
- track 1 petascale university of Illinois – but will only serve a small number of scientists
next generation – xd – xtreme digital (still looking at this)
- innovative ways to support digital services, has explicit visualization component
open science grid and loosely coupled science grids
- integrate national or international needs into campus needs/support
- some of this is aimed at the LHC, but there are other efforts that can use it, too
but what about software?
- no real program at NSF to build the software to take advantage of these nets
applications to take advantage of this
virtual organizations to work out how to do all of this
Blue Waters
IBM Power 7 based system
online 2011
1 petaflop sustaind performance on real applications
>200k cores
idea is not a whole bunch of jobs – but a few jobs that need this kind of complexity
coupled dynamic ensemble simulations, real-time simulations – real policy issues like scheduling
will keep going with the international programs
translation to programs – get bandwidth to the center but not to the labs where its needed
federated id management
what next?
pick up and do remainder of things in vision
need end-to-end integration
need to support a computational science community
he comes up with the third pillar, too, experiment, theory computational science
he’s a physicist and a computer scientist, formerly of LSU
how he got here – started in HEP then moved to general relativity-
- a series of problems that require supercomputers, computer scientists, high speed nets, grids, visualization
black hold perturbation theory – very hard, try supercomputers
black hole collisions
neutron star collisions
themes –
costs a lot, requires collaboration, idea to reuse tools..
coastal modeling – an example for cyberinfrastructure
another example LHC – quantity of data
data driven era of science
NSF vision (see Atkins report 2003 – Cyberinfrastructure Vision for the 21st century)
1 virtual organizations for distributed communities (large-scale teams to solve complex problems)
2. hpc
3. data visualization/interaction
4. learning and workforce
Data Net
- developing communities and tools to solve complex problems –example climate change : overlaying chemistry, environmental, etc.,
Virtual organizations for distributed communities
learning and workforce development
- need computational science (not cs but broader?) programs
Teragrid
- track 2 (Texas, Tenn, Pittsburgh… another under review)
- track 1 petascale university of Illinois – but will only serve a small number of scientists
next generation – xd – xtreme digital (still looking at this)
- innovative ways to support digital services, has explicit visualization component
open science grid and loosely coupled science grids
- integrate national or international needs into campus needs/support
- some of this is aimed at the LHC, but there are other efforts that can use it, too
but what about software?
- no real program at NSF to build the software to take advantage of these nets
applications to take advantage of this
virtual organizations to work out how to do all of this
Blue Waters
IBM Power 7 based system
online 2011
1 petaflop sustaind performance on real applications
>200k cores
idea is not a whole bunch of jobs – but a few jobs that need this kind of complexity
coupled dynamic ensemble simulations, real-time simulations – real policy issues like scheduling
will keep going with the international programs
translation to programs – get bandwidth to the center but not to the labs where its needed
federated id management
what next?
pick up and do remainder of things in vision
need end-to-end integration
need to support a computational science community
he comes up with the third pillar, too, experiment, theory computational science
Labels: IEEEeScience08
¶ 2:14 PM| (0) comments |cites (technorati) |
IEEE eScience: Science in the Clouds
Alexander Szalay, Science in the Cloud
movement over time first towards pcs and small scale and then back through Beowulf clusters, the grid, and now clouds
paradigm that the data was in one location and had to be moved and cleaned up afterward wasn’t efficient, so now we’re looking at distributed data
20% of the worlds servers are going to the big 5 (clouds?)
Clouds – have some great benefits, the university model really doesn’t work anymore
clear management advantages
base line cost that is hard to match
easy to grow dynamically
science problems:
- tightly coupled parallelism (low latency, MPI- maybe message passing interface?)
- very high I/O bandwidth
- geographically separate data (can’t keep it in one place, in synch…)
clouds are fractals – exist on all scales
little, big, all science can use the clouds
need a power law distribution of computing facilities for scientific computing
Astro trends -
Cosmic Microwave Background (COBE, 1990, 1000pixels; WMAP, 2003, 1Mpixels; Planck, 2008, 10Mpixels)
Galaxy Redshift Surveys ( 1986-3500 to 2005 750000)
Time domain is new
Lsstars (?) - Petabytes
LSST – total dataset in the 100 Petabytes
scientific data doubles every year – successive generations of inexpensive sensors (new CMOS) and exponentially faster computing – this changes the nature of scientific computing across all areas of science
- but it’s harder to extract knowledge
data on all scales – kb in Excel spreadsheets manually maintained vs. multi-terabye archive facilities (all medical labs are gathering digital data all the time)
industrial revolution in collecting scientific data
acquire data (doubling)…. bottom of the funnel, publication, only growing at 6%
challenges:
access – move analysis to the data
discovery – typically discovery is done at the edges so more doesn’t give us much more… but opening up other orthogonal dimensions gives us more discoveries. federation still requires data movement
analysis – only max NlogN algorithms possible,
data analysis-
on all scales across multiple locations
assumption has been that there’s one optimal solution, that we just need a large enough data set… with unlimited computing power
but this isn’t true any more, randomized incremental algorithms
analysis of scientific data
- working with Jim Gray – coping with the data explosion starting in astro with the SDSS… first data release from Sloan was 100GB, now the dataset is about 100TB
- interactions with every step of the scientific process
Jim Gray:
- scientific computing revolving around data – take analysis to data, need scale-out solution for analysis
- scientists give the database designer top 20 questions in English, and the database designer or computer scientist can design the database accordingly
- build a little something that works today, and then build bigger /scale and build something working tomorrow – go from working to working – build what the world looks like today, not for tomorrow
Projects at JHU
SDSS – finished, final data release has happened
- work has changed
- final archiving in progress (UChi library, JHU library, CAS Mirrors at FnAL +JHU Physics & Astro)
archive will contain >100 TB
- all raw data
- all processed calibrated data
- all versions of the database
- full email archive (capture technical changes not in official drawings) and technical drawings
- full software code repository
- telescope sensor stream, IR fisheye camera, etc.
Public use of the skyserver
- prototype in data publishing
- 500 Million web hits in 6 years, 1M distinct users but only 15k astromers in the world
- 50,000 lectures to high schools
- delivered >100B rows of data
interactive workbench
- sign up, own database, run query, pipe to own database… analysis tools to transfer only plot and not entire database over the wires
- 2,400 power users
GalaxyZoo
built on SkyServer, 27M visual galaxy classification by the public
Dutch school teacher discovery
Virtual Observatory
collaboration of 20 groups
15 countries – international virtual observatory alliance
interfaces were different, no central registry, underlying basic data formats were agreed upon
sociological barriers are much more difficult than technical challenges
Technology
- petabytes
- save, move, some processing near the instrument (for example) in Chile
- funding organizations have to understand the computing costs over time
- open ended modular system
- need Journal for Data (overlay to bridge the gap so that data sets don’t get lost) – curation is key, who does the long-term curation of data
Pan-STARRS
- detect killer asteroids
- >1Petabyte/year
- 80TB SQL Server database built at JHU – largest astro db in the world
Life Under Your Feet (http://lifeunderyourfeet.org/en/default.asp)
- role of soil in global change
- a few hundred wireless computers with 10 sensors each , long term continuous data, complex database of sensor data, built from the sky server
once the project is online, linear growth, exponential growth comes from new technologies new cameras … future might come from individual amateur astronomers using 20MB cameras on their telescopes and systematically gathering data
more growth coming from simulations (software is also an instrument)
(example of one that was so big the tape robot was inaccessible so the data is never used)
also need interactive, immersive usages (like for turbulence)
- store every time slice in the database
- turbulence.pha.jhu.edu (try it today!)
commonalities
- huge amounts of data, need aggregates but also access to raw data
- requests enormously benefit from indexing
Amdahl’s Laws for a balanced system – we’ve gone farther and farther from these
Comparisons of simulations and their data generation vs. what’s available in the computers
data analysis is maxing out the hardware because of the 10-100 TB – no one can really do anything with over 50 TB
IO limitations for analysis
we’re a factor of 500 off of what we can do to get to a 200TFlop Amdahl machine
they built a high io system using cheap components
the large datasets are here, the solutions are not – systems are choking on IO
scientists are cheap
data collection is separated from data analysis
(big experiments just collect data and store it, scientists come along later and analyze the data- decoupled)
How do users interact with petabytes
- can’t wait 2wks to do a sql query on a petabyte
- python crawlers
- partition queries
- MapReduce/Hadoop – but can’t do (or are very difficult) complex joins you need to do for data analysis
William Gibson – “The future is already here. It’s just not very evenly distributed”
data cloud vs. HPC or HTC
journal for data?
- with ApJ?
- example postdoc writes a paper, only a table goes into the suppl data, journal can't take the terabytes real data - need another archive for this that's linked to science article.
movement over time first towards pcs and small scale and then back through Beowulf clusters, the grid, and now clouds
paradigm that the data was in one location and had to be moved and cleaned up afterward wasn’t efficient, so now we’re looking at distributed data
20% of the worlds servers are going to the big 5 (clouds?)
Clouds – have some great benefits, the university model really doesn’t work anymore
clear management advantages
base line cost that is hard to match
easy to grow dynamically
science problems:
- tightly coupled parallelism (low latency, MPI- maybe message passing interface?)
- very high I/O bandwidth
- geographically separate data (can’t keep it in one place, in synch…)
clouds are fractals – exist on all scales
little, big, all science can use the clouds
need a power law distribution of computing facilities for scientific computing
Astro trends -
Cosmic Microwave Background (COBE, 1990, 1000pixels; WMAP, 2003, 1Mpixels; Planck, 2008, 10Mpixels)
Galaxy Redshift Surveys ( 1986-3500 to 2005 750000)
Time domain is new
Lsstars (?) - Petabytes
LSST – total dataset in the 100 Petabytes
scientific data doubles every year – successive generations of inexpensive sensors (new CMOS) and exponentially faster computing – this changes the nature of scientific computing across all areas of science
- but it’s harder to extract knowledge
data on all scales – kb in Excel spreadsheets manually maintained vs. multi-terabye archive facilities (all medical labs are gathering digital data all the time)
industrial revolution in collecting scientific data
acquire data (doubling)…. bottom of the funnel, publication, only growing at 6%
challenges:
access – move analysis to the data
discovery – typically discovery is done at the edges so more doesn’t give us much more… but opening up other orthogonal dimensions gives us more discoveries. federation still requires data movement
analysis – only max NlogN algorithms possible,
data analysis-
on all scales across multiple locations
assumption has been that there’s one optimal solution, that we just need a large enough data set… with unlimited computing power
but this isn’t true any more, randomized incremental algorithms
analysis of scientific data
- working with Jim Gray – coping with the data explosion starting in astro with the SDSS… first data release from Sloan was 100GB, now the dataset is about 100TB
- interactions with every step of the scientific process
Jim Gray:
- scientific computing revolving around data – take analysis to data, need scale-out solution for analysis
- scientists give the database designer top 20 questions in English, and the database designer or computer scientist can design the database accordingly
- build a little something that works today, and then build bigger /scale and build something working tomorrow – go from working to working – build what the world looks like today, not for tomorrow
Projects at JHU
SDSS – finished, final data release has happened
- work has changed
- final archiving in progress (UChi library, JHU library, CAS Mirrors at FnAL +JHU Physics & Astro)
archive will contain >100 TB
- all raw data
- all processed calibrated data
- all versions of the database
- full email archive (capture technical changes not in official drawings) and technical drawings
- full software code repository
- telescope sensor stream, IR fisheye camera, etc.
Public use of the skyserver
- prototype in data publishing
- 500 Million web hits in 6 years, 1M distinct users but only 15k astromers in the world
- 50,000 lectures to high schools
- delivered >100B rows of data
interactive workbench
- sign up, own database, run query, pipe to own database… analysis tools to transfer only plot and not entire database over the wires
- 2,400 power users
GalaxyZoo
built on SkyServer, 27M visual galaxy classification by the public
Dutch school teacher discovery
Virtual Observatory
collaboration of 20 groups
15 countries – international virtual observatory alliance
interfaces were different, no central registry, underlying basic data formats were agreed upon
sociological barriers are much more difficult than technical challenges
Technology
- petabytes
- save, move, some processing near the instrument (for example) in Chile
- funding organizations have to understand the computing costs over time
- open ended modular system
- need Journal for Data (overlay to bridge the gap so that data sets don’t get lost) – curation is key, who does the long-term curation of data
Pan-STARRS
- detect killer asteroids
- >1Petabyte/year
- 80TB SQL Server database built at JHU – largest astro db in the world
Life Under Your Feet (http://lifeunderyourfeet.org/en/default.asp)
- role of soil in global change
- a few hundred wireless computers with 10 sensors each , long term continuous data, complex database of sensor data, built from the sky server
once the project is online, linear growth, exponential growth comes from new technologies new cameras … future might come from individual amateur astronomers using 20MB cameras on their telescopes and systematically gathering data
more growth coming from simulations (software is also an instrument)
(example of one that was so big the tape robot was inaccessible so the data is never used)
also need interactive, immersive usages (like for turbulence)
- store every time slice in the database
- turbulence.pha.jhu.edu (try it today!)
commonalities
- huge amounts of data, need aggregates but also access to raw data
- requests enormously benefit from indexing
Amdahl’s Laws for a balanced system – we’ve gone farther and farther from these
Comparisons of simulations and their data generation vs. what’s available in the computers
data analysis is maxing out the hardware because of the 10-100 TB – no one can really do anything with over 50 TB
IO limitations for analysis
we’re a factor of 500 off of what we can do to get to a 200TFlop Amdahl machine
they built a high io system using cheap components
the large datasets are here, the solutions are not – systems are choking on IO
scientists are cheap
data collection is separated from data analysis
(big experiments just collect data and store it, scientists come along later and analyze the data- decoupled)
How do users interact with petabytes
- can’t wait 2wks to do a sql query on a petabyte
- python crawlers
- partition queries
- MapReduce/Hadoop – but can’t do (or are very difficult) complex joins you need to do for data analysis
William Gibson – “The future is already here. It’s just not very evenly distributed”
data cloud vs. HPC or HTC
journal for data?
- with ApJ?
- example postdoc writes a paper, only a table goes into the suppl data, journal can't take the terabytes real data - need another archive for this that's linked to science article.
Labels: IEEEeScience08
¶ 9:37 AM| (0) comments |cites (technorati) |
Thursday, December 11, 2008
IEEE eScience: ARCHER: An Enabler of Research Data Management
ARCHER – came in late
Speaker: Anthony Beitz
research data management
life cycle
conceive, design, experiment, analyze, collaborate, publish, expose
(this seems strange to put collaborate there)
theirs helps from experiment, analyze, publish
includes
- research repository – data curation, with rich metadata
- concurrent data capture and telemetry
- dataset manager including (web and desktop client for large datasets so no time out), metadata editing tool
- collaborative and adaptable research portal dev environment
currently available www.archer.edu.au
uses plone? xdms – also does automatic metadata extraction upon deposit
core metadata based on STFC’s scientific metadata model
flexible metadata for samples, datasets, and datafiles
stfc project > experiment > dataset > data file
for experiment: publication, keywords, topic list, investigator, sample
DIMSIM – distributed integrated multi-sensor & instrument middleware
- convenient for large data in bulk from instrument to repository
- enables concurrent analysis
XDMS – scientific dataset manager
web tool for researchers to manage and curate their research process
- formalized research data management
- automated metat extraction
- persistent identifiers for each dataset (I guess at the whole dataset level – using handle technology)
- powerful search capabilities ( he didn’t say much here)
(q: tell more about search – just metadata?)
- secure
- publish from there into their institutional repository
- currently customized to crystallography
Metadata editor validates schema
Hermes is the desktop tool
- doesn’t have data time out problems
Speaker: Anthony Beitz
research data management
life cycle
conceive, design, experiment, analyze, collaborate, publish, expose
(this seems strange to put collaborate there)
theirs helps from experiment, analyze, publish
includes
- research repository – data curation, with rich metadata
- concurrent data capture and telemetry
- dataset manager including (web and desktop client for large datasets so no time out), metadata editing tool
- collaborative and adaptable research portal dev environment
currently available www.archer.edu.au
uses plone? xdms – also does automatic metadata extraction upon deposit
core metadata based on STFC’s scientific metadata model
flexible metadata for samples, datasets, and datafiles
stfc project > experiment > dataset > data file
for experiment: publication, keywords, topic list, investigator, sample
DIMSIM – distributed integrated multi-sensor & instrument middleware
- convenient for large data in bulk from instrument to repository
- enables concurrent analysis
XDMS – scientific dataset manager
web tool for researchers to manage and curate their research process
- formalized research data management
- automated metat extraction
- persistent identifiers for each dataset (I guess at the whole dataset level – using handle technology)
- powerful search capabilities ( he didn’t say much here)
(q: tell more about search – just metadata?)
- secure
- publish from there into their institutional repository
- currently customized to crystallography
Metadata editor validates schema
Hermes is the desktop tool
- doesn’t have data time out problems
Labels: IEEEeScience08
¶ 12:09 PM| (0) comments |cites (technorati) |
IEEE eScience: Web services architecture for visualization
Web services architecture for visualization
Jeremy Walton, Numerical Algorithms Group
convert data to information leading to knowledge
visualization and statistical analysis – visual analytics
ADVISE system–
- visual analytics
- enables creation of analysis application (generic or domain specific)
- service oriented architecture
2 existing systems
IRIS explorer – visualization algorithms
GenStat – statistical algorithms
merge these > advise > clients
3 layers visualization components > web services middleware > user interface
modules connected in pipelines
(modules from the older systems, plus some new ones)
middleware – stateful service so that data in calls can be re-used, maintains a model of pipeline state..
user interface – in java, can use pre-built pipelines or build your own….
(he demoed one where you could pick off the modules and build it – and another with dropdown boxes and a go button)
future work-
real user data sets
intelligent data types (streaming data – that’s cool but does it come packaged?, sharing large data)
more fine grained visualization services
connecting to other web services
Jeremy Walton, Numerical Algorithms Group
convert data to information leading to knowledge
visualization and statistical analysis – visual analytics
ADVISE system–
- visual analytics
- enables creation of analysis application (generic or domain specific)
- service oriented architecture
2 existing systems
IRIS explorer – visualization algorithms
GenStat – statistical algorithms
merge these > advise > clients
3 layers visualization components > web services middleware > user interface
modules connected in pipelines
(modules from the older systems, plus some new ones)
middleware – stateful service so that data in calls can be re-used, maintains a model of pipeline state..
user interface – in java, can use pre-built pipelines or build your own….
(he demoed one where you could pick off the modules and build it – and another with dropdown boxes and a go button)
future work-
real user data sets
intelligent data types (streaming data – that’s cool but does it come packaged?, sharing large data)
more fine grained visualization services
connecting to other web services
Labels: IEEEeScience08
¶ 11:34 AM| (0) comments |cites (technorati) |
IEEE eScience: Dan Reed's Keynote
Keynote:
Daniel A. Reed
Microsoft Research
Cloud Seeding: Watering Research Flowers
(slides will not be shared per his request)
3 pillars of discovery: theory, experiment, and computational e-Science (ok, so observational sciences don’t provide new discoveries?)
(ew viscous flow in disposable diapers)
truisms- bulk computing is almost free but software and power aren’t
ubiquitous sensors – but lagging in data fusion
moving lots of data is still hard
people are expensive – robust software is extremely labor intensive
scientific challenges are complex and social engineering is not our forte – increasingly social engineering is the limiting factor in our success (pesky users)
political/technical approaches much change or we risk solving irrelevant problems
how do we develop effective programming tools for the average jane who is using at best Perl, MATLAB, etc…. typical scientist or engineer vs. savvy computer scientist
social implications of the data deluge
- hypothesis driven (data was pretty scarce in the past)
- exploratory –“what correlations can I glean from everyone’s data?”
- this requires different tools and techniquest
- massive multidisciplinary data rising rapidly at an unprecedented scale
clash of cosmologies – article that astro was going from observation to mere measurement
scientific computing is along for the ride instead of in the drivers seat – we use GPUs bcs the commercial industry is driving innovation in the processors and the average desktop machine doesn’t really need to be any faster for the typical office or home tasks… so we have to use these tools developed for the commercial market
use these commercial cores and parallelism – new approaches are needed to really take advantage of the parallelism
cloud application frameworks (slide from Dennis Gannon)
OS virtualization > software as a service > parallel frameworks (this is a triangle)
Hadoop over EC3 is 2/3 the way toward OS virt from parallel
GFS, BigTable,MapReduce at parallel frameworks
Amazon S3/EC2 at OS virt
Microsoft mesh at Software as a service
Microsoft Azure Services Platform- cloud services
services platform (hosted sharepoint, CRM, SQL, .NET….)
(I was hearing about software as a service ages ago so I guess now it’s becoming more mainstream)
Data Center costs for cloud computing- (physical plant and power issues are huge)
land 2%
core and shell costs 9%
architectural 7%
mechanical and electrical – rest
energy and supporting infrastructure cost like 4x the cost of the 1u server
researchers have different perceptions about the cost of ownership bcs they don’t see the power bills, only the server and people costs
15MW is what you need to build a cloud computing facility (15 yr amortization)
servers 2k each, 50,000
commercial power .07kWhr
security/administration 15 people @$100k/yr
$3M/months related to power
instruments and infrastructure
- from desktop to lab level to organization level maybe state level then national level and regional level
building blocks of cyberinfrastructure – this is his slide from 10 years ago
in the past 10 years
- commodity clusters – proliferation, of inexpensive hardware, race for MachoFLOPS, broadbase for enabling software, low level programming changes
- grids and distributed services – multidisciplinary collaborations, less broad base for enabling software
research money vs. production, maintenance, on going reliable work
teraflop is no news now, done in the lab with linux clusters
security – PII, HiPPA – research machines still have to be secure and patched and this is a real cost (not just an inconvenience)
business
- capital is cheap, labor is expensive, costs are explicit
academia, govt
- capital is hard, labor is seeming cheap (students!), costs are implicit
funding is at best flat
infrastructure inefficiency reduces research funding – need to become more efficient
so this is his argument to go to cloud computing
- elasticity
- economies of scale
- efficiency
- cost clarity
- pay as you go
- support
- geodistribution (security)
Daniel A. Reed
Microsoft Research
Cloud Seeding: Watering Research Flowers
(slides will not be shared per his request)
3 pillars of discovery: theory, experiment, and computational e-Science (ok, so observational sciences don’t provide new discoveries?)
(ew viscous flow in disposable diapers)
truisms- bulk computing is almost free but software and power aren’t
ubiquitous sensors – but lagging in data fusion
moving lots of data is still hard
people are expensive – robust software is extremely labor intensive
scientific challenges are complex and social engineering is not our forte – increasingly social engineering is the limiting factor in our success (pesky users)
political/technical approaches much change or we risk solving irrelevant problems
how do we develop effective programming tools for the average jane who is using at best Perl, MATLAB, etc…. typical scientist or engineer vs. savvy computer scientist
social implications of the data deluge
- hypothesis driven (data was pretty scarce in the past)
- exploratory –“what correlations can I glean from everyone’s data?”
- this requires different tools and techniquest
- massive multidisciplinary data rising rapidly at an unprecedented scale
clash of cosmologies – article that astro was going from observation to mere measurement
scientific computing is along for the ride instead of in the drivers seat – we use GPUs bcs the commercial industry is driving innovation in the processors and the average desktop machine doesn’t really need to be any faster for the typical office or home tasks… so we have to use these tools developed for the commercial market
use these commercial cores and parallelism – new approaches are needed to really take advantage of the parallelism
cloud application frameworks (slide from Dennis Gannon)
OS virtualization > software as a service > parallel frameworks (this is a triangle)
Hadoop over EC3 is 2/3 the way toward OS virt from parallel
GFS, BigTable,MapReduce at parallel frameworks
Amazon S3/EC2 at OS virt
Microsoft mesh at Software as a service
Microsoft Azure Services Platform- cloud services
services platform (hosted sharepoint, CRM, SQL, .NET….)
(I was hearing about software as a service ages ago so I guess now it’s becoming more mainstream)
Data Center costs for cloud computing- (physical plant and power issues are huge)
land 2%
core and shell costs 9%
architectural 7%
mechanical and electrical – rest
energy and supporting infrastructure cost like 4x the cost of the 1u server
researchers have different perceptions about the cost of ownership bcs they don’t see the power bills, only the server and people costs
15MW is what you need to build a cloud computing facility (15 yr amortization)
servers 2k each, 50,000
commercial power .07kWhr
security/administration 15 people @$100k/yr
$3M/months related to power
instruments and infrastructure
- from desktop to lab level to organization level maybe state level then national level and regional level
building blocks of cyberinfrastructure – this is his slide from 10 years ago
in the past 10 years
- commodity clusters – proliferation, of inexpensive hardware, race for MachoFLOPS, broadbase for enabling software, low level programming changes
- grids and distributed services – multidisciplinary collaborations, less broad base for enabling software
research money vs. production, maintenance, on going reliable work
teraflop is no news now, done in the lab with linux clusters
security – PII, HiPPA – research machines still have to be secure and patched and this is a real cost (not just an inconvenience)
business
- capital is cheap, labor is expensive, costs are explicit
academia, govt
- capital is hard, labor is seeming cheap (students!), costs are implicit
funding is at best flat
infrastructure inefficiency reduces research funding – need to become more efficient
so this is his argument to go to cloud computing
- elasticity
- economies of scale
- efficiency
- cost clarity
- pay as you go
- support
- geodistribution (security)
Labels: IEEEeScience08
¶ 10:07 AM| (0) comments |cites (technorati) |
Wednesday, December 10, 2008
IEEE eScience: Experiences from Cyberinfrastructure Development for Multiuser Remote Instrumentation
Experiences from Cyberinfrastructure Development for Multiuser Remote Instrumentation
Presenter Prasad Calyam, Ohio Supercomputer Center
remote instruments
- v.v. expensive in startup and maintenance so let other people use yours, use someone elses
- becoming a requirement from funding agencies
- better ROi on cost
- access via internet
pilot to leverage investments in networking hpc and scientific instruments – paid for by regents of university system of Ohio
SEM, raman spec, telescopes, accelerator, nmr spec
remote user site
-remove observation, operationi voice/text chat, lab notebook
osc
- portal dev, data storage, analytics, security (network and data)
instrument lab
- resource scheduling, billing, use policy, sample handling
Challenges
- last-mile network bottlenecks
- communications – remote user/operator communications, multi-device views for user workflows – multiple remote users, or single remote users – not useful off the shelf
- dead mans switch – if operator becomes incapacitated (or bad things happen at the instrument while being operated remotely)
- security for the data and network
Policy challenges
- prioritizing
- licensing
- SLA with vendors
- safe-use policy expert vs. novice use policies
- billing
Case studies
they build custom interfaces – the COTS ones they tried had some wonky problems
RICE features
network aware video encoding
control blocking – lock passing – only one operator at a time so they don’t step on each other
web-portal features
user account
management of instruments, people, data
access control
chat
storage of experiment data
more reasons
- remote participants can watch an expert control a scientific instrument – efficiently and reliably (network awareness mitigates instruments damage)
- expert can pass control to remote user (for training)
future directions
- wikis, lab notebooks, other communication tools
- human-centered remote so you get the “at-the-instrument” experiment
from audience: group working on remote instruments, etc., please join, etc.
Remote instrumentation services in grid environment
OGF RISGE-RG
forge.gridforum.org/sf/projects/risge-rg
Presenter Prasad Calyam, Ohio Supercomputer Center
remote instruments
- v.v. expensive in startup and maintenance so let other people use yours, use someone elses
- becoming a requirement from funding agencies
- better ROi on cost
- access via internet
pilot to leverage investments in networking hpc and scientific instruments – paid for by regents of university system of Ohio
SEM, raman spec, telescopes, accelerator, nmr spec
remote user site
-remove observation, operationi voice/text chat, lab notebook
osc
- portal dev, data storage, analytics, security (network and data)
instrument lab
- resource scheduling, billing, use policy, sample handling
Challenges
- last-mile network bottlenecks
- communications – remote user/operator communications, multi-device views for user workflows – multiple remote users, or single remote users – not useful off the shelf
- dead mans switch – if operator becomes incapacitated (or bad things happen at the instrument while being operated remotely)
- security for the data and network
Policy challenges
- prioritizing
- licensing
- SLA with vendors
- safe-use policy expert vs. novice use policies
- billing
Case studies
they build custom interfaces – the COTS ones they tried had some wonky problems
RICE features
network aware video encoding
control blocking – lock passing – only one operator at a time so they don’t step on each other
web-portal features
user account
management of instruments, people, data
access control
chat
storage of experiment data
more reasons
- remote participants can watch an expert control a scientific instrument – efficiently and reliably (network awareness mitigates instruments damage)
- expert can pass control to remote user (for training)
future directions
- wikis, lab notebooks, other communication tools
- human-centered remote so you get the “at-the-instrument” experiment
from audience: group working on remote instruments, etc., please join, etc.
Remote instrumentation services in grid environment
OGF RISGE-RG
forge.gridforum.org/sf/projects/risge-rg
Labels: IEEEeScience08
¶ 4:29 PM| (0) comments |cites (technorati) |
IEEE eScience: Classification of Different Approaches for e-Science Applications
Unfortunately not as high a level overview - I mean it was basically a typology of requests for and uses of the European grids - so one level of escience, for sure.
Classification of Different Approaches for e-Science Applications
what is e-science
sits on top of
theory & models + computational techniques + experiment
grids
HPC grids and then others that are HPG (high throughput grid) – national and then European
how do scientists get computational time (similar to like a telescope or other big science) proposal, granted, scheduled...
5 paradigms on deisa
1: simple scripts & control
- write in emacs or whatever
- use unicore client with the scripts - grid middleware does this
2: Scientific Application plug-ins
3: Complex workflow
- example QSAR
4: Interactive access
- more frequently physicists – ssh connection to the machine – to check results during computations
- computational steering – changing parameters on the fly
- collaborative online visualization and computational steering (COVS)
5: Interoperability
- using multiple grids
- grids are 6x and 4x overbooked
- can’t use the same middleware for both of their grids
Classification of Different Approaches for e-Science Applications
what is e-science
sits on top of
theory & models + computational techniques + experiment
grids
HPC grids and then others that are HPG (high throughput grid) – national and then European
how do scientists get computational time (similar to like a telescope or other big science) proposal, granted, scheduled...
5 paradigms on deisa
1: simple scripts & control
- write in emacs or whatever
- use unicore client with the scripts - grid middleware does this
2: Scientific Application plug-ins
3: Complex workflow
- example QSAR
4: Interactive access
- more frequently physicists – ssh connection to the machine – to check results during computations
- computational steering – changing parameters on the fly
- collaborative online visualization and computational steering (COVS)
5: Interoperability
- using multiple grids
- grids are 6x and 4x overbooked
- can’t use the same middleware for both of their grids
Labels: IEEEeScience08
¶ 3:54 PM| (0) comments |cites (technorati) |
IEEE eScience: MyExperiment
MyExperiment.org
myExperiment: Defining the Social Virtual Research Environment
David De Roure, University of Southampton
The point of this talk is to define what this thing is.
social process of science
- more to share
data, metadata, provenance, workflows, ontologies
these are the things that myexperiment is meant to help people share
workflow systems includes trident, taverna, kepler, triana, Ptolemy II
automate work to make reproducible and shareable
these workflows are difficult to build so it’s good to start with something – share
also they capture the context
it’s like sharing matlab scripts or models… not really sharing data, although you can do that, too.
example workflow for identifying biological pathways impliciated in resistance to x in cattle
jo working on whipworm in mouse – meets paul, uses his workflow without being changed, and fins the biological pathways for sex dependence in mice.
is
community social network
fine control over sharing
federated repository
gateway to other publishing environments
platform for launching workflows
started 3/2007, open beta 11/2007 – 1331 users, 536 workflows
workflow – social metadata including
license
credits
attributions
tags
uploade
(q: how do scientists search for and find relevant workflows? - he says there's a paper on their wiki)
groups – sometimes more formal to control access to information
metrics are complicated – what do downloads or visitors mean?
but more content is good
more content – harder to find – but more people, so more social and use information to help you find it (can add in recommender systems)
other “social web sites for scientists” are really just linkedin but with different people – they’re trying to differentiate themselves.
credits & attributions (critical)
fine control over visibility and sharing
packs (make a collection)
federation
enactment (actually run workflows)
Packs –
obvious but really necessary – to make a shopping basket (ppt slide from a workflow or collection of workflows)
collect internal things or link to external things
share, tag, discover, and discuss
enactor is complicated – if workflow has to be in the same place as the data, or if it doesn’t then…
It’s open source, and it also has RESTful sources, Google gadgets
Taverna Plugin
export packs as OAI-ORE (they would like someone to consume this – they’ve produced it, but would like to see repositories using it)
there’s also SPARQL searching
using SIOC: semantically-interlinked online communities
software design for empowering scientists (their article – next issue IEEE software)
six principles of software design to empower scientists
1 fit in, don’t force change
2 jam today and more jam tomorrow
3 just in time and just enough
4 act local, think global
5 enable users to add value
6 design for network effects
the process
discovery > acquisition > prep/cleaning > transformation > interpretations > dissemination > curation
across all of these, provenance… and two other things that I didn’t type fast enough and have forgotten
can they make an e-laboratory? (it’s the office of the future :) ) – can there be an ecosystem of cooperating e-laboratories
what are the research objects?
new project BioCatalogue – a catalog of biosciences web services (http://www.biocatalogue.org/)
What is a virtual research environment?
facilitate mgmt and sharing of research objects (whatever they are)
support the social model?
open extensible environment
platform to action research – to deliver research objects to remote services and software.
future – sharing R, matlab, statistical models
question: is the community big enough for reputation?
answer: well primary reputation comes through publication – but as the site grows this is developing, favoriting , annotations.
myExperiment: Defining the Social Virtual Research Environment
David De Roure, University of Southampton
The point of this talk is to define what this thing is.
social process of science
- more to share
data, metadata, provenance, workflows, ontologies
these are the things that myexperiment is meant to help people share
workflow systems includes trident, taverna, kepler, triana, Ptolemy II
automate work to make reproducible and shareable
these workflows are difficult to build so it’s good to start with something – share
also they capture the context
it’s like sharing matlab scripts or models… not really sharing data, although you can do that, too.
example workflow for identifying biological pathways impliciated in resistance to x in cattle
jo working on whipworm in mouse – meets paul, uses his workflow without being changed, and fins the biological pathways for sex dependence in mice.
is
community social network
fine control over sharing
federated repository
gateway to other publishing environments
platform for launching workflows
started 3/2007, open beta 11/2007 – 1331 users, 536 workflows
workflow – social metadata including
license
credits
attributions
tags
uploade
(q: how do scientists search for and find relevant workflows? - he says there's a paper on their wiki)
groups – sometimes more formal to control access to information
metrics are complicated – what do downloads or visitors mean?
but more content is good
more content – harder to find – but more people, so more social and use information to help you find it (can add in recommender systems)
other “social web sites for scientists” are really just linkedin but with different people – they’re trying to differentiate themselves.
credits & attributions (critical)
fine control over visibility and sharing
packs (make a collection)
federation
enactment (actually run workflows)
Packs –
obvious but really necessary – to make a shopping basket (ppt slide from a workflow or collection of workflows)
collect internal things or link to external things
share, tag, discover, and discuss
enactor is complicated – if workflow has to be in the same place as the data, or if it doesn’t then…
It’s open source, and it also has RESTful sources, Google gadgets
Taverna Plugin
export packs as OAI-ORE (they would like someone to consume this – they’ve produced it, but would like to see repositories using it)
there’s also SPARQL searching
using SIOC: semantically-interlinked online communities
software design for empowering scientists (their article – next issue IEEE software)
six principles of software design to empower scientists
1 fit in, don’t force change
2 jam today and more jam tomorrow
3 just in time and just enough
4 act local, think global
5 enable users to add value
6 design for network effects
the process
discovery > acquisition > prep/cleaning > transformation > interpretations > dissemination > curation
across all of these, provenance… and two other things that I didn’t type fast enough and have forgotten
can they make an e-laboratory? (it’s the office of the future :) ) – can there be an ecosystem of cooperating e-laboratories
what are the research objects?
new project BioCatalogue – a catalog of biosciences web services (http://www.biocatalogue.org/)
What is a virtual research environment?
facilitate mgmt and sharing of research objects (whatever they are)
support the social model?
open extensible environment
platform to action research – to deliver research objects to remote services and software.
future – sharing R, matlab, statistical models
question: is the community big enough for reputation?
answer: well primary reputation comes through publication – but as the site grows this is developing, favoriting , annotations.
Labels: IEEEeScience08
¶ 3:17 PM| (0) comments |cites (technorati) |
IEEE eScience: My Slides
Slightly more information in an earlier post.
Paper available only upon request until I figure out the rules (just e-mail me) - it will probably be in Xplore in a couple of months.
Slightly more information in an earlier post.
Paper available only upon request until I figure out the rules (just e-mail me) - it will probably be in Xplore in a couple of months.
Labels: IEEEeScience08, science blogging, sna
¶ 2:10 PM| (0) comments |cites (technorati) |
IEEE eScience: Acoustic Environment Observatory
Toward an Acoustic Environment Observatory
Authors from QUT
(Paul Roe from Microsoft QUT eResearch Centre speaking)
Ecosystem processes –
standard sensor monitor cycling of water and nutrients, energy transfers – but you need something different to look at biodiversity and species interactions with the environment – they’re using acoustics
scale up from traditional processes of monitoring with eyes and ears in the field (and taperecorders)
what to study with acoustics?
- detect
- behavior
- examples climate change (detect migratory species), invasive species, indicator species
- (they’re doing terrestrial, much has already been done with underwater
trying to do this at scale
- cheap sensors, automatic processing
- system is always evolving (data errors, broken sensors)
difference between – sensing the temperature 1x/hour and send, vs. continuous collection (kind of like reverse podcasting). Very different from like home security surveillance testing
data > middleware to manage sensors > signal labeling manual/semi-auto/auto (SciBot)> semantics, tag meaning and inference (behaviors and annotations)
they use solar power and cell phones with special software
web interface for scientists to schedule recordings, phones use 3G to upload the data
power is limiting – use too much for go too long without the sun – also uploading takes longer in bad weather
signal processing
- manual
- template based like speech recognition
- information theoretic approach to find signal
analysis and visualization
- point data this bird, this call, this time and place -- weave together to understand more about environmental health
useful in some environmental impact studies, tracking this rare and hard to see bird
also, why do koalas bellow
can listen: www.mquter.qut.edu.au/sensor
Authors from QUT
(Paul Roe from Microsoft QUT eResearch Centre speaking)
Ecosystem processes –
standard sensor monitor cycling of water and nutrients, energy transfers – but you need something different to look at biodiversity and species interactions with the environment – they’re using acoustics
scale up from traditional processes of monitoring with eyes and ears in the field (and taperecorders)
what to study with acoustics?
- detect
- behavior
- examples climate change (detect migratory species), invasive species, indicator species
- (they’re doing terrestrial, much has already been done with underwater
trying to do this at scale
- cheap sensors, automatic processing
- system is always evolving (data errors, broken sensors)
difference between – sensing the temperature 1x/hour and send, vs. continuous collection (kind of like reverse podcasting). Very different from like home security surveillance testing
data > middleware to manage sensors > signal labeling manual/semi-auto/auto (SciBot)> semantics, tag meaning and inference (behaviors and annotations)
they use solar power and cell phones with special software
web interface for scientists to schedule recordings, phones use 3G to upload the data
power is limiting – use too much for go too long without the sun – also uploading takes longer in bad weather
signal processing
- manual
- template based like speech recognition
- information theoretic approach to find signal
analysis and visualization
- point data this bird, this call, this time and place -- weave together to understand more about environmental health
useful in some environmental impact studies, tracking this rare and hard to see bird
also, why do koalas bellow
can listen: www.mquter.qut.edu.au/sensor
Labels: IEEEeScience08
¶ 11:43 AM| (0) comments |cites (technorati) |
IEEE eScience: Opening keynote
really bare bones notes
Rich Wolski, Keynote Speaker
UCSB
Building Science Clouds Using commodity, Open-Source Software Components
Start with what’s happening in the commercial clouds and work toward the scientific world
A lot of people getting excited in distributed computing – commercial entities going ahead, both big and small
virtualization < web services < SLAs – users : need to automate as much as possible, also need to make it clear to users what they are getting when they rent space in the cloud
public cloud vs. private cloud
What can be done in a cloud? – workflows are great, but there are things that don’t work well in grids
what extensions or mods are required for scientific applications, data assimilation, multiplayer gaming (latency constraints)
How do clouds interact with other systems – like mobile devices which are already in/on their own cloud
open source cloud: simple, extensible, widely available and popular technologies, easy to install and maintain
examples: U Chi, Nimbus, on GT4 and Globus (from grid computing), but not looking for when grids act like couds
Enomalism now called ECP (startup with open source), difficult and pretty opaque
They’re building on Eucalyptus.cs.ucsb.edu: elastic utility computing architecture linking your programs to useful systems
- linux (like Amazon)
How to know if it’s a cloud? try things on it that you can do on amazon web services (like in the interface) and see if it can do these things (no strict definition of “cloud” just knew a cloud when they saw one, AWS).
- software overlay so it didn’t really monkey with the underlying infrastructure (with some grid things you had to blow away your operating system, install a whole bunch of libraries, and it was really difficult to know what it was doing and to support it)
A driver of this was to save money – researchers wanted to work in the commercial cloud, but it’s really expensive. If nothing else, they can use this to debug before moving into the commercial cloud.
Goal to more like democratize – not to replace Amazon and Google services at all – but to allow for other people to try things, but you won’t have their data centers, you’ll still need to have the hardware (and other things).
Interface is based on Amazon’s published WSDL, EC2 command-line tools, REST interface
sys admin – cloud admin, toolset for user accounting and cloud management
Security – they figured it out (WS-security, SSH key generation, etc.), but makes installation and sequencing a bit complicated – refer to his slides
Performance – if Xen is installed right, they haven’t seen the performance hits people are concerned about with virtualization
You can play with the Eucalyptus Public Cloud – with limitations
Benchmarking tests show it’s running very fast and responding like the amazon services
Part of VGrADS – an NSF project – Linked Environments for Atmospheric Discovery – real time use of Doppler radar for local/regional forecasting
… lots of details
clouds and grids are distinct
cloud
- full private cluster is provisioned
- individual can only get a tiny fraction of the total resource pool
- no support for cloud federation
- opaque wrt resources
grid
- built so that a single user can get most or use the entire cloud for a single project
- federation as first principle
- resources are exposed
Questions from the audience:
- teragrid – I guess currently a 3week queue? maybe use this to get projects in faster?
- SLAs how to actually monitor and enforce?
Rich Wolski, Keynote Speaker
UCSB
Building Science Clouds Using commodity, Open-Source Software Components
Start with what’s happening in the commercial clouds and work toward the scientific world
A lot of people getting excited in distributed computing – commercial entities going ahead, both big and small
virtualization < web services < SLAs – users : need to automate as much as possible, also need to make it clear to users what they are getting when they rent space in the cloud
public cloud vs. private cloud
What can be done in a cloud? – workflows are great, but there are things that don’t work well in grids
what extensions or mods are required for scientific applications, data assimilation, multiplayer gaming (latency constraints)
How do clouds interact with other systems – like mobile devices which are already in/on their own cloud
open source cloud: simple, extensible, widely available and popular technologies, easy to install and maintain
examples: U Chi, Nimbus, on GT4 and Globus (from grid computing), but not looking for when grids act like couds
Enomalism now called ECP (startup with open source), difficult and pretty opaque
They’re building on Eucalyptus.cs.ucsb.edu: elastic utility computing architecture linking your programs to useful systems
- linux (like Amazon)
How to know if it’s a cloud? try things on it that you can do on amazon web services (like in the interface) and see if it can do these things (no strict definition of “cloud” just knew a cloud when they saw one, AWS).
- software overlay so it didn’t really monkey with the underlying infrastructure (with some grid things you had to blow away your operating system, install a whole bunch of libraries, and it was really difficult to know what it was doing and to support it)
A driver of this was to save money – researchers wanted to work in the commercial cloud, but it’s really expensive. If nothing else, they can use this to debug before moving into the commercial cloud.
Goal to more like democratize – not to replace Amazon and Google services at all – but to allow for other people to try things, but you won’t have their data centers, you’ll still need to have the hardware (and other things).
Interface is based on Amazon’s published WSDL, EC2 command-line tools, REST interface
sys admin – cloud admin, toolset for user accounting and cloud management
Security – they figured it out (WS-security, SSH key generation, etc.), but makes installation and sequencing a bit complicated – refer to his slides
Performance – if Xen is installed right, they haven’t seen the performance hits people are concerned about with virtualization
You can play with the Eucalyptus Public Cloud – with limitations
Benchmarking tests show it’s running very fast and responding like the amazon services
Part of VGrADS – an NSF project – Linked Environments for Atmospheric Discovery – real time use of Doppler radar for local/regional forecasting
… lots of details
clouds and grids are distinct
cloud
- full private cluster is provisioned
- individual can only get a tiny fraction of the total resource pool
- no support for cloud federation
- opaque wrt resources
grid
- built so that a single user can get most or use the entire cloud for a single project
- federation as first principle
- resources are exposed
Questions from the audience:
- teragrid – I guess currently a 3week queue? maybe use this to get projects in faster?
- SLAs how to actually monitor and enforce?
Labels: IEEEeScience08
¶ 10:47 AM| (0) comments |cites (technorati) |
