Automated Data Observatories

Automated data observatory

Yes!Delft AI+Blockchain Validation Lab

We ensure that individual researchers, artists, and professionals, as well as NGOs and small and large organizations can benefit equally from big data in the age of artificial intelligence.

The Data Sisyphus Poor metadata management causes much repeated tasks, errors, non-billable hours and uncredited work.	Open Data Open data cannot be just ‘downloaded’. It is not ready-to-use, and often not even public.	Trustworthy AI What can go wrong with the algorithm? Finding unwanted outcomes and correcting them in complex systems.	Research Automation Repeaded data processing and validation steps are best made, documented, logged by computers.
Data Curation Data sits everywhere and it is not easy to find even at home. Our curators know where to dig.	Professional Data Processing Uncut diamonds need to be polished. Data is only potential information, raw and unprocessed.	Metadata: Documentation & Codebooks Adding FAIR metadata exponentially increases the value of data. We use DataCite and SDMX statistical coding.	Data-as-Service Reusable, easy-to-import, interoperable, always fresh data in tidy formats with a modern API.

We are looking for partners to develop our technological solution in a financially sustainable way, bringing increasingly relevant curated open data to light.

Get involved in services: our ongoing projects, team of contributors, open-source libraries and use our data for publications. See some use cases.

Follow news about us or the more comprehensive Data & Lyrics blog.

Contact us .

Our Product/Market Fit was validated in the world’s 2nd ranked university-backed incubator program, the Yes!Delft AI Validation Lab.

Automated Data Observatories

Try and join our data ecosystems—Our modern data observatories are turning shared, pooled, public and open data with the help of our open-source statistical software into to data-as-service.

Creative & Cultural Sectors Industries Data Observatory

Relying on modern data science, the re-use of open governmental data, open science data, and novel harmonized data collection we aim to fill in the gaps left in the official statistics of the European Union about the creative and cultural sectors and industries.

Green Deal Data Observatory

An ambitious project to connect environmental sensory data, political and policy survey data with socio-economic indicators.

Competition Data Observatory

Our observatory is monitoring the certain segments of the European economy, and develops tools for computational antitrust in Europe.

Economy Data Observatory

An incubator for socio-economic data observatories. Its first offspring is the Competition Data Observatory.

Digital Music Observatory

The Digital Music Observatory is a fully automated, open source, open data observatory that links public datasets in order to provide a comprehensive view of the European music industry.

Recent Posts

How We Add Value to Public Data With Imputation and Forecasting

Public data sources are often plagued with missng values. Naively you may think that you can ignore them, but think twice: in most cases, missing data in a table is not missing information, but rather malformatted information which will destroy your beautiful visualization or stop your application from working. In this example we show how we increase the usable subset of a public dataset by 66.7%, rendering useful what would otherwise have been a deal-breaker in panel regressions or machine learning applications.

Daniel Antal

Last updated on Nov 11, 2021 7 min read

How We Add Value to Public Data With Imputation and Forecasting

How We Add Value to Public Data With Better Curation And Documentation?

Many people ask if we can really add value to free data that can be downloaded from the Internet by anybody. We do not only work with easy-to-download data, but we know that free, public data usually requires a lot of work to become really valuable. To start with, it is not always easy to find.

Daniel Antal

Last updated on Nov 8, 2021 5 min read

How We Add Value to Public Data With Better Curation And Documentation?

CCSI Data Observatory

The creative and cultural sectors and industries are mainly made of networks of freelancers and microenterprises, with very few medium-sized companies. Their economic performance, problems, and innovation capacities hidden. Our open collaboration to create this data observatory is committed to change this. Relying on modern data science, the re-use of open governmental data, open science data, and novel harmonized data collection we aim to fill in the gaps left in the official statistics of the European Union.

Daniel Antal

Oct 6, 2021 2 min read

The Data Sisyphus

Sisyphus was punished by being forced to roll an immense boulder up a hill only for it to roll down every time it neared the top, repeating this action for eternity. When was a file downloaded from the internet? What happened with it sense? Are their updates? Did the bibliographical reference was made for quotations? Missing values imputed? Currency translated? Who knows about it – who created a dataset, who contributed to it? Which is the final, checked, approved by a senior manager?

Daniel Antal

Jul 8, 2021 7 min read

Including Indicators from Arab Barometer in Our Observatory

A new version of the retroharmonize R package – which is working with retrospective, ex post harmonization of survey data – was released yesterday after peer-review on CRAN. It allows us to compare opinion polling data from the Arab Barometer with the Eurobarometer and Afrorbarometer. This is the first version that is released in the rOpenGov community, a community of R package developers on open government data analytics and related topics.

Daniel Antal

Jun 28, 2021 6 min read

Including Indicators from Arab Barometer in Our Observatory

See all posts

Data

API to our data, our sources, how we process and validate it, how we increase its quality.

Survey Harmonization

Ex post survey harmonization is in a way a passive form of pooling research funding because you can utilize information from surveying that were made on somebody else’s expense. The aim of ex ante survey harmonization is to maximize the value from future retrospective harmonization; in a way, it is an active form of pooling research funding, because you benefit from money spent on related open governmental and open science survey programs.

Jul 5, 2021

Data API

Get data from the Green Deal Data Observatory via our API

Daniel Antal, Botond Vitos

Last updated on Jul 8, 2021

Metadata

Uncut diamonds need to be cut, polished, and you have to make sure that they come from a legal source.

Daniel Antal

Last updated on Jul 8, 2021

Open Data

Many countries in the world allow access to a vast array of information, such as documents under freedom of information requests, statistics, datasets. In the European Union, most taxpayer financed data in government administration, transport, or meteorology, for example, can be usually re-used. More and more scientific output is expected to be reviewable and reproducible, which implies open access.

Last updated on Jun 25, 2021

Contributors of the Green Deal Data Observatory

Join our open collaboration team as a data curator, developer or business developer! More about contributing: Automated Observatory Contributors’ Handbook.

developers

Daniel Antal

Data Scientist & Founder of the Digital Music Observatory

Kasia Kulma

Contributor, data science and software engineering

Leo Lahti

rOpenGov coordinator

Botond Vitos

Data scientists and developer

Andrés García Molina, PhD

Data Scientist & Ethnomusicologist

Pyry Kantanen

R package testing and data curation.

data curators

Istvan Simon

Data curator, climate risk management

Stephan Okhuijsen

Data visualization and dissemination

service development team

Annette Wong

Contributor, digital strategist and product marketer

Robin Nagy

Mentor, Contributor, Business Development

Suzan Sidal

Business Case Development & Service Design

institutional partners

rOpenGov

rOpenGov network

Datagraver

Data dissemination partner

join us

New Curators

Future curator

New Developers

Future co-developer

Observatory Business Associate

Future Team Member

Milestones

Green Deal Open Collaboration

Reprex and collaborators

Mar 2021 – Present the Hague, Netherlands

International Open Data Day 2021:

We organized the only event in the Netherlands
Decided to start an open collaboration on music and climate change.

Trustworthy AI PPP

NLAIC + EU AI Alliance

Feb 2021 – Present the Netherlands

Joined the Dutch & the European AI PPPs.

Open-Source Software

Our peer-reviewed, open source statistical software packages

We believe that transparency is the key to the highest data quality. We use only open source software. We open up the critical elements of our software for peer-review.

We use open-source software, there is no vendor lock-in.
Our data products go through many, automated (unit) tests, replacing countless error-prone human validation working hours.
The critical elements of our code go through external validation and peer-review by computational statisticians and data scientists.

Quickly discover relevant content by filtering our software releases..

Daniel Antal

Last updated on Jun 28, 2021

retroharmonize R package for survey harmonization

The goal of retroharmonize is to facilitate retrospective (ex-post) harmonization of data, particularly survey data, in a reproducible manner.

Daniel Antal

Jun 3, 2020

regions R package to create sub-national statistical indicators

regions currently takes care of 20,000 sub-divisional boundary changes in Europe since 1999.

Daniel Antal

Jun 3, 2020

iotables R package for working with symmetric input-output tables

The goal of iotables is to make allow a programmatic acces to the symmetric input-output tables of Eurostat. It creates multipliers, calculates direct, indirect and induced effects from European SIOT tables.

Open-Source Software

Our peer-reviewed, open source statistical software packages

Quickly discover relevant content by filtering our software releases..

Contact

Recent Publications

Quickly discover relevant content by filtering publications.

Daniel Antal, Rita Kiss (2009). Market size of the re-usable public sector information in Hungary. Market size of the re-usable public sector information in Hungary.

Follow Connect