data observatory is a data and/or knowledge management system to collect, connect, process and provide information as a service. We have researched the European data observatory landscape and reviewed 62 public observatories that have been in operation since 1991.
The observatory metaphor refers to a location used for observing terrestrial or celestial events. The European Commission, the OECD, the UNESCO and the Council of Europe are supporting numerous permanent data observation programs in economics and social sciences to support research and development and evidence-based policymaking. We are creating automated observatories following the best practices of reproducible research.
private data observatoryhandles the proprietary data and knowledge of a single organisation and serves its information needs.
collaborative observatorycapitalises on the synergies coming from combining the information resources of two or more entities. Our demo observatories grew out of a large, collaborative observatory, CEEMID.
public observatoryis not only building on information sharing among members, but make the knowledge available as a public good or as a subscription service.
Recognition & Funding
We have found that less than
5%of the public observatories operate without public funding; collaborative data programs often have an incentive to remain hidden from the public eye.
60%of the public observatories communicate findings solely in publications and do not provide quantitative data access to the members or the public;
25%of the examined observatories have ceased their operation by 2020.
Recognized data observatories, which are often work in agreement in contractual with the EU, the OECD, the UNESCO, or the Council of Europe, or some national body, usually offer some services to the public, and in Europe, if they are funded by the taxpayer, usually have to make some of the data fully
open accessfor free.
All the above facts indicate that the operating model of data observatories could be improved. Our Demo Music Observatory and CCS Demo Observatory are aiming for international recognition and want to promote the highest quality standards for data observatories.
There are no universally accepted quality standards for data observatories.
- The algorithm or program that is creating the indicator data should be open source, and preferably peer-reviewed by statisticians;
- The data and procedures that create the data must be confirmable and auditable (See reproducible research standards.)
- If data is published, it should be tidy data.
- The data should be reviewable and reproducible;
- If the data is created from taxpayer funding, it should comply with the EU public sector re-use and open data regulations;
- When data is used in the public interest, it should follow the Open Policy Analysis standards of open materials, open analysis and open output.
These quality standards can be best achieved by reproducible research automation. Research automation has the benefit that it makes data creation, review and publication very fast and timely. Our demo observatories can be refreshed every day or every working day.
We automate observatory management following the best practices of reproducible research.
Applying the open collaboration approach of data sciences to the management of observatories means decentralized opt-in and opt-out for integrating open data with proprietary data assets. We leverage open data and our know-how in re-processing open data into business insight, while the confidential information of consortium members is concealed and used only as long as the owner understands the benefits. We keep the research costs low by automated data collection, integration and processing, and the quality high by making all our statistical software code peer-reviewed and open source.
Our apps can integrate your internal financial or transactional data with thousands of external indicators from official statistics to satellite sensory data. We make sure that your organization always has valid KPIs that are aware of the changes of your working environment.
Our work is historically based on
CEEMID, that integrated the data of about 100 music organizations with various other data sources, and created about 2000 statistical indicators for better pricing, advocating for better economic and tax policies, better granting or better marketing of music. CEEMID was constructed from the ad-hoc contributions of users, and was never meant to be public.
we are currently bringing out to the light in full transparency, with open code and open data several demo observatories for a permanent observation of social and economic data in an automated collection, processing, validation, documentation and publishing workflow. Most of this know-how was developed in CEEMID, but it is not specific to the music industry. We present them as a proof of concept for the best practices of our research automation.
- A demonstration and proof of concept that a modern, European data observatory can be in large part automated, and adhere to the highest standards of statistical disclosure, reproducible research and open policy analysis (see Evidence-based, Open Policy Analysis).
Demo Music Observatory
Data is power, and
big data creates injustice. Organizations that control large amounts of data, for example, the entire listening history of hundreds of millions of people in all major countries of the world, can train algorithms and robots that drive most of the music sales in the world. They can make your investment into a sound recording successful or doomed. They can circumvent or help a local content regulation, reinforce, or disable a national cultural policy goal. A country may introduce national artist quotas on radio, if all the youth will be personally recommended foreign songs in their music discovery age in the very same country.
We want big data to work for small venues, independent labels, startups, great and undiscovered artists. We believe that you cannot make a successful album launch, a market entry or introduce a successful cultural policy without large amounts of well processed and correctly analysed data. We want to create a
Music Observatory that integrates the small data of many small bands, small labels, small venues, small countries, and mount correct the injustice. Make algorithms transparent and the competition fair.
In 2020 most of the data is proprietary to a few, U.S.-based companies, while most of the paying audience is the European Union. This year, many European governments started to challenge the competition conditions. The tide is turning, and we do not see this as an EU-US rivalry, because many players of the American music industry share all the pains of the European music industry.
Our Demo Music Observatory in September 2020 and got into the prestigious Yes!Delft AI Validation Lab. Our demo observatory is an example how we believe the European Data Observatory should be built. An observatory is a permanent observation point for social and economic data. We want to prove that this process can be made cost-effectively and efficiently, providing a high-quality, valuable and timely product by employing best practices in research automation and open source software, using open data in open collaboration with the music industry, artists, technicians and managers.
The European Music Observatory will have three data pillars. Let us know if you need any data from these pillars and we will try to put it there in a correctly processed, tidy, well-documented format with a document identifier and citation template, and we will refresh it every day. Automatically. That is what we do: we automate research to make it error-free and timely.
Let us know if you like this structure and let us know if anything is missing.
We will work with you to provide affordable data: Our partners can name their price for the first 50 indicators. We are not stingy data producers.
We want to support the European music industry and our friends in North America, Australia, and all over the world to turn the tables with @ref(innovation) innovation coming from the open source community.
We want to help you to create alternative recommendation engines that actually recommend songs from your country, and we want to give you very clear export market targets with the help of AI. Our Listen Local Initiative is aiming to create recommendation engines for cities and regions, and make sure that local bands are recommended to local audiences and audiences in the regions where they will be touring after the Covid-19 pandemic.
We want your evidence to stand a fighting chance against large teams of professional legal and economics teams on the other side with proper valuations and damage claims. And we want to present all those hundreds and thousands of pages automatically, going through dozens and dozens of automated “unit-tests” until nobody can find errors.
We want you to be able to prove to your fans, the press, your economy minister that music in many countries has not been at the mercy of the taxpayer, but has been carrying far heavier tax burdens than manufacturers. We want to make your case that the music industry plays a vital role in the European economic recovery and job creation, because we can create economic impact assessments on GDP, employment, tax, import and export effects automatically.
Because music and culture are often managed at the level of cities, regions and communities, we want to give you all the data on sub-national levels, whether for regions, metropolitan areas or smaller divisions.
Creative & Cultural Sector Demo Observatory
With our partners we are developing a similar demonstration for the broader cultural, creative and copyright-based industries.
We are exploring the ideas of future data needs of several industries and policy areas with our partners.